Nov 01 20:27:56.640465 ip-10-0-141-35 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 01 20:27:56.640474 ip-10-0-141-35 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 01 20:27:56.640481 ip-10-0-141-35 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 01 20:27:56.640710 ip-10-0-141-35 systemd[1]: Failed to start Kubernetes Kubelet. Nov 01 20:28:06.890429 ip-10-0-141-35 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 01 20:28:06.890447 ip-10-0-141-35 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot af0d4f6a5d95484a9110d6ce5814e71c -- Nov 01 20:30:08.335281 ip-10-0-141-35 systemd[1]: Starting Kubernetes Kubelet... Nov 01 20:30:08.708150 ip-10-0-141-35 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 20:30:08.708150 ip-10-0-141-35 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 01 20:30:08.708150 ip-10-0-141-35 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 20:30:08.708150 ip-10-0-141-35 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 20:30:08.713116 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713031 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:30:08.713116 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713116 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:30:08.713116 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713120 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:30:08.713213 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713123 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:30:08.713213 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713126 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:30:08.713213 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713129 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:30:08.713213 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713133 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:30:08.713213 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713136 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:30:08.713213 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713138 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:30:08.713213 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713141 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:30:08.713213 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713144 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:30:08.713213 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713147 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:30:08.713213 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713150 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:30:08.713213 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713153 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:30:08.713213 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713155 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:30:08.713213 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713158 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:30:08.713213 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713160 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:30:08.713213 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713163 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:30:08.713213 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713166 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:30:08.713213 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713168 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:30:08.713213 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713171 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:30:08.713213 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713175 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:30:08.713213 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713178 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:30:08.713672 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713181 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:30:08.713672 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713185 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:30:08.713672 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713189 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:30:08.713672 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713192 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:30:08.713672 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713196 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:30:08.713672 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713200 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:30:08.713672 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713203 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:30:08.713672 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713205 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:30:08.713672 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713208 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:30:08.713672 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713211 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:30:08.713672 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713214 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:30:08.713672 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713216 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:30:08.713672 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713219 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:30:08.713672 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713222 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:30:08.713672 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713225 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:30:08.713672 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713228 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:30:08.713672 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713232 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:30:08.713672 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713234 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:30:08.713672 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713237 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:30:08.714156 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713239 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:30:08.714156 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713242 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:30:08.714156 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713245 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:30:08.714156 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713249 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:30:08.714156 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713251 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:30:08.714156 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713254 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:30:08.714156 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713258 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:30:08.714156 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713261 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:30:08.714156 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713264 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:30:08.714156 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713267 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:30:08.714156 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713269 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:30:08.714156 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713272 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:30:08.714156 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713274 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:30:08.714156 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713277 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:30:08.714156 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713280 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:30:08.714156 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713283 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:30:08.714156 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713286 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:30:08.714156 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713289 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:30:08.714156 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713291 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:30:08.714593 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713294 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:30:08.714593 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713297 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:30:08.714593 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713299 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:30:08.714593 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713302 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:30:08.714593 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713304 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:30:08.714593 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713307 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:30:08.714593 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713309 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:30:08.714593 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713312 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:30:08.714593 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713315 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:30:08.714593 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713317 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:30:08.714593 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713319 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:30:08.714593 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713323 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:30:08.714593 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713326 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:30:08.714593 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713328 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:30:08.714593 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713331 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:30:08.714593 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713333 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:30:08.714593 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713336 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:30:08.714593 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713338 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:30:08.714593 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713341 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:30:08.714593 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713343 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:30:08.715078 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713346 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:30:08.715078 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713348 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:30:08.715078 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713351 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:30:08.715078 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713354 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:30:08.715078 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713356 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:30:08.715078 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713359 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:30:08.715078 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713361 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:30:08.715078 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713365 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:30:08.715078 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713368 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:30:08.715078 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713371 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:30:08.715078 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713373 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:30:08.715078 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713376 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:30:08.715078 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713378 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:30:08.715078 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713381 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:30:08.715078 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713383 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:30:08.715078 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713386 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:30:08.715078 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713388 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:30:08.715078 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713391 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:30:08.715078 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713393 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:30:08.715569 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713396 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:30:08.715569 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713398 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:30:08.715569 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713402 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:30:08.715569 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713404 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:30:08.715569 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713407 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:30:08.715569 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.713410 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:30:08.715569 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714232 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:30:08.715569 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714239 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:30:08.715569 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714242 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:30:08.715569 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714245 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:30:08.715569 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714248 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:30:08.715569 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714250 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:30:08.715569 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714253 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:30:08.715569 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714255 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:30:08.715569 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714258 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:30:08.715569 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714261 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:30:08.715569 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714263 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:30:08.715569 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714266 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:30:08.715569 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714269 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:30:08.716040 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714271 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:30:08.716040 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714275 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:30:08.716040 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714278 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:30:08.716040 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714281 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:30:08.716040 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714284 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:30:08.716040 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714287 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:30:08.716040 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714290 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:30:08.716040 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714292 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:30:08.716040 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714295 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:30:08.716040 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714298 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:30:08.716040 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714300 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:30:08.716040 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714303 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:30:08.716040 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714305 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:30:08.716040 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714308 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:30:08.716040 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714310 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:30:08.716040 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714313 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:30:08.716040 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714315 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:30:08.716040 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714317 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:30:08.716040 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714320 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:30:08.716040 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714323 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:30:08.716511 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714326 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:30:08.716511 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714329 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:30:08.716511 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714332 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:30:08.716511 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714334 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:30:08.716511 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714337 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:30:08.716511 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714340 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:30:08.716511 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714342 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:30:08.716511 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714345 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:30:08.716511 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714347 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:30:08.716511 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714350 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:30:08.716511 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714352 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:30:08.716511 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714354 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:30:08.716511 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714357 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:30:08.716511 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714360 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:30:08.716511 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714363 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:30:08.716511 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714366 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:30:08.716511 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714369 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:30:08.716511 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714372 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:30:08.716511 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714374 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:30:08.716511 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714377 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:30:08.716990 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714379 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:30:08.716990 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714382 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:30:08.716990 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714384 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:30:08.716990 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714387 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:30:08.716990 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714390 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:30:08.716990 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714393 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:30:08.716990 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714395 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:30:08.716990 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714398 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:30:08.716990 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714400 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:30:08.716990 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714402 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:30:08.716990 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714405 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:30:08.716990 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714407 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:30:08.716990 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714410 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:30:08.716990 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714413 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:30:08.716990 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714416 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:30:08.716990 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714418 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:30:08.716990 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714421 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:30:08.716990 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714423 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:30:08.716990 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714426 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:30:08.717465 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714428 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:30:08.717465 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714431 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:30:08.717465 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714433 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:30:08.717465 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714436 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:30:08.717465 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714438 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:30:08.717465 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714441 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:30:08.717465 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714444 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:30:08.717465 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714447 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:30:08.717465 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714450 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:30:08.717465 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714452 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:30:08.717465 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714455 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:30:08.717465 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714457 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:30:08.717465 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714460 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:30:08.717465 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714462 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:30:08.717465 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714465 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:30:08.717465 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714468 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:30:08.717465 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714473 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:30:08.717465 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714477 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:30:08.717465 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714480 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:30:08.717937 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714483 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:30:08.717937 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714486 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:30:08.717937 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714489 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:30:08.717937 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714491 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:30:08.717937 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714494 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:30:08.717937 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714497 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:30:08.717937 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714500 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:30:08.717937 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714504 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:30:08.717937 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714506 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:30:08.717937 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714509 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:30:08.717937 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714512 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:30:08.717937 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714514 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:30:08.717937 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714517 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:30:08.717937 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714519 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:30:08.717937 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.714525 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:30:08.717937 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715021 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:30:08.717937 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715028 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:30:08.717937 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715031 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:30:08.717937 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715035 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:30:08.717937 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715039 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:30:08.718429 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715042 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:30:08.718429 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715045 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:30:08.718429 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715048 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:30:08.718429 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715052 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:30:08.718429 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715056 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:30:08.718429 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715059 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:30:08.718429 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715062 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:30:08.718429 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715065 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:30:08.718429 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715067 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:30:08.718429 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715070 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:30:08.718429 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715073 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:30:08.718429 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715075 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:30:08.718429 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715078 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:30:08.718429 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715081 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:30:08.718429 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715084 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:30:08.718429 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715087 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:30:08.718429 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715089 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:30:08.718429 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715092 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:30:08.718429 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715095 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:30:08.718429 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715098 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:30:08.718907 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715101 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:30:08.718907 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715104 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:30:08.718907 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715107 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:30:08.718907 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715110 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:30:08.718907 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715113 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:30:08.718907 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715115 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:30:08.718907 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715118 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:30:08.718907 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715120 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:30:08.718907 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715123 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:30:08.718907 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715126 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:30:08.718907 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715129 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:30:08.718907 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715132 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:30:08.718907 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715135 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:30:08.718907 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715137 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:30:08.718907 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715140 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:30:08.718907 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715143 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:30:08.718907 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715145 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:30:08.718907 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715148 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:30:08.718907 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715151 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:30:08.718907 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715153 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:30:08.719390 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715155 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:30:08.719390 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715158 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:30:08.719390 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715160 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:30:08.719390 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715163 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:30:08.719390 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715166 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:30:08.719390 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715168 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:30:08.719390 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715170 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:30:08.719390 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715173 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:30:08.719390 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715175 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:30:08.719390 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715178 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:30:08.719390 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715180 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:30:08.719390 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715183 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:30:08.719390 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715186 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:30:08.719390 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715189 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:30:08.719390 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715191 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:30:08.719390 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715194 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:30:08.719390 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715196 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:30:08.719390 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715199 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:30:08.719390 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715201 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:30:08.719939 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715204 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:30:08.719939 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715207 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:30:08.719939 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715210 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:30:08.719939 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715212 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:30:08.719939 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715216 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:30:08.719939 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715219 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:30:08.719939 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715221 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:30:08.719939 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715224 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:30:08.719939 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715226 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:30:08.719939 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715229 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:30:08.719939 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715231 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:30:08.719939 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715234 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:30:08.719939 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715236 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:30:08.719939 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715239 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:30:08.719939 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715241 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:30:08.719939 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715244 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:30:08.719939 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715248 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:30:08.719939 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715251 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:30:08.719939 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715254 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:30:08.720413 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715257 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:30:08.720413 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715259 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:30:08.720413 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715262 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:30:08.720413 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715264 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:30:08.720413 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715266 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:30:08.720413 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715269 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:30:08.720413 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715271 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:30:08.720413 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715274 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:30:08.720413 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715277 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:30:08.720413 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715282 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:30:08.720413 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715285 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:30:08.720413 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715287 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:30:08.720413 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715290 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:30:08.720413 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715292 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:30:08.720413 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715295 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:30:08.720413 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715298 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:30:08.720413 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715301 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:30:08.720413 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715304 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:30:08.720413 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715307 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:30:08.720413 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715310 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:30:08.720873 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715312 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:30:08.720873 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715315 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:30:08.720873 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.715317 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:30:08.720873 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716242 2580 flags.go:64] FLAG: --address="0.0.0.0" Nov 01 20:30:08.720873 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716256 2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 01 20:30:08.720873 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716264 2580 flags.go:64] FLAG: --anonymous-auth="true" Nov 01 20:30:08.720873 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716269 2580 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 01 20:30:08.720873 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716274 2580 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 01 20:30:08.720873 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716278 2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 01 20:30:08.720873 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716282 2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 01 20:30:08.720873 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716287 2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 01 20:30:08.720873 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716290 2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 01 20:30:08.720873 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716293 2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 01 20:30:08.720873 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716296 2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 01 20:30:08.720873 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716300 2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 01 20:30:08.720873 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716303 2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 01 20:30:08.720873 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716306 2580 flags.go:64] FLAG: --cgroup-root="" Nov 01 20:30:08.720873 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716309 2580 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 01 20:30:08.720873 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716313 2580 flags.go:64] FLAG: --client-ca-file="" Nov 01 20:30:08.720873 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716316 2580 flags.go:64] FLAG: --cloud-provider="external" Nov 01 20:30:08.720873 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716319 2580 flags.go:64] FLAG: --cluster-dns="[]" Nov 01 20:30:08.720873 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716325 2580 flags.go:64] FLAG: --cluster-domain="" Nov 01 20:30:08.720873 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716328 2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 01 20:30:08.721418 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716331 2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 01 20:30:08.721418 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716335 2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 01 20:30:08.721418 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716338 2580 flags.go:64] FLAG: --container-log-max-files="5" Nov 01 20:30:08.721418 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716342 2580 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 01 20:30:08.721418 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716346 2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 01 20:30:08.721418 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716350 2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 01 20:30:08.721418 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716353 2580 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 01 20:30:08.721418 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716356 2580 flags.go:64] FLAG: --contention-profiling="false" Nov 01 20:30:08.721418 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716359 2580 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 01 20:30:08.721418 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716362 2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 01 20:30:08.721418 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716366 2580 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 01 20:30:08.721418 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716369 2580 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 01 20:30:08.721418 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716373 2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 01 20:30:08.721418 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716376 2580 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 01 20:30:08.721418 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716379 2580 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 01 20:30:08.721418 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716383 2580 flags.go:64] FLAG: --enable-load-reader="false" Nov 01 20:30:08.721418 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716386 2580 flags.go:64] FLAG: --enable-server="true" Nov 01 20:30:08.721418 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716389 2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 01 20:30:08.721418 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716393 2580 flags.go:64] FLAG: --event-burst="100" Nov 01 20:30:08.721418 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716397 2580 flags.go:64] FLAG: --event-qps="50" Nov 01 20:30:08.721418 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716400 2580 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 01 20:30:08.721418 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716403 2580 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 01 20:30:08.721418 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716406 2580 flags.go:64] FLAG: --eviction-hard="" Nov 01 20:30:08.721418 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716410 2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 01 20:30:08.722024 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716413 2580 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 01 20:30:08.722024 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716416 2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 01 20:30:08.722024 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716420 2580 flags.go:64] FLAG: --eviction-soft="" Nov 01 20:30:08.722024 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716423 2580 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 01 20:30:08.722024 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716426 2580 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 01 20:30:08.722024 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716429 2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 01 20:30:08.722024 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716436 2580 flags.go:64] FLAG: --experimental-mounter-path="" Nov 01 20:30:08.722024 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716439 2580 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 01 20:30:08.722024 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716442 2580 flags.go:64] FLAG: --fail-swap-on="true" Nov 01 20:30:08.722024 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716445 2580 flags.go:64] FLAG: --feature-gates="" Nov 01 20:30:08.722024 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716449 2580 flags.go:64] FLAG: --file-check-frequency="20s" Nov 01 20:30:08.722024 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716452 2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 01 20:30:08.722024 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716456 2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 01 20:30:08.722024 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716460 2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 01 20:30:08.722024 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716463 2580 flags.go:64] FLAG: --healthz-port="10248" Nov 01 20:30:08.722024 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716466 2580 flags.go:64] FLAG: --help="false" Nov 01 20:30:08.722024 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716469 2580 flags.go:64] FLAG: --hostname-override="ip-10-0-141-35.us-west-1.compute.internal" Nov 01 20:30:08.722024 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716473 2580 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 01 20:30:08.722024 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716476 2580 flags.go:64] FLAG: --http-check-frequency="20s" Nov 01 20:30:08.722024 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716479 2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 01 20:30:08.722024 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716482 2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 01 20:30:08.722024 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716485 2580 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 01 20:30:08.722024 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716488 2580 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 01 20:30:08.722596 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716491 2580 flags.go:64] FLAG: --image-service-endpoint="" Nov 01 20:30:08.722596 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716495 2580 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 01 20:30:08.722596 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716498 2580 flags.go:64] FLAG: --kube-api-burst="100" Nov 01 20:30:08.722596 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716500 2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 01 20:30:08.722596 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716504 2580 flags.go:64] FLAG: --kube-api-qps="50" Nov 01 20:30:08.722596 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716506 2580 flags.go:64] FLAG: --kube-reserved="" Nov 01 20:30:08.722596 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716510 2580 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 01 20:30:08.722596 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716512 2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 01 20:30:08.722596 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716516 2580 flags.go:64] FLAG: --kubelet-cgroups="" Nov 01 20:30:08.722596 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716519 2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 01 20:30:08.722596 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716522 2580 flags.go:64] FLAG: --lock-file="" Nov 01 20:30:08.722596 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716525 2580 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 01 20:30:08.722596 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716528 2580 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 01 20:30:08.722596 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716531 2580 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 01 20:30:08.722596 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716537 2580 flags.go:64] FLAG: --log-json-split-stream="false" Nov 01 20:30:08.722596 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716542 2580 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 01 20:30:08.722596 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716546 2580 flags.go:64] FLAG: --log-text-split-stream="false" Nov 01 20:30:08.722596 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716549 2580 flags.go:64] FLAG: --logging-format="text" Nov 01 20:30:08.722596 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716552 2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 01 20:30:08.722596 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716555 2580 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 01 20:30:08.722596 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716558 2580 flags.go:64] FLAG: --manifest-url="" Nov 01 20:30:08.722596 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716561 2580 flags.go:64] FLAG: --manifest-url-header="" Nov 01 20:30:08.722596 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716566 2580 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 01 20:30:08.722596 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716570 2580 flags.go:64] FLAG: --max-open-files="1000000" Nov 01 20:30:08.722596 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716574 2580 flags.go:64] FLAG: --max-pods="110" Nov 01 20:30:08.723212 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716577 2580 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 01 20:30:08.723212 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716580 2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 01 20:30:08.723212 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716583 2580 flags.go:64] FLAG: --memory-manager-policy="None" Nov 01 20:30:08.723212 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716586 2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 01 20:30:08.723212 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716589 2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 01 20:30:08.723212 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716592 2580 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 01 20:30:08.723212 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716595 2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 01 20:30:08.723212 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716602 2580 flags.go:64] FLAG: --node-status-max-images="50" Nov 01 20:30:08.723212 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716605 2580 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 01 20:30:08.723212 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716608 2580 flags.go:64] FLAG: --oom-score-adj="-999" Nov 01 20:30:08.723212 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716611 2580 flags.go:64] FLAG: --pod-cidr="" Nov 01 20:30:08.723212 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716614 2580 flags.go:64] FLAG: --pod-manifest-path="" Nov 01 20:30:08.723212 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716617 2580 flags.go:64] FLAG: --pod-max-pids="-1" Nov 01 20:30:08.723212 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716620 2580 flags.go:64] FLAG: --pods-per-core="0" Nov 01 20:30:08.723212 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716623 2580 flags.go:64] FLAG: --port="10250" Nov 01 20:30:08.723212 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716626 2580 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 01 20:30:08.723212 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716629 2580 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0a39cb5957849d0db" Nov 01 20:30:08.723212 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716632 2580 flags.go:64] FLAG: --qos-reserved="" Nov 01 20:30:08.723212 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716635 2580 flags.go:64] FLAG: --read-only-port="10255" Nov 01 20:30:08.723212 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716638 2580 flags.go:64] FLAG: --register-node="true" Nov 01 20:30:08.723212 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716641 2580 flags.go:64] FLAG: --register-with-taints="" Nov 01 20:30:08.723212 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716645 2580 flags.go:64] FLAG: --registry-burst="10" Nov 01 20:30:08.723212 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716649 2580 flags.go:64] FLAG: --registry-qps="5" Nov 01 20:30:08.723212 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716652 2580 flags.go:64] FLAG: --reserved-cpus="" Nov 01 20:30:08.723212 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716656 2580 flags.go:64] FLAG: --reserved-memory="" Nov 01 20:30:08.723212 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716659 2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 01 20:30:08.723864 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716663 2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 01 20:30:08.723864 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716666 2580 flags.go:64] FLAG: --rotate-certificates="false" Nov 01 20:30:08.723864 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716669 2580 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 01 20:30:08.723864 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716672 2580 flags.go:64] FLAG: --runonce="false" Nov 01 20:30:08.723864 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716675 2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 01 20:30:08.723864 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716678 2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 01 20:30:08.723864 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716681 2580 flags.go:64] FLAG: --seccomp-default="false" Nov 01 20:30:08.723864 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716684 2580 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 01 20:30:08.723864 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716687 2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 01 20:30:08.723864 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716690 2580 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 01 20:30:08.723864 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716693 2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 01 20:30:08.723864 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716696 2580 flags.go:64] FLAG: --storage-driver-password="root" Nov 01 20:30:08.723864 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716699 2580 flags.go:64] FLAG: --storage-driver-secure="false" Nov 01 20:30:08.723864 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716702 2580 flags.go:64] FLAG: --storage-driver-table="stats" Nov 01 20:30:08.723864 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716705 2580 flags.go:64] FLAG: --storage-driver-user="root" Nov 01 20:30:08.723864 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716708 2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 01 20:30:08.723864 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716712 2580 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 01 20:30:08.723864 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716715 2580 flags.go:64] FLAG: --system-cgroups="" Nov 01 20:30:08.723864 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716718 2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 01 20:30:08.723864 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716723 2580 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 01 20:30:08.723864 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716726 2580 flags.go:64] FLAG: --tls-cert-file="" Nov 01 20:30:08.723864 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716729 2580 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 01 20:30:08.723864 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716733 2580 flags.go:64] FLAG: --tls-min-version="" Nov 01 20:30:08.723864 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716736 2580 flags.go:64] FLAG: --tls-private-key-file="" Nov 01 20:30:08.723864 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716739 2580 flags.go:64] FLAG: --topology-manager-policy="none" Nov 01 20:30:08.724476 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716741 2580 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 01 20:30:08.724476 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716744 2580 flags.go:64] FLAG: --topology-manager-scope="container" Nov 01 20:30:08.724476 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716747 2580 flags.go:64] FLAG: --v="2" Nov 01 20:30:08.724476 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716754 2580 flags.go:64] FLAG: --version="false" Nov 01 20:30:08.724476 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716758 2580 flags.go:64] FLAG: --vmodule="" Nov 01 20:30:08.724476 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716762 2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 01 20:30:08.724476 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.716766 2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 01 20:30:08.724476 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.716906 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:30:08.724476 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.716910 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:30:08.724476 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.716914 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:30:08.724476 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.716916 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:30:08.724476 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.716919 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:30:08.724476 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.716921 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:30:08.724476 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.716924 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:30:08.724476 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.716927 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:30:08.724476 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.716929 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:30:08.724476 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.716932 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:30:08.724476 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.716934 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:30:08.724476 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.716937 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:30:08.724476 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.716940 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:30:08.724476 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.716943 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:30:08.724970 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.716945 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:30:08.724970 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.716948 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:30:08.724970 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.716950 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:30:08.724970 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.716953 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:30:08.724970 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.716955 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:30:08.724970 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.716958 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:30:08.724970 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.716960 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:30:08.724970 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.716963 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:30:08.724970 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.716965 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:30:08.724970 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.716968 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:30:08.724970 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.716970 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:30:08.724970 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.716973 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:30:08.724970 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.716975 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:30:08.724970 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.716978 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:30:08.724970 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.716982 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:30:08.724970 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.716985 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:30:08.724970 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.716987 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:30:08.724970 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717002 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:30:08.724970 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717007 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:30:08.725447 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717021 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:30:08.725447 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717024 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:30:08.725447 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717027 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:30:08.725447 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717030 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:30:08.725447 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717032 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:30:08.725447 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717035 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:30:08.725447 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717037 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:30:08.725447 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717040 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:30:08.725447 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717042 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:30:08.725447 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717045 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:30:08.725447 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717048 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:30:08.725447 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717050 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:30:08.725447 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717053 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:30:08.725447 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717055 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:30:08.725447 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717058 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:30:08.725447 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717060 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:30:08.725447 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717063 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:30:08.725447 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717065 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:30:08.725447 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717068 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:30:08.725447 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717071 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:30:08.725928 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717073 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:30:08.725928 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717077 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:30:08.725928 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717080 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:30:08.725928 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717082 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:30:08.725928 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717085 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:30:08.725928 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717087 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:30:08.725928 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717090 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:30:08.725928 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717094 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:30:08.725928 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717099 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:30:08.725928 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717102 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:30:08.725928 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717104 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:30:08.725928 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717107 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:30:08.725928 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717110 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:30:08.725928 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717113 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:30:08.725928 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717115 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:30:08.725928 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717118 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:30:08.725928 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717120 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:30:08.725928 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717123 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:30:08.725928 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717126 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:30:08.725928 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717128 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:30:08.726427 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717131 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:30:08.726427 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717133 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:30:08.726427 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717136 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:30:08.726427 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717138 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:30:08.726427 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717141 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:30:08.726427 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717143 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:30:08.726427 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717146 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:30:08.726427 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717148 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:30:08.726427 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717151 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:30:08.726427 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717153 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:30:08.726427 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717156 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:30:08.726427 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717158 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:30:08.726427 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717161 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:30:08.726427 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717163 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:30:08.726427 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717166 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:30:08.726427 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717170 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:30:08.726427 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717173 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:30:08.726427 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717176 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:30:08.726427 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717179 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:30:08.726882 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717183 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:30:08.726882 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717187 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:30:08.726882 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717190 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:30:08.726882 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717193 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:30:08.726882 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717195 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:30:08.726882 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717198 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:30:08.726882 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717201 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:30:08.726882 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717204 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:30:08.726882 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717207 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:30:08.726882 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717210 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:30:08.726882 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717212 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:30:08.726882 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717215 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:30:08.726882 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717217 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:30:08.726882 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717220 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:30:08.726882 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.717247 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 20:30:08.726882 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717281 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:30:08.727309 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717286 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:30:08.727309 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717289 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:30:08.727309 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717291 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:30:08.727309 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717294 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:30:08.727309 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717296 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:30:08.727309 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717299 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:30:08.727309 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717302 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:30:08.727309 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717306 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:30:08.727309 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717309 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:30:08.727309 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717312 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:30:08.727309 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717315 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:30:08.727309 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717318 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:30:08.727309 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717320 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:30:08.727309 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717323 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:30:08.727309 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717325 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:30:08.727309 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717328 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:30:08.727309 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717331 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:30:08.727309 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717338 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:30:08.727309 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717340 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:30:08.727744 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717343 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:30:08.727744 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717346 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:30:08.727744 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717348 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:30:08.727744 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717351 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:30:08.727744 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717354 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:30:08.727744 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717357 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:30:08.727744 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717359 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:30:08.727744 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717362 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:30:08.727744 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717365 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:30:08.727744 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717367 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:30:08.727744 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717370 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:30:08.727744 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717373 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:30:08.727744 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717375 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:30:08.727744 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717377 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:30:08.727744 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717380 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:30:08.727744 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717382 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:30:08.727744 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717385 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:30:08.727744 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717387 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:30:08.727744 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717390 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:30:08.727744 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717392 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:30:08.728230 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717395 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:30:08.728230 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717398 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:30:08.728230 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717400 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:30:08.728230 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717403 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:30:08.728230 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717406 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:30:08.728230 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717408 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:30:08.728230 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717411 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:30:08.728230 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717413 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:30:08.728230 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717416 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:30:08.728230 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717419 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:30:08.728230 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717424 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:30:08.728230 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717427 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:30:08.728230 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717431 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:30:08.728230 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717434 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:30:08.728230 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717436 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:30:08.728230 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717439 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:30:08.728230 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717441 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:30:08.728230 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717445 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:30:08.728230 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717447 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:30:08.728714 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717450 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:30:08.728714 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717452 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:30:08.728714 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717455 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:30:08.728714 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717457 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:30:08.728714 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717460 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:30:08.728714 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717462 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:30:08.728714 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717466 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:30:08.728714 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717470 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:30:08.728714 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717473 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:30:08.728714 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717476 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:30:08.728714 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717478 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:30:08.728714 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717481 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:30:08.728714 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717484 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:30:08.728714 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717486 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:30:08.728714 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717489 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:30:08.728714 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717492 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:30:08.728714 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717494 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:30:08.728714 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717497 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:30:08.728714 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717500 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:30:08.728714 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717502 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:30:08.729200 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717505 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:30:08.729200 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717508 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:30:08.729200 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717510 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:30:08.729200 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717515 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:30:08.729200 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717518 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:30:08.729200 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717520 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:30:08.729200 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717523 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:30:08.729200 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717525 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:30:08.729200 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717528 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:30:08.729200 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717530 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:30:08.729200 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717533 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:30:08.729200 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717536 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:30:08.729200 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717539 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:30:08.729200 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717541 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:30:08.729200 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717544 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:30:08.729200 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717547 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:30:08.729200 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717550 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:30:08.729200 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717552 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:30:08.729200 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717555 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:30:08.729200 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717557 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:30:08.729670 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717560 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:30:08.729670 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717562 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:30:08.729670 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717565 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:30:08.729670 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717567 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:30:08.729670 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717570 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:30:08.729670 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717572 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:30:08.729670 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.717574 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:30:08.729670 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.725388 2580 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 01 20:30:08.729670 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.725406 2580 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 01 20:30:08.729670 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.725425 2580 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 01 20:30:08.729670 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.725430 2580 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 01 20:30:08.729670 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725489 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:30:08.729670 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725494 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:30:08.729670 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725497 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:30:08.729670 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725500 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:30:08.729670 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725502 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:30:08.729670 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725505 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:30:08.729670 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725508 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:30:08.729670 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725510 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:30:08.730141 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725513 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:30:08.730141 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725516 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:30:08.730141 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725519 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:30:08.730141 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725521 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:30:08.730141 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725525 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:30:08.730141 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725528 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:30:08.730141 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725531 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:30:08.730141 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725534 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:30:08.730141 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725536 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:30:08.730141 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725539 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:30:08.730141 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725541 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:30:08.730141 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725544 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:30:08.730141 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725547 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:30:08.730141 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725550 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:30:08.730141 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725552 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:30:08.730141 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725555 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:30:08.730141 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725557 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:30:08.730141 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725560 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:30:08.730141 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725562 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:30:08.730141 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725565 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:30:08.730611 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725568 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:30:08.730611 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725571 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:30:08.730611 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725574 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:30:08.730611 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725577 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:30:08.730611 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725580 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:30:08.730611 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725582 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:30:08.730611 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725585 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:30:08.730611 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725588 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:30:08.730611 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725590 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:30:08.730611 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725593 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:30:08.730611 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725595 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:30:08.730611 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725598 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:30:08.730611 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725600 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:30:08.730611 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725603 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:30:08.730611 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725605 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:30:08.730611 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725608 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:30:08.730611 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725610 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:30:08.730611 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725613 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:30:08.730611 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725617 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:30:08.730611 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725619 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:30:08.731104 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725622 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:30:08.731104 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725625 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:30:08.731104 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725627 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:30:08.731104 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725630 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:30:08.731104 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725632 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:30:08.731104 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725635 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:30:08.731104 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725637 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:30:08.731104 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725640 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:30:08.731104 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725643 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:30:08.731104 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725645 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:30:08.731104 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725648 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:30:08.731104 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725652 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:30:08.731104 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725656 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:30:08.731104 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725659 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:30:08.731104 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725661 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:30:08.731104 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725664 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:30:08.731104 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725667 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:30:08.731104 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725670 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:30:08.731104 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725672 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:30:08.731551 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725675 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:30:08.731551 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725677 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:30:08.731551 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725680 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:30:08.731551 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725683 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:30:08.731551 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725685 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:30:08.731551 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725688 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:30:08.731551 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725690 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:30:08.731551 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725693 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:30:08.731551 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725696 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:30:08.731551 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725699 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:30:08.731551 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725701 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:30:08.731551 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725705 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:30:08.731551 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725707 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:30:08.731551 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725710 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:30:08.731551 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725713 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:30:08.731551 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725716 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:30:08.731551 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725718 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:30:08.731551 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725721 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:30:08.731551 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725723 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:30:08.731551 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725726 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:30:08.732022 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725729 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:30:08.732022 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725731 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:30:08.732022 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725734 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:30:08.732022 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725738 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:30:08.732022 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725741 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:30:08.732022 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725745 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:30:08.732022 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725748 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:30:08.732022 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725751 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:30:08.732022 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725753 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:30:08.732022 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725756 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:30:08.732022 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725759 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:30:08.732022 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725761 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:30:08.732022 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725764 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:30:08.732022 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725766 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:30:08.732022 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725769 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:30:08.732022 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725771 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:30:08.732022 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725774 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:30:08.732022 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725776 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:30:08.732022 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725779 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:30:08.732470 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.725815 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 20:30:08.732470 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725953 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:30:08.732470 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725957 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:30:08.732470 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725960 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:30:08.732470 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725963 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:30:08.732470 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725967 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:30:08.732470 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725970 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:30:08.732470 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725972 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:30:08.732470 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725974 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:30:08.732470 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725977 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:30:08.732470 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725979 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:30:08.732470 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725982 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:30:08.732470 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725985 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:30:08.732470 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725987 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:30:08.732470 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.725990 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:30:08.732470 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726013 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:30:08.732848 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726018 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:30:08.732848 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726021 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:30:08.732848 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726024 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:30:08.732848 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726027 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:30:08.732848 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726029 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:30:08.732848 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726032 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:30:08.732848 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726035 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:30:08.732848 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726038 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:30:08.732848 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726041 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:30:08.732848 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726043 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:30:08.732848 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726046 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:30:08.732848 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726049 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:30:08.732848 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726051 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:30:08.732848 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726054 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:30:08.732848 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726057 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:30:08.732848 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726059 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:30:08.732848 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726062 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:30:08.732848 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726064 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:30:08.732848 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726067 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:30:08.732848 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726069 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:30:08.733375 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726072 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:30:08.733375 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726075 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:30:08.733375 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726080 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:30:08.733375 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726083 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:30:08.733375 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726085 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:30:08.733375 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726088 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:30:08.733375 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726092 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:30:08.733375 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726096 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:30:08.733375 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726099 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:30:08.733375 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726102 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:30:08.733375 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726105 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:30:08.733375 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726107 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:30:08.733375 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726110 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:30:08.733375 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726112 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:30:08.733375 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726115 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:30:08.733375 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726117 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:30:08.733375 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726120 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:30:08.733375 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726123 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:30:08.733375 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726126 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:30:08.733375 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726129 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:30:08.733841 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726131 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:30:08.733841 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726133 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:30:08.733841 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726136 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:30:08.733841 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726138 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:30:08.733841 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726141 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:30:08.733841 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726144 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:30:08.733841 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726146 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:30:08.733841 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726149 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:30:08.733841 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726151 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:30:08.733841 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726154 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:30:08.733841 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726156 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:30:08.733841 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726159 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:30:08.733841 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726161 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:30:08.733841 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726164 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:30:08.733841 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726166 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:30:08.733841 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726169 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:30:08.733841 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726172 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:30:08.733841 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726174 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:30:08.733841 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726177 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:30:08.733841 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726179 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:30:08.734311 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726182 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:30:08.734311 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726186 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:30:08.734311 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726189 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:30:08.734311 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726192 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:30:08.734311 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726195 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:30:08.734311 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726197 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:30:08.734311 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726200 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:30:08.734311 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726203 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:30:08.734311 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726205 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:30:08.734311 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726208 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:30:08.734311 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726211 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:30:08.734311 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726213 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:30:08.734311 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726215 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:30:08.734311 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726218 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:30:08.734311 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726220 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:30:08.734311 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726223 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:30:08.734311 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726226 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:30:08.734311 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726228 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:30:08.734311 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726231 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:30:08.734803 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726233 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:30:08.734803 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726236 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:30:08.734803 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726238 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:30:08.734803 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726241 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:30:08.734803 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726243 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:30:08.734803 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726246 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:30:08.734803 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726248 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:30:08.734803 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726250 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:30:08.734803 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726253 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:30:08.734803 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726256 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:30:08.734803 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726258 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:30:08.734803 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726261 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:30:08.734803 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.726284 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 20:30:08.734803 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726328 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:30:08.734803 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726332 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:30:08.734803 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726335 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:30:08.735454 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726338 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:30:08.735454 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726341 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:30:08.735454 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726343 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:30:08.735454 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726346 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:30:08.735454 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726349 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:30:08.735454 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726352 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:30:08.735454 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726354 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:30:08.735454 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726357 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:30:08.735454 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726360 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:30:08.735454 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726362 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:30:08.735454 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726365 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:30:08.735454 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726367 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:30:08.735454 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726370 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:30:08.735454 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726376 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:30:08.735454 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726378 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:30:08.735454 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726381 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:30:08.735454 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726383 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:30:08.735454 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726386 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:30:08.735454 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726388 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:30:08.735454 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726391 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:30:08.736294 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726393 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:30:08.736294 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726396 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:30:08.736294 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726398 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:30:08.736294 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726401 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:30:08.736294 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726403 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:30:08.736294 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726406 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:30:08.736294 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726409 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:30:08.736294 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726412 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:30:08.736294 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726415 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:30:08.736294 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726417 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:30:08.736294 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726420 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:30:08.736294 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726422 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:30:08.736294 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726425 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:30:08.736294 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726427 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:30:08.736294 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726430 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:30:08.736294 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726433 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:30:08.736294 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726436 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:30:08.736294 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726439 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:30:08.736294 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726441 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:30:08.736882 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726444 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:30:08.736882 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726447 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:30:08.736882 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726449 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:30:08.736882 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726452 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:30:08.736882 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726454 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:30:08.736882 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726457 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:30:08.736882 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726461 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:30:08.736882 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726464 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:30:08.736882 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726466 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:30:08.736882 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726469 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:30:08.736882 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726472 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:30:08.736882 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726475 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:30:08.736882 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726478 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:30:08.736882 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726480 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:30:08.736882 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726483 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:30:08.736882 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726485 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:30:08.736882 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726489 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:30:08.736882 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726492 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:30:08.736882 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726495 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:30:08.736882 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726498 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:30:08.737651 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726501 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:30:08.737651 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726503 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:30:08.737651 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726506 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:30:08.737651 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726509 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:30:08.737651 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726511 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:30:08.737651 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726514 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:30:08.737651 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726516 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:30:08.737651 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726518 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:30:08.737651 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726521 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:30:08.737651 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726524 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:30:08.737651 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726526 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:30:08.737651 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726529 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:30:08.737651 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726531 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:30:08.737651 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726534 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:30:08.737651 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726536 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:30:08.737651 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726539 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:30:08.737651 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726541 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:30:08.737651 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726544 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:30:08.737651 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726548 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:30:08.738166 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726551 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:30:08.738166 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726553 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:30:08.738166 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726556 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:30:08.738166 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726558 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:30:08.738166 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726561 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:30:08.738166 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726563 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:30:08.738166 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726566 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:30:08.738166 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726568 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:30:08.738166 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726571 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:30:08.738166 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726573 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:30:08.738166 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726576 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:30:08.738166 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726579 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:30:08.738166 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726582 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:30:08.738166 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726586 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:30:08.738166 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726589 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:30:08.738166 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726592 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:30:08.738166 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726595 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:30:08.738166 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726597 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:30:08.738166 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726601 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:30:08.738687 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726604 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:30:08.738687 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726606 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:30:08.738687 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726609 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:30:08.738687 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726611 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:30:08.738687 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726614 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:30:08.738687 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:08.726617 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:30:08.738687 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.726672 2580 server.go:957] "Client rotation is on, will bootstrap in background" Nov 01 20:30:08.738687 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.729860 2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 01 20:30:08.738687 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.730850 2580 server.go:1014] "Starting client certificate rotation" Nov 01 20:30:08.738687 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.731343 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 20:30:08.738687 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.732167 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 20:30:08.738687 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.733147 2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 20:30:08.738687 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.736677 2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 20:30:08.749833 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.749811 2580 log.go:25] "Validated CRI v1 runtime API" Nov 01 20:30:08.754239 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.754221 2580 log.go:25] "Validated CRI v1 image API" Nov 01 20:30:08.755135 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.755106 2580 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 01 20:30:08.759979 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.759959 2580 fs.go:135] Filesystem UUIDs: map[4a81d1fc-210d-4b90-b516-5f74051a24f7:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 a58926c8-7ad4-4602-9128-d2a090c34e44:/dev/nvme0n1p4] Nov 01 20:30:08.760053 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.759980 2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 01 20:30:08.765340 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.765206 2580 manager.go:217] Machine: {Timestamp:2026-11-01 20:30:08.763806414 +0000 UTC m=+0.333294009 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100892 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25db723a05217783923f1e742ba382 SystemUUID:ec25db72-3a05-2177-8392-3f1e742ba382 BootID:af0d4f6a-5d95-484a-9110-d6ce5814e71c Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:38:0f:d6:e4:2f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:38:0f:d6:e4:2f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e6:e2:fc:c5:b7:06 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 01 20:30:08.765340 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.765327 2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 01 20:30:08.765501 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.765450 2580 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 01 20:30:08.769093 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.769061 2580 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 01 20:30:08.769238 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.769096 2580 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-35.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 01 20:30:08.769281 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.769240 2580 topology_manager.go:143] "Creating topology manager with none policy" Nov 01 20:30:08.769281 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.769249 2580 container_manager_linux.go:311] "Creating device plugin manager" Nov 01 20:30:08.769343 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.769287 2580 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 20:30:08.769343 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.769307 2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 20:30:08.770553 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.770539 2580 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 01 20:30:08.774892 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.774878 2580 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 01 20:30:08.775043 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.775034 2580 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 01 20:30:08.775894 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.775884 2580 kubelet.go:493] "Attempting to sync node with API server" Nov 01 20:30:08.775927 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.775900 2580 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 01 20:30:08.775927 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.775911 2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 01 20:30:08.775927 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.775922 2580 kubelet.go:405] "Adding apiserver pod source" Nov 01 20:30:08.776040 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.775932 2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 01 20:30:08.776347 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.776324 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 20:30:08.778800 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.778783 2580 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 01 20:30:08.786436 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.786420 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 20:30:08.787337 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.787322 2580 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 01 20:30:08.787374 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.787364 2580 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 01 20:30:08.787581 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.787567 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 01 20:30:08.787620 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.787594 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 01 20:30:08.787620 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.787608 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 01 20:30:08.787620 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.787617 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 01 20:30:08.787700 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.787627 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 01 20:30:08.787700 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.787638 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 01 20:30:08.788288 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.788278 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 01 20:30:08.788318 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.788292 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 01 20:30:08.788318 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.788304 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 01 20:30:08.788318 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.788312 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 01 20:30:08.788395 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.788331 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 01 20:30:08.788395 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.788345 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 01 20:30:08.789111 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.789101 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 01 20:30:08.789146 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.789114 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 01 20:30:08.794076 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.794058 2580 server.go:1263] "Started kubelet" Nov 01 20:30:08.794167 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.794103 2580 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 01 20:30:08.794358 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.794190 2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 01 20:30:08.794438 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.794428 2580 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 01 20:30:08.794970 ip-10-0-141-35 systemd[1]: Started Kubernetes Kubelet. Nov 01 20:30:08.800495 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.800469 2580 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 01 20:30:08.801565 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.801546 2580 server.go:317] "Adding debug handlers to kubelet server" Nov 01 20:30:08.803090 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:08.803064 2580 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 01 20:30:08.808411 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.808383 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 01 20:30:08.809775 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.809759 2580 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 01 20:30:08.809956 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:08.809934 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-35.us-west-1.compute.internal\" not found" Nov 01 20:30:08.810735 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.810714 2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 01 20:30:08.810735 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.810735 2580 factory.go:55] Registering systemd factory Nov 01 20:30:08.810863 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.810745 2580 factory.go:223] Registration of the systemd container factory successfully Nov 01 20:30:08.810863 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.810846 2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 01 20:30:08.810965 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.810948 2580 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 01 20:30:08.811059 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.811044 2580 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 01 20:30:08.811137 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.811008 2580 factory.go:153] Registering CRI-O factory Nov 01 20:30:08.811137 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.811094 2580 factory.go:223] Registration of the crio container factory successfully Nov 01 20:30:08.811137 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.811118 2580 factory.go:103] Registering Raw factory Nov 01 20:30:08.811137 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.811134 2580 reconstruct.go:97] "Volume reconstruction finished" Nov 01 20:30:08.811255 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.811142 2580 reconciler.go:29] "Reconciler: start to sync state" Nov 01 20:30:08.811255 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.811160 2580 manager.go:1199] Started watching for new ooms in manager Nov 01 20:30:08.812187 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.812167 2580 manager.go:319] Starting recovery of all containers Nov 01 20:30:08.823171 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.823150 2580 manager.go:324] Recovery completed Nov 01 20:30:08.827716 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.827696 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 20:30:08.830330 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.830307 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-35.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 20:30:08.830419 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.830344 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-35.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 20:30:08.830419 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.830355 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-35.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 20:30:08.831040 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.831014 2580 cpu_manager.go:226] "Starting" policy="none" Nov 01 20:30:08.831040 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.831026 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 20:30:08.831134 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.831029 2580 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 01 20:30:08.831134 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.831127 2580 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 01 20:30:08.833533 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.833520 2580 policy_none.go:50] "Start" Nov 01 20:30:08.833586 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.833537 2580 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 01 20:30:08.833586 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.833553 2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 01 20:30:08.835607 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.835596 2580 policy_none.go:44] "Start" Nov 01 20:30:08.838448 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.838430 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-35.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 20:30:08.838541 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.838464 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-35.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 20:30:08.838541 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.838477 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-35.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 20:30:08.883061 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.882987 2580 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 01 20:30:08.887357 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.887338 2580 manager.go:341] "Starting Device Plugin manager" Nov 01 20:30:08.887496 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:08.887371 2580 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 01 20:30:08.887496 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.887382 2580 server.go:87] "Starting device plugin registration server" Nov 01 20:30:08.887713 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.887697 2580 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 01 20:30:08.887776 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.887714 2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 01 20:30:08.887833 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.887788 2580 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 01 20:30:08.887881 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.887875 2580 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 01 20:30:08.887936 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.887887 2580 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 01 20:30:08.888551 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:08.888525 2580 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 01 20:30:08.888640 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:08.888601 2580 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-35.us-west-1.compute.internal\" not found" Nov 01 20:30:08.933275 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:08.932620 2580 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-35.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 01 20:30:08.933275 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.932629 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-35.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 01 20:30:08.935262 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:08.932622 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-35.us-west-1.compute.internal.18e405df6eb8f036 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-35.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-35.us-west-1.compute.internal,},FirstTimestamp:2026-11-01 20:30:08.793759798 +0000 UTC m=+0.363247381,LastTimestamp:2026-11-01 20:30:08.793759798 +0000 UTC m=+0.363247381,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-35.us-west-1.compute.internal,}" Nov 01 20:30:08.961024 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.960942 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 01 20:30:08.962164 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.962134 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 01 20:30:08.962164 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.962161 2580 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 01 20:30:08.962333 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.962185 2580 kubelet.go:2521] "Starting kubelet main sync loop" Nov 01 20:30:08.962333 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:08.962227 2580 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 01 20:30:08.988407 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.988383 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 20:30:08.989745 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.989727 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-35.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 20:30:08.989840 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.989759 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-35.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 20:30:08.989840 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.989770 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-35.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 20:30:08.989840 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:08.989796 2580 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-35.us-west-1.compute.internal" Nov 01 20:30:08.992117 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:08.992023 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-35.us-west-1.compute.internal.18e405df70e6f064 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-35.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-35.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-35.us-west-1.compute.internal,},FirstTimestamp:2026-11-01 20:30:08.830328932 +0000 UTC m=+0.399816514,LastTimestamp:2026-11-01 20:30:08.830328932 +0000 UTC m=+0.399816514,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-35.us-west-1.compute.internal,}" Nov 01 20:30:08.992365 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:08.992333 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-35.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 01 20:30:08.992467 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:08.992390 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 01 20:30:09.054564 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:09.054527 2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 01 20:30:09.054748 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:09.054600 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vqj2s" Nov 01 20:30:09.062552 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:09.062530 2580 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-141-35.us-west-1.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-35.us-west-1.compute.internal"] Nov 01 20:30:09.062660 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:09.062588 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 20:30:09.063615 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:09.063595 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-35.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 20:30:09.063697 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:09.063634 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-35.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 20:30:09.063697 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:09.063646 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-35.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 20:30:09.065818 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:09.065805 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 20:30:09.065971 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:09.065955 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-35.us-west-1.compute.internal" Nov 01 20:30:09.066019 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:09.066010 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 20:30:09.066556 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:09.066538 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-35.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 20:30:09.066597 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:09.066542 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-35.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 20:30:09.066597 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:09.066595 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-35.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 20:30:09.066657 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:09.066607 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-35.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 20:30:09.066657 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:09.066570 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-35.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 20:30:09.066657 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:09.066643 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-35.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 20:30:09.068722 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:09.068704 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-35.us-west-1.compute.internal" Nov 01 20:30:09.068815 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:09.068731 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 20:30:09.069413 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:09.069398 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-35.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 20:30:09.069474 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:09.069424 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-35.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 20:30:09.069474 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:09.069440 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-35.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 20:30:09.086643 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:09.086610 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 01 20:30:09.086776 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:09.086588 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-35.us-west-1.compute.internal.18e405df70e73fa3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-35.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-141-35.us-west-1.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-141-35.us-west-1.compute.internal,},FirstTimestamp:2026-11-01 20:30:08.830349219 +0000 UTC m=+0.399836801,LastTimestamp:2026-11-01 20:30:08.830349219 +0000 UTC m=+0.399836801,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-35.us-west-1.compute.internal,}" Nov 01 20:30:09.094828 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:09.094804 2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-35.us-west-1.compute.internal\" not found" node="ip-10-0-141-35.us-west-1.compute.internal" Nov 01 20:30:09.100187 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:09.100164 2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-35.us-west-1.compute.internal\" not found" node="ip-10-0-141-35.us-west-1.compute.internal" Nov 01 20:30:09.112564 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:09.112534 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/91ec11d0905cc00deef779d06bc93fe3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-35.us-west-1.compute.internal\" (UID: \"91ec11d0905cc00deef779d06bc93fe3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-35.us-west-1.compute.internal" Nov 01 20:30:09.112652 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:09.112566 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91ec11d0905cc00deef779d06bc93fe3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-35.us-west-1.compute.internal\" (UID: \"91ec11d0905cc00deef779d06bc93fe3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-35.us-west-1.compute.internal" Nov 01 20:30:09.112652 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:09.112584 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f192393bd55f8e34f5db37f9db86763a-config\") pod \"kube-apiserver-proxy-ip-10-0-141-35.us-west-1.compute.internal\" (UID: \"f192393bd55f8e34f5db37f9db86763a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-35.us-west-1.compute.internal" Nov 01 20:30:09.146318 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:09.146292 2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 20:30:09.146712 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:09.146695 2580 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-141-35.us-west-1.compute.internal\" not found" node="ip-10-0-141-35.us-west-1.compute.internal" Nov 01 20:30:09.213528 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:09.213451 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91ec11d0905cc00deef779d06bc93fe3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-35.us-west-1.compute.internal\" (UID: \"91ec11d0905cc00deef779d06bc93fe3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-35.us-west-1.compute.internal" Nov 01 20:30:09.213528 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:09.213483 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f192393bd55f8e34f5db37f9db86763a-config\") pod \"kube-apiserver-proxy-ip-10-0-141-35.us-west-1.compute.internal\" (UID: \"f192393bd55f8e34f5db37f9db86763a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-35.us-west-1.compute.internal" Nov 01 20:30:09.213528 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:09.213499 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/91ec11d0905cc00deef779d06bc93fe3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-35.us-west-1.compute.internal\" (UID: \"91ec11d0905cc00deef779d06bc93fe3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-35.us-west-1.compute.internal" Nov 01 20:30:09.213721 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:09.213565 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f192393bd55f8e34f5db37f9db86763a-config\") pod \"kube-apiserver-proxy-ip-10-0-141-35.us-west-1.compute.internal\" (UID: \"f192393bd55f8e34f5db37f9db86763a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-35.us-west-1.compute.internal" Nov 01 20:30:09.213721 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:09.213565 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/91ec11d0905cc00deef779d06bc93fe3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-35.us-west-1.compute.internal\" (UID: \"91ec11d0905cc00deef779d06bc93fe3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-35.us-west-1.compute.internal" Nov 01 20:30:09.213721 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:09.213566 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91ec11d0905cc00deef779d06bc93fe3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-35.us-west-1.compute.internal\" (UID: \"91ec11d0905cc00deef779d06bc93fe3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-35.us-west-1.compute.internal" Nov 01 20:30:09.235650 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:09.235620 2580 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-141-35.us-west-1.compute.internal\" not found" node="ip-10-0-141-35.us-west-1.compute.internal" Nov 01 20:30:09.346844 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:09.346804 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 20:30:09.347767 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:09.347750 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-35.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 20:30:09.347860 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:09.347779 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-35.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 20:30:09.347860 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:09.347790 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-35.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 20:30:09.347860 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:09.347819 2580 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-35.us-west-1.compute.internal" Nov 01 20:30:09.398470 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:09.398432 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-35.us-west-1.compute.internal" Nov 01 20:30:09.402343 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:09.402318 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-35.us-west-1.compute.internal" Nov 01 20:30:09.405564 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:09.405546 2580 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-35.us-west-1.compute.internal" Nov 01 20:30:09.405631 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:09.405573 2580 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-35.us-west-1.compute.internal\": node \"ip-10-0-141-35.us-west-1.compute.internal\" not found" Nov 01 20:30:09.528498 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:09.528411 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-35.us-west-1.compute.internal\" not found" Nov 01 20:30:09.629325 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:09.629284 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-35.us-west-1.compute.internal\" not found" Nov 01 20:30:09.729842 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:09.729810 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-35.us-west-1.compute.internal\" not found" Nov 01 20:30:09.730997 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:09.730979 2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 01 20:30:09.731123 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:09.731104 2580 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 01 20:30:09.731248 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:09.731162 2580 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a9584bc60fd4645d0a6dd8d98805fda7-6263b06240bad7c4.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-141-35.us-west-1.compute.internal.18e405df70e6f064\": read tcp 10.0.141.35:51926->18.224.125.164:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-141-35.us-west-1.compute.internal.18e405df70e6f064 default 5208 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-35.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-35.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-35.us-west-1.compute.internal,},FirstTimestamp:2026-11-01 20:30:08 +0000 UTC,LastTimestamp:2026-11-01 20:30:09.063614597 +0000 UTC m=+0.633102180,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-35.us-west-1.compute.internal,}" Nov 01 20:30:09.831031 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:09.830941 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-35.us-west-1.compute.internal\" not found" Nov 01 20:30:09.932116 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:09.932083 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-35.us-west-1.compute.internal\" not found" Nov 01 20:30:10.032815 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:10.032771 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-35.us-west-1.compute.internal\" not found" Nov 01 20:30:10.058935 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.058885 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 20:25:09 +0000 UTC" deadline="2028-04-01 08:21:41.985159609 +0000 UTC" Nov 01 20:30:10.058935 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.058931 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12395h51m31.926232004s" Nov 01 20:30:10.094489 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.094397 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 20:30:10.095484 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.095464 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-35.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 20:30:10.095564 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.095500 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-35.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 20:30:10.095564 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.095511 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-35.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 20:30:10.133256 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:10.133212 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-35.us-west-1.compute.internal\" not found" Nov 01 20:30:10.215835 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.215792 2580 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 20:30:10.216047 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.215915 2580 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 20:30:10.310574 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.310519 2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-35.us-west-1.compute.internal" Nov 01 20:30:10.374356 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.374287 2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 20:30:10.375743 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.375725 2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-35.us-west-1.compute.internal" Nov 01 20:30:10.401650 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.401620 2580 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 20:30:10.439456 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.439428 2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 20:30:10.552677 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:10.552648 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf192393bd55f8e34f5db37f9db86763a.slice/crio-6e9ed6520b34ab71b0c03cc82e133742536fd7ed40a5a40364132cc19eef6fbc WatchSource:0}: Error finding container 6e9ed6520b34ab71b0c03cc82e133742536fd7ed40a5a40364132cc19eef6fbc: Status 404 returned error can't find the container with id 6e9ed6520b34ab71b0c03cc82e133742536fd7ed40a5a40364132cc19eef6fbc Nov 01 20:30:10.553192 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:10.553166 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91ec11d0905cc00deef779d06bc93fe3.slice/crio-7fc4de5f427b94b5aeece9302a62a6fade2769408cde35df2e0058c848f3b483 WatchSource:0}: Error finding container 7fc4de5f427b94b5aeece9302a62a6fade2769408cde35df2e0058c848f3b483: Status 404 returned error can't find the container with id 7fc4de5f427b94b5aeece9302a62a6fade2769408cde35df2e0058c848f3b483 Nov 01 20:30:10.558460 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.558439 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 20:30:10.777421 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.777329 2580 apiserver.go:52] "Watching apiserver" Nov 01 20:30:10.811717 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.811664 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 01 20:30:10.890191 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.890149 2580 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 01 20:30:10.890543 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.890523 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-qjtnx","openshift-dns/node-resolver-w6nn2","openshift-multus/multus-5k2px","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb94","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-35.us-west-1.compute.internal","kube-system/global-pull-secret-syncer-xh6zg","kube-system/konnectivity-agent-2f2ct","openshift-image-registry/node-ca-bztv7","openshift-multus/multus-additional-cni-plugins-fmnrq","openshift-multus/network-metrics-daemon-vxzfk","openshift-network-diagnostics/network-check-target-ghh5z","openshift-network-operator/iptables-alerter-w7qmz","openshift-ovn-kubernetes/ovnkube-node-tv7jl","kube-system/kube-apiserver-proxy-ip-10-0-141-35.us-west-1.compute.internal"] Nov 01 20:30:10.894597 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.894574 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bztv7" Nov 01 20:30:10.896857 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.896837 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-w6nn2" Nov 01 20:30:10.899287 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.899269 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:10.901663 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.901641 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:10.904088 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.904049 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxzfk" Nov 01 20:30:10.904158 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:10.904137 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxzfk" podUID="970d71e5-f861-4277-8fa5-6ec740418e51" Nov 01 20:30:10.907144 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.907124 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fmnrq" Nov 01 20:30:10.909457 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.909435 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb94" Nov 01 20:30:10.911906 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.911885 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2f2ct" Nov 01 20:30:10.914502 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.914484 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xh6zg" Nov 01 20:30:10.914600 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:10.914553 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xh6zg" podUID="6b4f17a0-efd1-4024-a201-4078069b04e7" Nov 01 20:30:10.916845 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.916826 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ghh5z" Nov 01 20:30:10.916930 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:10.916894 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ghh5z" podUID="9ee67cbd-8115-4c48-b5cd-5299ed284bce" Nov 01 20:30:10.919344 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.919326 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-w7qmz" Nov 01 20:30:10.921901 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.921882 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5k2px" Nov 01 20:30:10.922673 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.922644 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm9p9\" (UniqueName: \"kubernetes.io/projected/a4342ddb-e06e-4d6d-9337-651349cf133f-kube-api-access-mm9p9\") pod \"multus-additional-cni-plugins-fmnrq\" (UID: \"a4342ddb-e06e-4d6d-9337-651349cf133f\") " pod="openshift-multus/multus-additional-cni-plugins-fmnrq" Nov 01 20:30:10.922883 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.922862 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5800567f-f0e1-4cf0-8338-75b09d2d1255-konnectivity-ca\") pod \"konnectivity-agent-2f2ct\" (UID: \"5800567f-f0e1-4cf0-8338-75b09d2d1255\") " pod="kube-system/konnectivity-agent-2f2ct" Nov 01 20:30:10.922949 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.922900 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-systemd-units\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:10.922949 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.922936 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vm67\" (UniqueName: \"kubernetes.io/projected/970d71e5-f861-4277-8fa5-6ec740418e51-kube-api-access-6vm67\") pod \"network-metrics-daemon-vxzfk\" (UID: \"970d71e5-f861-4277-8fa5-6ec740418e51\") " pod="openshift-multus/network-metrics-daemon-vxzfk" Nov 01 20:30:10.923132 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.922972 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4342ddb-e06e-4d6d-9337-651349cf133f-system-cni-dir\") pod \"multus-additional-cni-plugins-fmnrq\" (UID: \"a4342ddb-e06e-4d6d-9337-651349cf133f\") " pod="openshift-multus/multus-additional-cni-plugins-fmnrq" Nov 01 20:30:10.923132 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.923022 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-run-openvswitch\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:10.923132 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.923054 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-node-log\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:10.923132 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.923083 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e8ad39cd-9b6a-464c-aa75-1bfc54b23891-serviceca\") pod \"node-ca-bztv7\" (UID: \"e8ad39cd-9b6a-464c-aa75-1bfc54b23891\") " pod="openshift-image-registry/node-ca-bztv7" Nov 01 20:30:10.923132 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.923114 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-host-cni-bin\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:10.923374 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.923144 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-etc-modprobe-d\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:10.923374 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.923177 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-var-lib-kubelet\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:10.923374 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.923206 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/90e25e61-9761-4042-9dd4-88e1a6f8aa3c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ljb94\" (UID: \"90e25e61-9761-4042-9dd4-88e1a6f8aa3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb94" Nov 01 20:30:10.923374 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.923232 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsbqz\" (UniqueName: \"kubernetes.io/projected/90e25e61-9761-4042-9dd4-88e1a6f8aa3c-kube-api-access-gsbqz\") pod \"aws-ebs-csi-driver-node-ljb94\" (UID: \"90e25e61-9761-4042-9dd4-88e1a6f8aa3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb94" Nov 01 20:30:10.923374 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.923263 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-etc-systemd\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:10.923374 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.923293 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a4342ddb-e06e-4d6d-9337-651349cf133f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fmnrq\" (UID: \"a4342ddb-e06e-4d6d-9337-651349cf133f\") " pod="openshift-multus/multus-additional-cni-plugins-fmnrq" Nov 01 20:30:10.923374 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.923333 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a4342ddb-e06e-4d6d-9337-651349cf133f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fmnrq\" (UID: \"a4342ddb-e06e-4d6d-9337-651349cf133f\") " pod="openshift-multus/multus-additional-cni-plugins-fmnrq" Nov 01 20:30:10.923374 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.923366 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjfv4\" (UniqueName: \"kubernetes.io/projected/04ec71b7-390e-4e40-abce-1c395a7a51ab-kube-api-access-jjfv4\") pod \"node-resolver-w6nn2\" (UID: \"04ec71b7-390e-4e40-abce-1c395a7a51ab\") " pod="openshift-dns/node-resolver-w6nn2" Nov 01 20:30:10.923743 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.923391 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-ovnkube-config\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:10.923743 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.923430 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/90e25e61-9761-4042-9dd4-88e1a6f8aa3c-socket-dir\") pod \"aws-ebs-csi-driver-node-ljb94\" (UID: \"90e25e61-9761-4042-9dd4-88e1a6f8aa3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb94" Nov 01 20:30:10.923743 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.923467 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:10.923743 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.923498 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khz9n\" (UniqueName: \"kubernetes.io/projected/e8ad39cd-9b6a-464c-aa75-1bfc54b23891-kube-api-access-khz9n\") pod \"node-ca-bztv7\" (UID: \"e8ad39cd-9b6a-464c-aa75-1bfc54b23891\") " pod="openshift-image-registry/node-ca-bztv7" Nov 01 20:30:10.923743 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.923530 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/04ec71b7-390e-4e40-abce-1c395a7a51ab-hosts-file\") pod \"node-resolver-w6nn2\" (UID: \"04ec71b7-390e-4e40-abce-1c395a7a51ab\") " pod="openshift-dns/node-resolver-w6nn2" Nov 01 20:30:10.923743 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.923554 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-host\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:10.923743 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.923583 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e8ad39cd-9b6a-464c-aa75-1bfc54b23891-host\") pod \"node-ca-bztv7\" (UID: \"e8ad39cd-9b6a-464c-aa75-1bfc54b23891\") " pod="openshift-image-registry/node-ca-bztv7" Nov 01 20:30:10.923743 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.923613 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-host-kubelet\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:10.923743 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.923645 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-env-overrides\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:10.923743 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.923677 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-etc-kubernetes\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:10.923743 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.923712 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a4342ddb-e06e-4d6d-9337-651349cf133f-cni-binary-copy\") pod \"multus-additional-cni-plugins-fmnrq\" (UID: \"a4342ddb-e06e-4d6d-9337-651349cf133f\") " pod="openshift-multus/multus-additional-cni-plugins-fmnrq" Nov 01 20:30:10.923743 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.923745 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a4342ddb-e06e-4d6d-9337-651349cf133f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fmnrq\" (UID: \"a4342ddb-e06e-4d6d-9337-651349cf133f\") " pod="openshift-multus/multus-additional-cni-plugins-fmnrq" Nov 01 20:30:10.924304 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.923776 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04ec71b7-390e-4e40-abce-1c395a7a51ab-tmp-dir\") pod \"node-resolver-w6nn2\" (UID: \"04ec71b7-390e-4e40-abce-1c395a7a51ab\") " pod="openshift-dns/node-resolver-w6nn2" Nov 01 20:30:10.924304 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.923808 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-host-run-ovn-kubernetes\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:10.924304 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.923839 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-lib-modules\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:10.924304 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.923865 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/970d71e5-f861-4277-8fa5-6ec740418e51-metrics-certs\") pod \"network-metrics-daemon-vxzfk\" (UID: \"970d71e5-f861-4277-8fa5-6ec740418e51\") " pod="openshift-multus/network-metrics-daemon-vxzfk" Nov 01 20:30:10.924304 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.923899 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-host-slash\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:10.924304 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.923929 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-host-cni-netd\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:10.924304 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.923958 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-etc-sysctl-d\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:10.924304 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.923992 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-var-lib-openvswitch\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:10.924304 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.924022 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-etc-openvswitch\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:10.924304 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.924047 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzsrc\" (UniqueName: \"kubernetes.io/projected/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-kube-api-access-qzsrc\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:10.924304 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.924147 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-run\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:10.924304 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.924179 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/90e25e61-9761-4042-9dd4-88e1a6f8aa3c-device-dir\") pod \"aws-ebs-csi-driver-node-ljb94\" (UID: \"90e25e61-9761-4042-9dd4-88e1a6f8aa3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb94" Nov 01 20:30:10.924304 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.924212 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-run-ovn\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:10.924304 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.924251 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-log-socket\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:10.924304 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.924280 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-etc-sysctl-conf\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:10.924304 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.924311 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5800567f-f0e1-4cf0-8338-75b09d2d1255-agent-certs\") pod \"konnectivity-agent-2f2ct\" (UID: \"5800567f-f0e1-4cf0-8338-75b09d2d1255\") " pod="kube-system/konnectivity-agent-2f2ct" Nov 01 20:30:10.925024 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.924343 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-ovn-node-metrics-cert\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:10.925024 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.924373 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-sys\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:10.925024 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.924403 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fce64bd-a955-4e2c-8844-a25eaeb73862-tmp\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:10.925024 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.924428 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/90e25e61-9761-4042-9dd4-88e1a6f8aa3c-sys-fs\") pod \"aws-ebs-csi-driver-node-ljb94\" (UID: \"90e25e61-9761-4042-9dd4-88e1a6f8aa3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb94" Nov 01 20:30:10.925024 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.924466 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-run-systemd\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:10.925024 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.924497 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3fce64bd-a955-4e2c-8844-a25eaeb73862-etc-tuned\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:10.925024 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.924529 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a4342ddb-e06e-4d6d-9337-651349cf133f-cnibin\") pod \"multus-additional-cni-plugins-fmnrq\" (UID: \"a4342ddb-e06e-4d6d-9337-651349cf133f\") " pod="openshift-multus/multus-additional-cni-plugins-fmnrq" Nov 01 20:30:10.925024 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.924560 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/90e25e61-9761-4042-9dd4-88e1a6f8aa3c-registration-dir\") pod \"aws-ebs-csi-driver-node-ljb94\" (UID: \"90e25e61-9761-4042-9dd4-88e1a6f8aa3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb94" Nov 01 20:30:10.925024 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.924587 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-etc-sysconfig\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:10.925024 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.924618 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5pn9\" (UniqueName: \"kubernetes.io/projected/3fce64bd-a955-4e2c-8844-a25eaeb73862-kube-api-access-d5pn9\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:10.925024 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.924653 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-host-run-netns\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:10.925024 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.924687 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-ovnkube-script-lib\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:10.925024 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.924715 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/90e25e61-9761-4042-9dd4-88e1a6f8aa3c-etc-selinux\") pod \"aws-ebs-csi-driver-node-ljb94\" (UID: \"90e25e61-9761-4042-9dd4-88e1a6f8aa3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb94" Nov 01 20:30:10.952426 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.952400 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 01 20:30:10.952426 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.952406 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 01 20:30:10.952635 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.952447 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 01 20:30:10.952635 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.952475 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ggf7k\"" Nov 01 20:30:10.953460 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.953441 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 01 20:30:10.953560 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.953533 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 01 20:30:10.953560 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.953548 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pjdf7\"" Nov 01 20:30:10.956202 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.956165 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 01 20:30:10.956335 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.956246 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 01 20:30:10.956465 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.956426 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wk8gn\"" Nov 01 20:30:10.956565 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.956535 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 01 20:30:10.956617 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.956541 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 01 20:30:10.956617 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.956626 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 01 20:30:10.956806 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.956790 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 01 20:30:10.958293 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.958276 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6bdld\"" Nov 01 20:30:10.958379 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.958296 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 01 20:30:10.966591 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.966547 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-35.us-west-1.compute.internal" event={"ID":"f192393bd55f8e34f5db37f9db86763a","Type":"ContainerStarted","Data":"6e9ed6520b34ab71b0c03cc82e133742536fd7ed40a5a40364132cc19eef6fbc"} Nov 01 20:30:10.967728 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.967707 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-35.us-west-1.compute.internal" event={"ID":"91ec11d0905cc00deef779d06bc93fe3","Type":"ContainerStarted","Data":"7fc4de5f427b94b5aeece9302a62a6fade2769408cde35df2e0058c848f3b483"} Nov 01 20:30:10.980986 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.980956 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 01 20:30:10.981227 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.981200 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 01 20:30:10.981277 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.981264 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 01 20:30:10.981525 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.981508 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-j72sw\"" Nov 01 20:30:10.981613 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.981606 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 01 20:30:10.981852 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.981834 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 01 20:30:10.981948 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.981896 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 01 20:30:10.981948 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.981924 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 01 20:30:10.982094 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.982057 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-g9xjd\"" Nov 01 20:30:10.982094 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.982059 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 01 20:30:10.982094 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.982084 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-r4cnd\"" Nov 01 20:30:10.982218 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.982134 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5cgzw\"" Nov 01 20:30:10.982267 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.982227 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 01 20:30:10.982267 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.982241 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-h288r\"" Nov 01 20:30:10.982412 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.982394 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 01 20:30:10.982476 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.982443 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 01 20:30:10.982476 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.982462 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 01 20:30:10.982476 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.982467 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 01 20:30:10.982601 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.982491 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 01 20:30:10.982601 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:10.982504 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 01 20:30:11.012300 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.012273 2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 01 20:30:11.025515 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.025485 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:11.025628 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.025521 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khz9n\" (UniqueName: \"kubernetes.io/projected/e8ad39cd-9b6a-464c-aa75-1bfc54b23891-kube-api-access-khz9n\") pod \"node-ca-bztv7\" (UID: \"e8ad39cd-9b6a-464c-aa75-1bfc54b23891\") " pod="openshift-image-registry/node-ca-bztv7" Nov 01 20:30:11.025628 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.025545 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/04ec71b7-390e-4e40-abce-1c395a7a51ab-hosts-file\") pod \"node-resolver-w6nn2\" (UID: \"04ec71b7-390e-4e40-abce-1c395a7a51ab\") " pod="openshift-dns/node-resolver-w6nn2" Nov 01 20:30:11.025628 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.025563 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-host\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:11.025628 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.025582 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6b4f17a0-efd1-4024-a201-4078069b04e7-kubelet-config\") pod \"global-pull-secret-syncer-xh6zg\" (UID: \"6b4f17a0-efd1-4024-a201-4078069b04e7\") " pod="kube-system/global-pull-secret-syncer-xh6zg" Nov 01 20:30:11.025628 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.025599 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px48q\" (UniqueName: \"kubernetes.io/projected/2288d739-5eb3-4186-a32e-187945267de1-kube-api-access-px48q\") pod \"iptables-alerter-w7qmz\" (UID: \"2288d739-5eb3-4186-a32e-187945267de1\") " pod="openshift-network-operator/iptables-alerter-w7qmz" Nov 01 20:30:11.025628 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.025614 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:11.025869 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.025662 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/04ec71b7-390e-4e40-abce-1c395a7a51ab-hosts-file\") pod \"node-resolver-w6nn2\" (UID: \"04ec71b7-390e-4e40-abce-1c395a7a51ab\") " pod="openshift-dns/node-resolver-w6nn2" Nov 01 20:30:11.025869 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.025685 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-multus-cni-dir\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:11.025869 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.025663 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-host\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:11.025869 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.025717 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-cni-binary-copy\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:11.025869 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.025758 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e8ad39cd-9b6a-464c-aa75-1bfc54b23891-host\") pod \"node-ca-bztv7\" (UID: \"e8ad39cd-9b6a-464c-aa75-1bfc54b23891\") " pod="openshift-image-registry/node-ca-bztv7" Nov 01 20:30:11.025869 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.025809 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-host-kubelet\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:11.025869 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.025818 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e8ad39cd-9b6a-464c-aa75-1bfc54b23891-host\") pod \"node-ca-bztv7\" (UID: \"e8ad39cd-9b6a-464c-aa75-1bfc54b23891\") " pod="openshift-image-registry/node-ca-bztv7" Nov 01 20:30:11.025869 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.025840 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-env-overrides\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:11.025869 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.025858 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-etc-kubernetes\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:11.025869 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.025874 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a4342ddb-e06e-4d6d-9337-651349cf133f-cni-binary-copy\") pod \"multus-additional-cni-plugins-fmnrq\" (UID: \"a4342ddb-e06e-4d6d-9337-651349cf133f\") " pod="openshift-multus/multus-additional-cni-plugins-fmnrq" Nov 01 20:30:11.026349 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.025891 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a4342ddb-e06e-4d6d-9337-651349cf133f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fmnrq\" (UID: \"a4342ddb-e06e-4d6d-9337-651349cf133f\") " pod="openshift-multus/multus-additional-cni-plugins-fmnrq" Nov 01 20:30:11.026349 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.025909 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-etc-kubernetes\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:11.026349 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.025839 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-host-kubelet\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:11.026349 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.025932 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgl7f\" (UniqueName: \"kubernetes.io/projected/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-kube-api-access-vgl7f\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:11.026349 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.025968 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-etc-kubernetes\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:11.026349 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.026150 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04ec71b7-390e-4e40-abce-1c395a7a51ab-tmp-dir\") pod \"node-resolver-w6nn2\" (UID: \"04ec71b7-390e-4e40-abce-1c395a7a51ab\") " pod="openshift-dns/node-resolver-w6nn2" Nov 01 20:30:11.026349 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.026182 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-host-run-ovn-kubernetes\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:11.026349 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.026208 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-lib-modules\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:11.026349 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.026232 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/970d71e5-f861-4277-8fa5-6ec740418e51-metrics-certs\") pod \"network-metrics-daemon-vxzfk\" (UID: \"970d71e5-f861-4277-8fa5-6ec740418e51\") " pod="openshift-multus/network-metrics-daemon-vxzfk" Nov 01 20:30:11.026349 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.026255 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-host-run-ovn-kubernetes\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:11.026349 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.026260 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-system-cni-dir\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:11.026349 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.026308 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-host-slash\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:11.026349 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.026334 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-host-cni-netd\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:11.026918 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.026368 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-etc-sysctl-d\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:11.026918 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.026389 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-lib-modules\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:11.026918 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.026399 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btdtr\" (UniqueName: \"kubernetes.io/projected/9ee67cbd-8115-4c48-b5cd-5299ed284bce-kube-api-access-btdtr\") pod \"network-check-target-ghh5z\" (UID: \"9ee67cbd-8115-4c48-b5cd-5299ed284bce\") " pod="openshift-network-diagnostics/network-check-target-ghh5z" Nov 01 20:30:11.026918 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.026453 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-host-var-lib-cni-bin\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:11.026918 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.026486 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-host-var-lib-cni-multus\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:11.026918 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.026547 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04ec71b7-390e-4e40-abce-1c395a7a51ab-tmp-dir\") pod \"node-resolver-w6nn2\" (UID: \"04ec71b7-390e-4e40-abce-1c395a7a51ab\") " pod="openshift-dns/node-resolver-w6nn2" Nov 01 20:30:11.026918 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:11.026580 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 20:30:11.026918 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.026622 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-host-var-lib-kubelet\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:11.026918 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.026649 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-etc-sysctl-d\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:11.026918 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.026687 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-host-slash\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:11.026918 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.026698 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-var-lib-openvswitch\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:11.026918 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.026732 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-host-cni-netd\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:11.026918 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.026740 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-env-overrides\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:11.026918 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:11.026757 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/970d71e5-f861-4277-8fa5-6ec740418e51-metrics-certs podName:970d71e5-f861-4277-8fa5-6ec740418e51 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:11.526723731 +0000 UTC m=+3.096211317 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/970d71e5-f861-4277-8fa5-6ec740418e51-metrics-certs") pod "network-metrics-daemon-vxzfk" (UID: "970d71e5-f861-4277-8fa5-6ec740418e51") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 20:30:11.026918 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.026785 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-etc-openvswitch\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:11.026918 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.026808 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a4342ddb-e06e-4d6d-9337-651349cf133f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fmnrq\" (UID: \"a4342ddb-e06e-4d6d-9337-651349cf133f\") " pod="openshift-multus/multus-additional-cni-plugins-fmnrq" Nov 01 20:30:11.026918 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.026810 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-var-lib-openvswitch\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:11.027809 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.026823 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a4342ddb-e06e-4d6d-9337-651349cf133f-cni-binary-copy\") pod \"multus-additional-cni-plugins-fmnrq\" (UID: \"a4342ddb-e06e-4d6d-9337-651349cf133f\") " pod="openshift-multus/multus-additional-cni-plugins-fmnrq" Nov 01 20:30:11.027809 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.026845 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzsrc\" (UniqueName: \"kubernetes.io/projected/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-kube-api-access-qzsrc\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:11.027809 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.026867 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-run\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:11.027809 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.026897 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-etc-openvswitch\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:11.027809 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.026952 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-run\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:11.027809 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.026952 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/90e25e61-9761-4042-9dd4-88e1a6f8aa3c-device-dir\") pod \"aws-ebs-csi-driver-node-ljb94\" (UID: \"90e25e61-9761-4042-9dd4-88e1a6f8aa3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb94" Nov 01 20:30:11.027809 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.026988 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/90e25e61-9761-4042-9dd4-88e1a6f8aa3c-device-dir\") pod \"aws-ebs-csi-driver-node-ljb94\" (UID: \"90e25e61-9761-4042-9dd4-88e1a6f8aa3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb94" Nov 01 20:30:11.027809 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.027025 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-run-ovn\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:11.027809 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.027057 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-log-socket\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:11.027809 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.027076 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-run-ovn\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:11.027809 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.027095 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-log-socket\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:11.027809 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.027126 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-etc-sysctl-conf\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:11.027809 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.027148 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5800567f-f0e1-4cf0-8338-75b09d2d1255-agent-certs\") pod \"konnectivity-agent-2f2ct\" (UID: \"5800567f-f0e1-4cf0-8338-75b09d2d1255\") " pod="kube-system/konnectivity-agent-2f2ct" Nov 01 20:30:11.027809 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.027264 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b4f17a0-efd1-4024-a201-4078069b04e7-original-pull-secret\") pod \"global-pull-secret-syncer-xh6zg\" (UID: \"6b4f17a0-efd1-4024-a201-4078069b04e7\") " pod="kube-system/global-pull-secret-syncer-xh6zg" Nov 01 20:30:11.027809 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.027300 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-host-run-k8s-cni-cncf-io\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:11.027809 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.027322 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-hostroot\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:11.027809 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.027338 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-etc-sysctl-conf\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:11.028642 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.027347 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-ovn-node-metrics-cert\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:11.028642 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.027374 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-sys\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:11.028642 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.027396 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fce64bd-a955-4e2c-8844-a25eaeb73862-tmp\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:11.028642 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.027395 2580 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 01 20:30:11.028642 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.027443 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-sys\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:11.028642 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.027457 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/90e25e61-9761-4042-9dd4-88e1a6f8aa3c-sys-fs\") pod \"aws-ebs-csi-driver-node-ljb94\" (UID: \"90e25e61-9761-4042-9dd4-88e1a6f8aa3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb94" Nov 01 20:30:11.028642 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.027513 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-run-systemd\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:11.028642 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.027543 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/90e25e61-9761-4042-9dd4-88e1a6f8aa3c-sys-fs\") pod \"aws-ebs-csi-driver-node-ljb94\" (UID: \"90e25e61-9761-4042-9dd4-88e1a6f8aa3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb94" Nov 01 20:30:11.028642 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.027557 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3fce64bd-a955-4e2c-8844-a25eaeb73862-etc-tuned\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:11.028642 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.027608 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a4342ddb-e06e-4d6d-9337-651349cf133f-cnibin\") pod \"multus-additional-cni-plugins-fmnrq\" (UID: \"a4342ddb-e06e-4d6d-9337-651349cf133f\") " pod="openshift-multus/multus-additional-cni-plugins-fmnrq" Nov 01 20:30:11.028642 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.027615 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-run-systemd\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:11.028642 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.027637 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/90e25e61-9761-4042-9dd4-88e1a6f8aa3c-registration-dir\") pod \"aws-ebs-csi-driver-node-ljb94\" (UID: \"90e25e61-9761-4042-9dd4-88e1a6f8aa3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb94" Nov 01 20:30:11.028642 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.027662 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a4342ddb-e06e-4d6d-9337-651349cf133f-cnibin\") pod \"multus-additional-cni-plugins-fmnrq\" (UID: \"a4342ddb-e06e-4d6d-9337-651349cf133f\") " pod="openshift-multus/multus-additional-cni-plugins-fmnrq" Nov 01 20:30:11.028642 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.027665 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-etc-sysconfig\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:11.028642 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.027722 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5pn9\" (UniqueName: \"kubernetes.io/projected/3fce64bd-a955-4e2c-8844-a25eaeb73862-kube-api-access-d5pn9\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:11.028642 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.027730 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-etc-sysconfig\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:11.028642 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.027752 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-cnibin\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:11.028642 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.027767 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/90e25e61-9761-4042-9dd4-88e1a6f8aa3c-registration-dir\") pod \"aws-ebs-csi-driver-node-ljb94\" (UID: \"90e25e61-9761-4042-9dd4-88e1a6f8aa3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb94" Nov 01 20:30:11.029632 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.027778 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-host-run-multus-certs\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:11.029632 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.027842 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-host-run-netns\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:11.029632 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.028328 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-ovnkube-script-lib\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:11.029632 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.028359 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/90e25e61-9761-4042-9dd4-88e1a6f8aa3c-etc-selinux\") pod \"aws-ebs-csi-driver-node-ljb94\" (UID: \"90e25e61-9761-4042-9dd4-88e1a6f8aa3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb94" Nov 01 20:30:11.029632 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.028389 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-multus-conf-dir\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:11.029632 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.028419 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm9p9\" (UniqueName: \"kubernetes.io/projected/a4342ddb-e06e-4d6d-9337-651349cf133f-kube-api-access-mm9p9\") pod \"multus-additional-cni-plugins-fmnrq\" (UID: \"a4342ddb-e06e-4d6d-9337-651349cf133f\") " pod="openshift-multus/multus-additional-cni-plugins-fmnrq" Nov 01 20:30:11.029632 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.028437 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5800567f-f0e1-4cf0-8338-75b09d2d1255-konnectivity-ca\") pod \"konnectivity-agent-2f2ct\" (UID: \"5800567f-f0e1-4cf0-8338-75b09d2d1255\") " pod="kube-system/konnectivity-agent-2f2ct" Nov 01 20:30:11.029632 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.028453 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2288d739-5eb3-4186-a32e-187945267de1-host-slash\") pod \"iptables-alerter-w7qmz\" (UID: \"2288d739-5eb3-4186-a32e-187945267de1\") " pod="openshift-network-operator/iptables-alerter-w7qmz" Nov 01 20:30:11.029632 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.028468 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-systemd-units\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:11.029632 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.028491 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6b4f17a0-efd1-4024-a201-4078069b04e7-dbus\") pod \"global-pull-secret-syncer-xh6zg\" (UID: \"6b4f17a0-efd1-4024-a201-4078069b04e7\") " pod="kube-system/global-pull-secret-syncer-xh6zg" Nov 01 20:30:11.029632 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.028485 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/90e25e61-9761-4042-9dd4-88e1a6f8aa3c-etc-selinux\") pod \"aws-ebs-csi-driver-node-ljb94\" (UID: \"90e25e61-9761-4042-9dd4-88e1a6f8aa3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb94" Nov 01 20:30:11.029632 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.028520 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-host-run-netns\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:11.029632 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.028545 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-multus-daemon-config\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:11.029632 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.028572 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vm67\" (UniqueName: \"kubernetes.io/projected/970d71e5-f861-4277-8fa5-6ec740418e51-kube-api-access-6vm67\") pod \"network-metrics-daemon-vxzfk\" (UID: \"970d71e5-f861-4277-8fa5-6ec740418e51\") " pod="openshift-multus/network-metrics-daemon-vxzfk" Nov 01 20:30:11.029632 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.028601 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4342ddb-e06e-4d6d-9337-651349cf133f-system-cni-dir\") pod \"multus-additional-cni-plugins-fmnrq\" (UID: \"a4342ddb-e06e-4d6d-9337-651349cf133f\") " pod="openshift-multus/multus-additional-cni-plugins-fmnrq" Nov 01 20:30:11.029632 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.028628 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-run-openvswitch\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:11.029632 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.028657 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-node-log\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:11.030490 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.028706 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e8ad39cd-9b6a-464c-aa75-1bfc54b23891-serviceca\") pod \"node-ca-bztv7\" (UID: \"e8ad39cd-9b6a-464c-aa75-1bfc54b23891\") " pod="openshift-image-registry/node-ca-bztv7" Nov 01 20:30:11.030490 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.028731 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-host-cni-bin\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:11.030490 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.028792 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-etc-modprobe-d\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:11.030490 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.028793 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-run-openvswitch\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:11.030490 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.028814 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-host-cni-bin\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:11.030490 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.028861 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-node-log\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:11.030490 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.029090 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-systemd-units\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:11.030490 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.029129 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-var-lib-kubelet\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:11.030490 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.029157 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/90e25e61-9761-4042-9dd4-88e1a6f8aa3c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ljb94\" (UID: \"90e25e61-9761-4042-9dd4-88e1a6f8aa3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb94" Nov 01 20:30:11.030490 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.029197 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4342ddb-e06e-4d6d-9337-651349cf133f-system-cni-dir\") pod \"multus-additional-cni-plugins-fmnrq\" (UID: \"a4342ddb-e06e-4d6d-9337-651349cf133f\") " pod="openshift-multus/multus-additional-cni-plugins-fmnrq" Nov 01 20:30:11.030490 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.029243 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsbqz\" (UniqueName: \"kubernetes.io/projected/90e25e61-9761-4042-9dd4-88e1a6f8aa3c-kube-api-access-gsbqz\") pod \"aws-ebs-csi-driver-node-ljb94\" (UID: \"90e25e61-9761-4042-9dd4-88e1a6f8aa3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb94" Nov 01 20:30:11.030490 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.029273 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-multus-socket-dir-parent\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:11.030490 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.029299 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-etc-systemd\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:11.030490 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.029326 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a4342ddb-e06e-4d6d-9337-651349cf133f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fmnrq\" (UID: \"a4342ddb-e06e-4d6d-9337-651349cf133f\") " pod="openshift-multus/multus-additional-cni-plugins-fmnrq" Nov 01 20:30:11.030490 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.029390 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a4342ddb-e06e-4d6d-9337-651349cf133f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fmnrq\" (UID: \"a4342ddb-e06e-4d6d-9337-651349cf133f\") " pod="openshift-multus/multus-additional-cni-plugins-fmnrq" Nov 01 20:30:11.030490 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.029425 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2288d739-5eb3-4186-a32e-187945267de1-iptables-alerter-script\") pod \"iptables-alerter-w7qmz\" (UID: \"2288d739-5eb3-4186-a32e-187945267de1\") " pod="openshift-network-operator/iptables-alerter-w7qmz" Nov 01 20:30:11.030490 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.029472 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjfv4\" (UniqueName: \"kubernetes.io/projected/04ec71b7-390e-4e40-abce-1c395a7a51ab-kube-api-access-jjfv4\") pod \"node-resolver-w6nn2\" (UID: \"04ec71b7-390e-4e40-abce-1c395a7a51ab\") " pod="openshift-dns/node-resolver-w6nn2" Nov 01 20:30:11.031175 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.029500 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-ovnkube-config\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:11.031175 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.029525 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/90e25e61-9761-4042-9dd4-88e1a6f8aa3c-socket-dir\") pod \"aws-ebs-csi-driver-node-ljb94\" (UID: \"90e25e61-9761-4042-9dd4-88e1a6f8aa3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb94" Nov 01 20:30:11.031175 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.029540 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e8ad39cd-9b6a-464c-aa75-1bfc54b23891-serviceca\") pod \"node-ca-bztv7\" (UID: \"e8ad39cd-9b6a-464c-aa75-1bfc54b23891\") " pod="openshift-image-registry/node-ca-bztv7" Nov 01 20:30:11.031175 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.029548 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5800567f-f0e1-4cf0-8338-75b09d2d1255-konnectivity-ca\") pod \"konnectivity-agent-2f2ct\" (UID: \"5800567f-f0e1-4cf0-8338-75b09d2d1255\") " pod="kube-system/konnectivity-agent-2f2ct" Nov 01 20:30:11.031175 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.029700 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/90e25e61-9761-4042-9dd4-88e1a6f8aa3c-socket-dir\") pod \"aws-ebs-csi-driver-node-ljb94\" (UID: \"90e25e61-9761-4042-9dd4-88e1a6f8aa3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb94" Nov 01 20:30:11.031175 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.029709 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a4342ddb-e06e-4d6d-9337-651349cf133f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fmnrq\" (UID: \"a4342ddb-e06e-4d6d-9337-651349cf133f\") " pod="openshift-multus/multus-additional-cni-plugins-fmnrq" Nov 01 20:30:11.031175 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.029768 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/90e25e61-9761-4042-9dd4-88e1a6f8aa3c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ljb94\" (UID: \"90e25e61-9761-4042-9dd4-88e1a6f8aa3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb94" Nov 01 20:30:11.031175 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.029769 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-etc-modprobe-d\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:11.031175 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.029828 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-etc-systemd\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:11.031175 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.029873 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-var-lib-kubelet\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:11.031175 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.030077 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a4342ddb-e06e-4d6d-9337-651349cf133f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fmnrq\" (UID: \"a4342ddb-e06e-4d6d-9337-651349cf133f\") " pod="openshift-multus/multus-additional-cni-plugins-fmnrq" Nov 01 20:30:11.031175 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.030507 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-ovnkube-config\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:11.031175 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.030551 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-host-run-netns\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:11.031716 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.031268 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-ovnkube-script-lib\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:11.031776 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.031722 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fce64bd-a955-4e2c-8844-a25eaeb73862-tmp\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:11.031776 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.031741 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3fce64bd-a955-4e2c-8844-a25eaeb73862-etc-tuned\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:11.031881 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.031839 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-ovn-node-metrics-cert\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:11.032038 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.032011 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5800567f-f0e1-4cf0-8338-75b09d2d1255-agent-certs\") pod \"konnectivity-agent-2f2ct\" (UID: \"5800567f-f0e1-4cf0-8338-75b09d2d1255\") " pod="kube-system/konnectivity-agent-2f2ct" Nov 01 20:30:11.052418 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.052394 2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 20:30:11.055234 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.055214 2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 01 20:30:11.055321 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.055254 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2gwlq" Nov 01 20:30:11.088041 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.087997 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khz9n\" (UniqueName: \"kubernetes.io/projected/e8ad39cd-9b6a-464c-aa75-1bfc54b23891-kube-api-access-khz9n\") pod \"node-ca-bztv7\" (UID: \"e8ad39cd-9b6a-464c-aa75-1bfc54b23891\") " pod="openshift-image-registry/node-ca-bztv7" Nov 01 20:30:11.088580 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.088560 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzsrc\" (UniqueName: \"kubernetes.io/projected/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-kube-api-access-qzsrc\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:11.089570 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.089549 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5pn9\" (UniqueName: \"kubernetes.io/projected/3fce64bd-a955-4e2c-8844-a25eaeb73862-kube-api-access-d5pn9\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:11.090410 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.090385 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm9p9\" (UniqueName: \"kubernetes.io/projected/a4342ddb-e06e-4d6d-9337-651349cf133f-kube-api-access-mm9p9\") pod \"multus-additional-cni-plugins-fmnrq\" (UID: \"a4342ddb-e06e-4d6d-9337-651349cf133f\") " pod="openshift-multus/multus-additional-cni-plugins-fmnrq" Nov 01 20:30:11.091046 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.091009 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vm67\" (UniqueName: \"kubernetes.io/projected/970d71e5-f861-4277-8fa5-6ec740418e51-kube-api-access-6vm67\") pod \"network-metrics-daemon-vxzfk\" (UID: \"970d71e5-f861-4277-8fa5-6ec740418e51\") " pod="openshift-multus/network-metrics-daemon-vxzfk" Nov 01 20:30:11.091341 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.091320 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsbqz\" (UniqueName: \"kubernetes.io/projected/90e25e61-9761-4042-9dd4-88e1a6f8aa3c-kube-api-access-gsbqz\") pod \"aws-ebs-csi-driver-node-ljb94\" (UID: \"90e25e61-9761-4042-9dd4-88e1a6f8aa3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb94" Nov 01 20:30:11.091421 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.091394 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjfv4\" (UniqueName: \"kubernetes.io/projected/04ec71b7-390e-4e40-abce-1c395a7a51ab-kube-api-access-jjfv4\") pod \"node-resolver-w6nn2\" (UID: \"04ec71b7-390e-4e40-abce-1c395a7a51ab\") " pod="openshift-dns/node-resolver-w6nn2" Nov 01 20:30:11.130598 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.130552 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2288d739-5eb3-4186-a32e-187945267de1-iptables-alerter-script\") pod \"iptables-alerter-w7qmz\" (UID: \"2288d739-5eb3-4186-a32e-187945267de1\") " pod="openshift-network-operator/iptables-alerter-w7qmz" Nov 01 20:30:11.130736 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.130610 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6b4f17a0-efd1-4024-a201-4078069b04e7-kubelet-config\") pod \"global-pull-secret-syncer-xh6zg\" (UID: \"6b4f17a0-efd1-4024-a201-4078069b04e7\") " pod="kube-system/global-pull-secret-syncer-xh6zg" Nov 01 20:30:11.130736 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.130643 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px48q\" (UniqueName: \"kubernetes.io/projected/2288d739-5eb3-4186-a32e-187945267de1-kube-api-access-px48q\") pod \"iptables-alerter-w7qmz\" (UID: \"2288d739-5eb3-4186-a32e-187945267de1\") " pod="openshift-network-operator/iptables-alerter-w7qmz" Nov 01 20:30:11.130736 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.130676 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-multus-cni-dir\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:11.130736 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.130701 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-cni-binary-copy\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:11.130736 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.130733 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-etc-kubernetes\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:11.130944 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.130735 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6b4f17a0-efd1-4024-a201-4078069b04e7-kubelet-config\") pod \"global-pull-secret-syncer-xh6zg\" (UID: \"6b4f17a0-efd1-4024-a201-4078069b04e7\") " pod="kube-system/global-pull-secret-syncer-xh6zg" Nov 01 20:30:11.130944 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.130772 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-etc-kubernetes\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:11.131207 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.130882 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-multus-cni-dir\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:11.131320 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.131218 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgl7f\" (UniqueName: \"kubernetes.io/projected/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-kube-api-access-vgl7f\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:11.131320 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.131314 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-system-cni-dir\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:11.131439 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.131353 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btdtr\" (UniqueName: \"kubernetes.io/projected/9ee67cbd-8115-4c48-b5cd-5299ed284bce-kube-api-access-btdtr\") pod \"network-check-target-ghh5z\" (UID: \"9ee67cbd-8115-4c48-b5cd-5299ed284bce\") " pod="openshift-network-diagnostics/network-check-target-ghh5z" Nov 01 20:30:11.131439 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.131388 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-host-var-lib-cni-bin\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:11.131439 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.131420 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-host-var-lib-cni-multus\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:11.131586 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.131463 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-host-var-lib-kubelet\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:11.131586 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.131507 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b4f17a0-efd1-4024-a201-4078069b04e7-original-pull-secret\") pod \"global-pull-secret-syncer-xh6zg\" (UID: \"6b4f17a0-efd1-4024-a201-4078069b04e7\") " pod="kube-system/global-pull-secret-syncer-xh6zg" Nov 01 20:30:11.131586 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.131514 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-system-cni-dir\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:11.131586 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.131543 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-host-run-k8s-cni-cncf-io\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:11.131586 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.131555 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-host-var-lib-cni-bin\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:11.131848 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.131642 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-host-run-k8s-cni-cncf-io\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:11.131848 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.131730 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-host-var-lib-kubelet\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:11.131848 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:11.131794 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 20:30:11.131991 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:11.131873 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b4f17a0-efd1-4024-a201-4078069b04e7-original-pull-secret podName:6b4f17a0-efd1-4024-a201-4078069b04e7 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:11.631844868 +0000 UTC m=+3.201332458 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b4f17a0-efd1-4024-a201-4078069b04e7-original-pull-secret") pod "global-pull-secret-syncer-xh6zg" (UID: "6b4f17a0-efd1-4024-a201-4078069b04e7") : object "kube-system"/"original-pull-secret" not registered Nov 01 20:30:11.131991 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.131869 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-host-var-lib-cni-multus\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:11.131991 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.131885 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2288d739-5eb3-4186-a32e-187945267de1-iptables-alerter-script\") pod \"iptables-alerter-w7qmz\" (UID: \"2288d739-5eb3-4186-a32e-187945267de1\") " pod="openshift-network-operator/iptables-alerter-w7qmz" Nov 01 20:30:11.131991 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.131898 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-hostroot\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:11.131991 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.131945 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-hostroot\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:11.131991 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.131961 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-cnibin\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:11.131991 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.131921 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-cni-binary-copy\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:11.132376 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.131999 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-host-run-multus-certs\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:11.132376 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.132009 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-cnibin\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:11.132376 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.132046 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-host-run-multus-certs\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:11.132376 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.132099 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-multus-conf-dir\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:11.132376 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.132138 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2288d739-5eb3-4186-a32e-187945267de1-host-slash\") pod \"iptables-alerter-w7qmz\" (UID: \"2288d739-5eb3-4186-a32e-187945267de1\") " pod="openshift-network-operator/iptables-alerter-w7qmz" Nov 01 20:30:11.132376 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.132180 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6b4f17a0-efd1-4024-a201-4078069b04e7-dbus\") pod \"global-pull-secret-syncer-xh6zg\" (UID: \"6b4f17a0-efd1-4024-a201-4078069b04e7\") " pod="kube-system/global-pull-secret-syncer-xh6zg" Nov 01 20:30:11.132376 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.132207 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-host-run-netns\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:11.132376 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.132241 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-multus-daemon-config\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:11.132376 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.132290 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-multus-socket-dir-parent\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:11.132760 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.132388 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-multus-socket-dir-parent\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:11.132760 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.132492 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2288d739-5eb3-4186-a32e-187945267de1-host-slash\") pod \"iptables-alerter-w7qmz\" (UID: \"2288d739-5eb3-4186-a32e-187945267de1\") " pod="openshift-network-operator/iptables-alerter-w7qmz" Nov 01 20:30:11.132760 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.132555 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6b4f17a0-efd1-4024-a201-4078069b04e7-dbus\") pod \"global-pull-secret-syncer-xh6zg\" (UID: \"6b4f17a0-efd1-4024-a201-4078069b04e7\") " pod="kube-system/global-pull-secret-syncer-xh6zg" Nov 01 20:30:11.132760 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.132601 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-host-run-netns\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:11.134399 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.133108 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-multus-conf-dir\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:11.134399 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.133115 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-multus-daemon-config\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:11.190977 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:11.190941 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 20:30:11.190977 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:11.190976 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 20:30:11.191214 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:11.190992 2580 projected.go:196] Error preparing data for projected volume kube-api-access-btdtr for pod openshift-network-diagnostics/network-check-target-ghh5z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 20:30:11.191214 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:11.191095 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ee67cbd-8115-4c48-b5cd-5299ed284bce-kube-api-access-btdtr podName:9ee67cbd-8115-4c48-b5cd-5299ed284bce nodeName:}" failed. No retries permitted until 2026-11-01 20:30:11.691072372 +0000 UTC m=+3.260559963 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-btdtr" (UniqueName: "kubernetes.io/projected/9ee67cbd-8115-4c48-b5cd-5299ed284bce-kube-api-access-btdtr") pod "network-check-target-ghh5z" (UID: "9ee67cbd-8115-4c48-b5cd-5299ed284bce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 20:30:11.193018 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.192994 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px48q\" (UniqueName: \"kubernetes.io/projected/2288d739-5eb3-4186-a32e-187945267de1-kube-api-access-px48q\") pod \"iptables-alerter-w7qmz\" (UID: \"2288d739-5eb3-4186-a32e-187945267de1\") " pod="openshift-network-operator/iptables-alerter-w7qmz" Nov 01 20:30:11.193242 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.193223 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgl7f\" (UniqueName: \"kubernetes.io/projected/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-kube-api-access-vgl7f\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:11.206585 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.206555 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bztv7" Nov 01 20:30:11.212993 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.212973 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-w6nn2" Nov 01 20:30:11.217014 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:11.216986 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8ad39cd_9b6a_464c_aa75_1bfc54b23891.slice/crio-030f4716dcaaa8462c101a58de0015dbb55dca028de259c1fc7c3915330d1b7e WatchSource:0}: Error finding container 030f4716dcaaa8462c101a58de0015dbb55dca028de259c1fc7c3915330d1b7e: Status 404 returned error can't find the container with id 030f4716dcaaa8462c101a58de0015dbb55dca028de259c1fc7c3915330d1b7e Nov 01 20:30:11.222169 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:11.222144 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04ec71b7_390e_4e40_abce_1c395a7a51ab.slice/crio-5882d5d1fac98d98c7525baa5d1d7ea2c0071b79c1c9f344d61fd84c7f11bfaf WatchSource:0}: Error finding container 5882d5d1fac98d98c7525baa5d1d7ea2c0071b79c1c9f344d61fd84c7f11bfaf: Status 404 returned error can't find the container with id 5882d5d1fac98d98c7525baa5d1d7ea2c0071b79c1c9f344d61fd84c7f11bfaf Nov 01 20:30:11.233623 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.233601 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:11.238717 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.238698 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:11.243652 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:11.243619 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fce64bd_a955_4e2c_8844_a25eaeb73862.slice/crio-9844bfd7762554a3667f4db4086ff74120af96dddcf30fc796e0ac2cb13ead26 WatchSource:0}: Error finding container 9844bfd7762554a3667f4db4086ff74120af96dddcf30fc796e0ac2cb13ead26: Status 404 returned error can't find the container with id 9844bfd7762554a3667f4db4086ff74120af96dddcf30fc796e0ac2cb13ead26 Nov 01 20:30:11.248525 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:11.248498 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcad48e5e_da24_4aa6_bbb4_3f24c9fd4bfc.slice/crio-5160691e25c668734b4c024993d5bf4c3ed08d76e846e24ca73845637ad73521 WatchSource:0}: Error finding container 5160691e25c668734b4c024993d5bf4c3ed08d76e846e24ca73845637ad73521: Status 404 returned error can't find the container with id 5160691e25c668734b4c024993d5bf4c3ed08d76e846e24ca73845637ad73521 Nov 01 20:30:11.251908 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.251891 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2f2ct" Nov 01 20:30:11.255007 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.254987 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb94" Nov 01 20:30:11.259203 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.259185 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-w7qmz" Nov 01 20:30:11.261440 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.261420 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5k2px" Nov 01 20:30:11.261904 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:11.261883 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5800567f_f0e1_4cf0_8338_75b09d2d1255.slice/crio-df6c54825fb1d72061008ce2c740924d1b7e34f39b8db1676980e72afd1a970e WatchSource:0}: Error finding container df6c54825fb1d72061008ce2c740924d1b7e34f39b8db1676980e72afd1a970e: Status 404 returned error can't find the container with id df6c54825fb1d72061008ce2c740924d1b7e34f39b8db1676980e72afd1a970e Nov 01 20:30:11.263893 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:11.263871 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90e25e61_9761_4042_9dd4_88e1a6f8aa3c.slice/crio-8d5ba769477249f45bb9343ec9a64905df58d817e34e232a4b8b906025665541 WatchSource:0}: Error finding container 8d5ba769477249f45bb9343ec9a64905df58d817e34e232a4b8b906025665541: Status 404 returned error can't find the container with id 8d5ba769477249f45bb9343ec9a64905df58d817e34e232a4b8b906025665541 Nov 01 20:30:11.267710 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.267629 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fmnrq" Nov 01 20:30:11.270199 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:11.270179 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2288d739_5eb3_4186_a32e_187945267de1.slice/crio-dcd1501c6ab24fec2ddbec2a6a81381d321b3ca29979d2de8e25807600c2199d WatchSource:0}: Error finding container dcd1501c6ab24fec2ddbec2a6a81381d321b3ca29979d2de8e25807600c2199d: Status 404 returned error can't find the container with id dcd1501c6ab24fec2ddbec2a6a81381d321b3ca29979d2de8e25807600c2199d Nov 01 20:30:11.273241 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:11.273215 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcc5ee5c_08d8_49ba_9d7b_04e7a684e427.slice/crio-f35238c9dff3501e77884444e067b5978d544dd583eaace87752ad74535b250a WatchSource:0}: Error finding container f35238c9dff3501e77884444e067b5978d544dd583eaace87752ad74535b250a: Status 404 returned error can't find the container with id f35238c9dff3501e77884444e067b5978d544dd583eaace87752ad74535b250a Nov 01 20:30:11.279219 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:11.279199 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4342ddb_e06e_4d6d_9337_651349cf133f.slice/crio-ee503b738885a5321cca7fa7f3c6defd4071ab42689f058bea78da5918ac14e7 WatchSource:0}: Error finding container ee503b738885a5321cca7fa7f3c6defd4071ab42689f058bea78da5918ac14e7: Status 404 returned error can't find the container with id ee503b738885a5321cca7fa7f3c6defd4071ab42689f058bea78da5918ac14e7 Nov 01 20:30:11.536070 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.535967 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/970d71e5-f861-4277-8fa5-6ec740418e51-metrics-certs\") pod \"network-metrics-daemon-vxzfk\" (UID: \"970d71e5-f861-4277-8fa5-6ec740418e51\") " pod="openshift-multus/network-metrics-daemon-vxzfk" Nov 01 20:30:11.536222 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:11.536138 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 20:30:11.536277 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:11.536239 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/970d71e5-f861-4277-8fa5-6ec740418e51-metrics-certs podName:970d71e5-f861-4277-8fa5-6ec740418e51 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:12.536214979 +0000 UTC m=+4.105702565 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/970d71e5-f861-4277-8fa5-6ec740418e51-metrics-certs") pod "network-metrics-daemon-vxzfk" (UID: "970d71e5-f861-4277-8fa5-6ec740418e51") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 20:30:11.637316 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.637042 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b4f17a0-efd1-4024-a201-4078069b04e7-original-pull-secret\") pod \"global-pull-secret-syncer-xh6zg\" (UID: \"6b4f17a0-efd1-4024-a201-4078069b04e7\") " pod="kube-system/global-pull-secret-syncer-xh6zg" Nov 01 20:30:11.637316 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:11.637203 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 20:30:11.637316 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:11.637276 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b4f17a0-efd1-4024-a201-4078069b04e7-original-pull-secret podName:6b4f17a0-efd1-4024-a201-4078069b04e7 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:12.637255294 +0000 UTC m=+4.206742881 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b4f17a0-efd1-4024-a201-4078069b04e7-original-pull-secret") pod "global-pull-secret-syncer-xh6zg" (UID: "6b4f17a0-efd1-4024-a201-4078069b04e7") : object "kube-system"/"original-pull-secret" not registered Nov 01 20:30:11.737591 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.737548 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btdtr\" (UniqueName: \"kubernetes.io/projected/9ee67cbd-8115-4c48-b5cd-5299ed284bce-kube-api-access-btdtr\") pod \"network-check-target-ghh5z\" (UID: \"9ee67cbd-8115-4c48-b5cd-5299ed284bce\") " pod="openshift-network-diagnostics/network-check-target-ghh5z" Nov 01 20:30:11.737765 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:11.737691 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 20:30:11.737765 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:11.737713 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 20:30:11.737765 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:11.737725 2580 projected.go:196] Error preparing data for projected volume kube-api-access-btdtr for pod openshift-network-diagnostics/network-check-target-ghh5z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 20:30:11.737910 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:11.737782 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ee67cbd-8115-4c48-b5cd-5299ed284bce-kube-api-access-btdtr podName:9ee67cbd-8115-4c48-b5cd-5299ed284bce nodeName:}" failed. No retries permitted until 2026-11-01 20:30:12.737768547 +0000 UTC m=+4.307256123 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-btdtr" (UniqueName: "kubernetes.io/projected/9ee67cbd-8115-4c48-b5cd-5299ed284bce-kube-api-access-btdtr") pod "network-check-target-ghh5z" (UID: "9ee67cbd-8115-4c48-b5cd-5299ed284bce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 20:30:11.971222 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.971120 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" event={"ID":"3fce64bd-a955-4e2c-8844-a25eaeb73862","Type":"ContainerStarted","Data":"9844bfd7762554a3667f4db4086ff74120af96dddcf30fc796e0ac2cb13ead26"} Nov 01 20:30:11.972617 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.972586 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-w6nn2" event={"ID":"04ec71b7-390e-4e40-abce-1c395a7a51ab","Type":"ContainerStarted","Data":"5882d5d1fac98d98c7525baa5d1d7ea2c0071b79c1c9f344d61fd84c7f11bfaf"} Nov 01 20:30:11.974057 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.974029 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5k2px" event={"ID":"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427","Type":"ContainerStarted","Data":"f35238c9dff3501e77884444e067b5978d544dd583eaace87752ad74535b250a"} Nov 01 20:30:11.975395 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.975370 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-w7qmz" event={"ID":"2288d739-5eb3-4186-a32e-187945267de1","Type":"ContainerStarted","Data":"dcd1501c6ab24fec2ddbec2a6a81381d321b3ca29979d2de8e25807600c2199d"} Nov 01 20:30:11.976954 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.976917 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bztv7" event={"ID":"e8ad39cd-9b6a-464c-aa75-1bfc54b23891","Type":"ContainerStarted","Data":"030f4716dcaaa8462c101a58de0015dbb55dca028de259c1fc7c3915330d1b7e"} Nov 01 20:30:11.978472 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.978439 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fmnrq" event={"ID":"a4342ddb-e06e-4d6d-9337-651349cf133f","Type":"ContainerStarted","Data":"ee503b738885a5321cca7fa7f3c6defd4071ab42689f058bea78da5918ac14e7"} Nov 01 20:30:11.980126 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.980051 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb94" event={"ID":"90e25e61-9761-4042-9dd4-88e1a6f8aa3c","Type":"ContainerStarted","Data":"8d5ba769477249f45bb9343ec9a64905df58d817e34e232a4b8b906025665541"} Nov 01 20:30:11.981276 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.981246 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2f2ct" event={"ID":"5800567f-f0e1-4cf0-8338-75b09d2d1255","Type":"ContainerStarted","Data":"df6c54825fb1d72061008ce2c740924d1b7e34f39b8db1676980e72afd1a970e"} Nov 01 20:30:11.982851 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:11.982827 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" event={"ID":"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc","Type":"ContainerStarted","Data":"5160691e25c668734b4c024993d5bf4c3ed08d76e846e24ca73845637ad73521"} Nov 01 20:30:12.060285 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:12.060239 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 20:25:11 +0000 UTC" deadline="2028-06-16 01:29:45.332777602 +0000 UTC" Nov 01 20:30:12.060285 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:12.060283 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14212h59m33.272499749s" Nov 01 20:30:12.544667 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:12.544623 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/970d71e5-f861-4277-8fa5-6ec740418e51-metrics-certs\") pod \"network-metrics-daemon-vxzfk\" (UID: \"970d71e5-f861-4277-8fa5-6ec740418e51\") " pod="openshift-multus/network-metrics-daemon-vxzfk" Nov 01 20:30:12.544887 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:12.544851 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 20:30:12.544965 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:12.544926 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/970d71e5-f861-4277-8fa5-6ec740418e51-metrics-certs podName:970d71e5-f861-4277-8fa5-6ec740418e51 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:14.544905514 +0000 UTC m=+6.114393099 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/970d71e5-f861-4277-8fa5-6ec740418e51-metrics-certs") pod "network-metrics-daemon-vxzfk" (UID: "970d71e5-f861-4277-8fa5-6ec740418e51") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 20:30:12.646155 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:12.645404 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b4f17a0-efd1-4024-a201-4078069b04e7-original-pull-secret\") pod \"global-pull-secret-syncer-xh6zg\" (UID: \"6b4f17a0-efd1-4024-a201-4078069b04e7\") " pod="kube-system/global-pull-secret-syncer-xh6zg" Nov 01 20:30:12.646155 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:12.645567 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 20:30:12.646155 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:12.645636 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b4f17a0-efd1-4024-a201-4078069b04e7-original-pull-secret podName:6b4f17a0-efd1-4024-a201-4078069b04e7 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:14.645617941 +0000 UTC m=+6.215105515 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b4f17a0-efd1-4024-a201-4078069b04e7-original-pull-secret") pod "global-pull-secret-syncer-xh6zg" (UID: "6b4f17a0-efd1-4024-a201-4078069b04e7") : object "kube-system"/"original-pull-secret" not registered Nov 01 20:30:12.746225 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:12.746183 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btdtr\" (UniqueName: \"kubernetes.io/projected/9ee67cbd-8115-4c48-b5cd-5299ed284bce-kube-api-access-btdtr\") pod \"network-check-target-ghh5z\" (UID: \"9ee67cbd-8115-4c48-b5cd-5299ed284bce\") " pod="openshift-network-diagnostics/network-check-target-ghh5z" Nov 01 20:30:12.746423 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:12.746410 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 20:30:12.746487 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:12.746431 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 20:30:12.746487 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:12.746445 2580 projected.go:196] Error preparing data for projected volume kube-api-access-btdtr for pod openshift-network-diagnostics/network-check-target-ghh5z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 20:30:12.746596 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:12.746510 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ee67cbd-8115-4c48-b5cd-5299ed284bce-kube-api-access-btdtr podName:9ee67cbd-8115-4c48-b5cd-5299ed284bce nodeName:}" failed. No retries permitted until 2026-11-01 20:30:14.746490839 +0000 UTC m=+6.315978435 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-btdtr" (UniqueName: "kubernetes.io/projected/9ee67cbd-8115-4c48-b5cd-5299ed284bce-kube-api-access-btdtr") pod "network-check-target-ghh5z" (UID: "9ee67cbd-8115-4c48-b5cd-5299ed284bce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 20:30:12.966520 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:12.966445 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xh6zg" Nov 01 20:30:12.966520 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:12.966471 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxzfk" Nov 01 20:30:12.966748 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:12.966548 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ghh5z" Nov 01 20:30:12.966748 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:12.966640 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ghh5z" podUID="9ee67cbd-8115-4c48-b5cd-5299ed284bce" Nov 01 20:30:12.966748 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:12.966728 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxzfk" podUID="970d71e5-f861-4277-8fa5-6ec740418e51" Nov 01 20:30:12.966902 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:12.966784 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xh6zg" podUID="6b4f17a0-efd1-4024-a201-4078069b04e7" Nov 01 20:30:13.060882 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:13.060811 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 20:25:11 +0000 UTC" deadline="2028-06-16 13:12:24.524095311 +0000 UTC" Nov 01 20:30:13.060882 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:13.060864 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14224h42m11.463235509s" Nov 01 20:30:14.570732 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:14.570487 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/970d71e5-f861-4277-8fa5-6ec740418e51-metrics-certs\") pod \"network-metrics-daemon-vxzfk\" (UID: \"970d71e5-f861-4277-8fa5-6ec740418e51\") " pod="openshift-multus/network-metrics-daemon-vxzfk" Nov 01 20:30:14.571208 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:14.570860 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 20:30:14.571208 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:14.570952 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/970d71e5-f861-4277-8fa5-6ec740418e51-metrics-certs podName:970d71e5-f861-4277-8fa5-6ec740418e51 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:18.570931489 +0000 UTC m=+10.140419075 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/970d71e5-f861-4277-8fa5-6ec740418e51-metrics-certs") pod "network-metrics-daemon-vxzfk" (UID: "970d71e5-f861-4277-8fa5-6ec740418e51") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 20:30:14.672187 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:14.672155 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b4f17a0-efd1-4024-a201-4078069b04e7-original-pull-secret\") pod \"global-pull-secret-syncer-xh6zg\" (UID: \"6b4f17a0-efd1-4024-a201-4078069b04e7\") " pod="kube-system/global-pull-secret-syncer-xh6zg" Nov 01 20:30:14.672353 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:14.672332 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 20:30:14.672416 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:14.672404 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b4f17a0-efd1-4024-a201-4078069b04e7-original-pull-secret podName:6b4f17a0-efd1-4024-a201-4078069b04e7 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:18.672384185 +0000 UTC m=+10.241871776 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b4f17a0-efd1-4024-a201-4078069b04e7-original-pull-secret") pod "global-pull-secret-syncer-xh6zg" (UID: "6b4f17a0-efd1-4024-a201-4078069b04e7") : object "kube-system"/"original-pull-secret" not registered Nov 01 20:30:14.773002 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:14.772883 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btdtr\" (UniqueName: \"kubernetes.io/projected/9ee67cbd-8115-4c48-b5cd-5299ed284bce-kube-api-access-btdtr\") pod \"network-check-target-ghh5z\" (UID: \"9ee67cbd-8115-4c48-b5cd-5299ed284bce\") " pod="openshift-network-diagnostics/network-check-target-ghh5z" Nov 01 20:30:14.773172 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:14.773094 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 20:30:14.773172 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:14.773119 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 20:30:14.773172 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:14.773131 2580 projected.go:196] Error preparing data for projected volume kube-api-access-btdtr for pod openshift-network-diagnostics/network-check-target-ghh5z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 20:30:14.773362 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:14.773194 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ee67cbd-8115-4c48-b5cd-5299ed284bce-kube-api-access-btdtr podName:9ee67cbd-8115-4c48-b5cd-5299ed284bce nodeName:}" failed. No retries permitted until 2026-11-01 20:30:18.773175092 +0000 UTC m=+10.342662665 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-btdtr" (UniqueName: "kubernetes.io/projected/9ee67cbd-8115-4c48-b5cd-5299ed284bce-kube-api-access-btdtr") pod "network-check-target-ghh5z" (UID: "9ee67cbd-8115-4c48-b5cd-5299ed284bce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 20:30:14.965734 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:14.965699 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xh6zg" Nov 01 20:30:14.965930 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:14.965849 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xh6zg" podUID="6b4f17a0-efd1-4024-a201-4078069b04e7" Nov 01 20:30:14.966002 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:14.965986 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ghh5z" Nov 01 20:30:14.966218 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:14.966165 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ghh5z" podUID="9ee67cbd-8115-4c48-b5cd-5299ed284bce" Nov 01 20:30:14.966438 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:14.966391 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxzfk" Nov 01 20:30:14.966719 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:14.966694 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxzfk" podUID="970d71e5-f861-4277-8fa5-6ec740418e51" Nov 01 20:30:15.019854 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:15.019334 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-35.us-west-1.compute.internal" event={"ID":"f192393bd55f8e34f5db37f9db86763a","Type":"ContainerStarted","Data":"3b894ae806f84c5a33a7700aabef954a09499835545c482ef4080d88aee3d680"} Nov 01 20:30:15.027036 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:15.026995 2580 generic.go:356] "Generic (PLEG): container finished" podID="91ec11d0905cc00deef779d06bc93fe3" containerID="f8e0d927fc93d1c4dbbd959fec7aa6c7672882ba2e5c8a7226ce0c3f1b35150f" exitCode=0 Nov 01 20:30:15.027216 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:15.027053 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-35.us-west-1.compute.internal" event={"ID":"91ec11d0905cc00deef779d06bc93fe3","Type":"ContainerDied","Data":"f8e0d927fc93d1c4dbbd959fec7aa6c7672882ba2e5c8a7226ce0c3f1b35150f"} Nov 01 20:30:15.320932 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:15.320872 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-35.us-west-1.compute.internal" podStartSLOduration=5.320853106 podStartE2EDuration="5.320853106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:30:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:15.197086134 +0000 UTC m=+6.766573731" watchObservedRunningTime="2026-11-01 20:30:15.320853106 +0000 UTC m=+6.890340701" Nov 01 20:30:16.034687 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:16.034635 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-35.us-west-1.compute.internal" event={"ID":"91ec11d0905cc00deef779d06bc93fe3","Type":"ContainerStarted","Data":"336ddb021d7c9688585026d477e6f3d59e5bafc1b41545387c8322fae130823a"} Nov 01 20:30:16.041150 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:16.040582 2580 generic.go:356] "Generic (PLEG): container finished" podID="a4342ddb-e06e-4d6d-9337-651349cf133f" containerID="3f64bb4b02db198c2e89007a8a297ae463ac8545263ef08c20209abe3769d61f" exitCode=0 Nov 01 20:30:16.041150 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:16.040702 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fmnrq" event={"ID":"a4342ddb-e06e-4d6d-9337-651349cf133f","Type":"ContainerDied","Data":"3f64bb4b02db198c2e89007a8a297ae463ac8545263ef08c20209abe3769d61f"} Nov 01 20:30:16.286493 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:16.286391 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-35.us-west-1.compute.internal" podStartSLOduration=6.286369069 podStartE2EDuration="6.286369069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:30:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:16.159389019 +0000 UTC m=+7.728876614" watchObservedRunningTime="2026-11-01 20:30:16.286369069 +0000 UTC m=+7.855856665" Nov 01 20:30:16.965354 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:16.965327 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxzfk" Nov 01 20:30:16.965524 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:16.965432 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxzfk" podUID="970d71e5-f861-4277-8fa5-6ec740418e51" Nov 01 20:30:16.965524 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:16.965496 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xh6zg" Nov 01 20:30:16.965657 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:16.965567 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xh6zg" podUID="6b4f17a0-efd1-4024-a201-4078069b04e7" Nov 01 20:30:16.965657 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:16.965616 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ghh5z" Nov 01 20:30:16.965749 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:16.965674 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ghh5z" podUID="9ee67cbd-8115-4c48-b5cd-5299ed284bce" Nov 01 20:30:17.051770 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:17.051729 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2f2ct" event={"ID":"5800567f-f0e1-4cf0-8338-75b09d2d1255","Type":"ContainerStarted","Data":"5f4428137809487d72004051513ca6e03d877d778a6177900ee83d3fca36e8ee"} Nov 01 20:30:17.168040 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:17.167960 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-2f2ct" podStartSLOduration=2.391337078 podStartE2EDuration="7.167940197s" totalImagesPullingTime="4.776603119s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:30:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:17.167824005 +0000 UTC m=+8.737311675" watchObservedRunningTime="2026-11-01 20:30:17.167940197 +0000 UTC m=+8.737427785" Nov 01 20:30:18.061627 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:18.061588 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bztv7" event={"ID":"e8ad39cd-9b6a-464c-aa75-1bfc54b23891","Type":"ContainerStarted","Data":"daae1d9fcb701cef0642d3a381a96ad81a72ff626df21d83a5541e6c42ab5391"} Nov 01 20:30:18.607035 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:18.606980 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/970d71e5-f861-4277-8fa5-6ec740418e51-metrics-certs\") pod \"network-metrics-daemon-vxzfk\" (UID: \"970d71e5-f861-4277-8fa5-6ec740418e51\") " pod="openshift-multus/network-metrics-daemon-vxzfk" Nov 01 20:30:18.607218 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:18.607162 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 20:30:18.607278 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:18.607230 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/970d71e5-f861-4277-8fa5-6ec740418e51-metrics-certs podName:970d71e5-f861-4277-8fa5-6ec740418e51 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:26.607210647 +0000 UTC m=+18.176698234 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/970d71e5-f861-4277-8fa5-6ec740418e51-metrics-certs") pod "network-metrics-daemon-vxzfk" (UID: "970d71e5-f861-4277-8fa5-6ec740418e51") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 20:30:18.707841 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:18.707800 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b4f17a0-efd1-4024-a201-4078069b04e7-original-pull-secret\") pod \"global-pull-secret-syncer-xh6zg\" (UID: \"6b4f17a0-efd1-4024-a201-4078069b04e7\") " pod="kube-system/global-pull-secret-syncer-xh6zg" Nov 01 20:30:18.708011 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:18.707984 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 20:30:18.708073 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:18.708061 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b4f17a0-efd1-4024-a201-4078069b04e7-original-pull-secret podName:6b4f17a0-efd1-4024-a201-4078069b04e7 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:26.708043714 +0000 UTC m=+18.277531302 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b4f17a0-efd1-4024-a201-4078069b04e7-original-pull-secret") pod "global-pull-secret-syncer-xh6zg" (UID: "6b4f17a0-efd1-4024-a201-4078069b04e7") : object "kube-system"/"original-pull-secret" not registered Nov 01 20:30:18.808769 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:18.808726 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btdtr\" (UniqueName: \"kubernetes.io/projected/9ee67cbd-8115-4c48-b5cd-5299ed284bce-kube-api-access-btdtr\") pod \"network-check-target-ghh5z\" (UID: \"9ee67cbd-8115-4c48-b5cd-5299ed284bce\") " pod="openshift-network-diagnostics/network-check-target-ghh5z" Nov 01 20:30:18.808962 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:18.808940 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 20:30:18.809044 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:18.808971 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 20:30:18.809044 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:18.808986 2580 projected.go:196] Error preparing data for projected volume kube-api-access-btdtr for pod openshift-network-diagnostics/network-check-target-ghh5z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 20:30:18.809132 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:18.809059 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ee67cbd-8115-4c48-b5cd-5299ed284bce-kube-api-access-btdtr podName:9ee67cbd-8115-4c48-b5cd-5299ed284bce nodeName:}" failed. No retries permitted until 2026-11-01 20:30:26.80904203 +0000 UTC m=+18.378529614 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-btdtr" (UniqueName: "kubernetes.io/projected/9ee67cbd-8115-4c48-b5cd-5299ed284bce-kube-api-access-btdtr") pod "network-check-target-ghh5z" (UID: "9ee67cbd-8115-4c48-b5cd-5299ed284bce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 20:30:18.965754 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:18.965681 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ghh5z" Nov 01 20:30:18.965911 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:18.965810 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ghh5z" podUID="9ee67cbd-8115-4c48-b5cd-5299ed284bce" Nov 01 20:30:18.965911 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:18.965890 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxzfk" Nov 01 20:30:18.966087 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:18.965972 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxzfk" podUID="970d71e5-f861-4277-8fa5-6ec740418e51" Nov 01 20:30:18.966087 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:18.966042 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xh6zg" Nov 01 20:30:18.966222 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:18.966113 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xh6zg" podUID="6b4f17a0-efd1-4024-a201-4078069b04e7" Nov 01 20:30:19.066496 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:19.066444 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb94" event={"ID":"90e25e61-9761-4042-9dd4-88e1a6f8aa3c","Type":"ContainerStarted","Data":"5add9a29da14111b2481e477c852eb5d40d195ba46f9d5e40e504ad45c222112"} Nov 01 20:30:20.074600 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:20.074565 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-w6nn2" event={"ID":"04ec71b7-390e-4e40-abce-1c395a7a51ab","Type":"ContainerStarted","Data":"a3c75442a6d897020572030f36cb747d0d0ed6cc20effd9a4595f0f017b368a7"} Nov 01 20:30:20.102466 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:20.102420 2580 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 01 20:30:20.189084 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:20.189023 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bztv7" podStartSLOduration=5.1724331 podStartE2EDuration="11.188988824s" totalImagesPullingTime="6.016555724s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:30:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:18.175750176 +0000 UTC m=+9.745237769" watchObservedRunningTime="2026-11-01 20:30:20.188988824 +0000 UTC m=+11.758476414" Nov 01 20:30:20.899615 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:20.899466 2580 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T20:30:20.102443943Z","UUID":"3ae036df-7add-4631-b9b7-b0f3cb9bfb0c","Handler":null,"Name":"","Endpoint":""} Nov 01 20:30:20.901049 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:20.901020 2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 01 20:30:20.901049 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:20.901054 2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 01 20:30:20.963584 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:20.963553 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxzfk" Nov 01 20:30:20.963750 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:20.963720 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxzfk" podUID="970d71e5-f861-4277-8fa5-6ec740418e51" Nov 01 20:30:20.963853 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:20.963833 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xh6zg" Nov 01 20:30:20.967646 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:20.964015 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xh6zg" podUID="6b4f17a0-efd1-4024-a201-4078069b04e7" Nov 01 20:30:20.967646 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:20.964046 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ghh5z" Nov 01 20:30:20.967646 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:20.964213 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ghh5z" podUID="9ee67cbd-8115-4c48-b5cd-5299ed284bce" Nov 01 20:30:21.006428 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:21.006395 2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-2f2ct" Nov 01 20:30:21.007951 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:21.007922 2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-2f2ct" Nov 01 20:30:21.080243 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:21.080209 2580 generic.go:356] "Generic (PLEG): container finished" podID="a4342ddb-e06e-4d6d-9337-651349cf133f" containerID="11361c228fb886de0dac0d8390c5fe9112e63f6df59b51e90092f9f39bf2453a" exitCode=0 Nov 01 20:30:21.080647 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:21.080290 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fmnrq" event={"ID":"a4342ddb-e06e-4d6d-9337-651349cf133f","Type":"ContainerDied","Data":"11361c228fb886de0dac0d8390c5fe9112e63f6df59b51e90092f9f39bf2453a"} Nov 01 20:30:21.095524 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:21.095483 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb94" event={"ID":"90e25e61-9761-4042-9dd4-88e1a6f8aa3c","Type":"ContainerStarted","Data":"d9c90bf93a4c9beca9b3c159270e89508c235564a3b7c86f503e536f332d85fd"} Nov 01 20:30:21.098726 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:21.098698 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5k2px" event={"ID":"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427","Type":"ContainerStarted","Data":"410002bc9687f7fa18016a0ebbe912f73dfe130a665ecbf7820f5c58adc2b074"} Nov 01 20:30:21.125020 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:21.124941 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-w6nn2" podStartSLOduration=3.683187737 podStartE2EDuration="12.124922611s" totalImagesPullingTime="8.441734874s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:30:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:20.188765391 +0000 UTC m=+11.758252982" watchObservedRunningTime="2026-11-01 20:30:21.124922611 +0000 UTC m=+12.694410204" Nov 01 20:30:21.358604 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:21.358533 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-5k2px" podStartSLOduration=2.63368674 podStartE2EDuration="11.358509509s" totalImagesPullingTime="8.724822769s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:30:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:21.358248322 +0000 UTC m=+12.927735925" watchObservedRunningTime="2026-11-01 20:30:21.358509509 +0000 UTC m=+12.927997102" Nov 01 20:30:22.102656 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:22.102616 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb94" event={"ID":"90e25e61-9761-4042-9dd4-88e1a6f8aa3c","Type":"ContainerStarted","Data":"e4efb7c12658664ddad4dcd2b3a81d46441479e9532554ebbb00788c2896fc1d"} Nov 01 20:30:22.104561 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:22.104530 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-w7qmz" event={"ID":"2288d739-5eb3-4186-a32e-187945267de1","Type":"ContainerStarted","Data":"7fc6379be13259c0b1fae9aadad95d69a26a0f2ef5c2809dc3ed923eedd689e5"} Nov 01 20:30:22.220985 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:22.220886 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb94" podStartSLOduration=2.096475191 podStartE2EDuration="12.220872244s" totalImagesPullingTime="10.124397053s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:30:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:22.220265882 +0000 UTC m=+13.789753475" watchObservedRunningTime="2026-11-01 20:30:22.220872244 +0000 UTC m=+13.790359834" Nov 01 20:30:22.225749 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:22.224426 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-2f2ct" Nov 01 20:30:22.225866 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:22.225833 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2f2ct" Nov 01 20:30:22.446989 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:22.446562 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-w7qmz" podStartSLOduration=3.428086709 podStartE2EDuration="12.446545201s" totalImagesPullingTime="9.018458492s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:30:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:22.332783528 +0000 UTC m=+13.902271119" watchObservedRunningTime="2026-11-01 20:30:22.446545201 +0000 UTC m=+14.016032791" Nov 01 20:30:22.963442 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:22.963406 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxzfk" Nov 01 20:30:22.963442 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:22.963411 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xh6zg" Nov 01 20:30:22.963619 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:22.963575 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxzfk" podUID="970d71e5-f861-4277-8fa5-6ec740418e51" Nov 01 20:30:22.963672 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:22.963620 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ghh5z" Nov 01 20:30:22.963731 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:22.963703 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xh6zg" podUID="6b4f17a0-efd1-4024-a201-4078069b04e7" Nov 01 20:30:22.963772 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:22.963751 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ghh5z" podUID="9ee67cbd-8115-4c48-b5cd-5299ed284bce" Nov 01 20:30:23.108771 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:23.108741 2580 generic.go:356] "Generic (PLEG): container finished" podID="a4342ddb-e06e-4d6d-9337-651349cf133f" containerID="fffe5da2ab8079269a0d6709fe0379a04a5bab5f19657a39b965765763e9ccb1" exitCode=0 Nov 01 20:30:23.109393 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:23.108830 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fmnrq" event={"ID":"a4342ddb-e06e-4d6d-9337-651349cf133f","Type":"ContainerDied","Data":"fffe5da2ab8079269a0d6709fe0379a04a5bab5f19657a39b965765763e9ccb1"} Nov 01 20:30:23.110729 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:23.110703 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" event={"ID":"3fce64bd-a955-4e2c-8844-a25eaeb73862","Type":"ContainerStarted","Data":"c2b048e6d3c6384d2a1c87a636bdf1932110439cc94be6e07ea28f078f032f12"} Nov 01 20:30:23.342472 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:23.342411 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" podStartSLOduration=3.139263208 podStartE2EDuration="14.342391345s" totalImagesPullingTime="11.203128137s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:30:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:23.341818508 +0000 UTC m=+14.911306098" watchObservedRunningTime="2026-11-01 20:30:23.342391345 +0000 UTC m=+14.911878940" Nov 01 20:30:24.963500 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:24.963293 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xh6zg" Nov 01 20:30:24.964034 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:24.963293 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxzfk" Nov 01 20:30:24.964034 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:24.963302 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ghh5z" Nov 01 20:30:24.964034 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:24.963607 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xh6zg" podUID="6b4f17a0-efd1-4024-a201-4078069b04e7" Nov 01 20:30:24.964034 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:24.963685 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ghh5z" podUID="9ee67cbd-8115-4c48-b5cd-5299ed284bce" Nov 01 20:30:24.964034 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:24.963743 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxzfk" podUID="970d71e5-f861-4277-8fa5-6ec740418e51" Nov 01 20:30:25.117531 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:25.117495 2580 generic.go:356] "Generic (PLEG): container finished" podID="a4342ddb-e06e-4d6d-9337-651349cf133f" containerID="c8a5d7fd2b1aaf93c0900511bad19aa46eb48f8c046e6e291601da629866c347" exitCode=0 Nov 01 20:30:25.117690 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:25.117568 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fmnrq" event={"ID":"a4342ddb-e06e-4d6d-9337-651349cf133f","Type":"ContainerDied","Data":"c8a5d7fd2b1aaf93c0900511bad19aa46eb48f8c046e6e291601da629866c347"} Nov 01 20:30:26.671470 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:26.671443 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/970d71e5-f861-4277-8fa5-6ec740418e51-metrics-certs\") pod \"network-metrics-daemon-vxzfk\" (UID: \"970d71e5-f861-4277-8fa5-6ec740418e51\") " pod="openshift-multus/network-metrics-daemon-vxzfk" Nov 01 20:30:26.671880 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:26.671673 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 20:30:26.671880 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:26.671738 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/970d71e5-f861-4277-8fa5-6ec740418e51-metrics-certs podName:970d71e5-f861-4277-8fa5-6ec740418e51 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:42.671719902 +0000 UTC m=+34.241207489 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/970d71e5-f861-4277-8fa5-6ec740418e51-metrics-certs") pod "network-metrics-daemon-vxzfk" (UID: "970d71e5-f861-4277-8fa5-6ec740418e51") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 20:30:26.772569 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:26.772536 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b4f17a0-efd1-4024-a201-4078069b04e7-original-pull-secret\") pod \"global-pull-secret-syncer-xh6zg\" (UID: \"6b4f17a0-efd1-4024-a201-4078069b04e7\") " pod="kube-system/global-pull-secret-syncer-xh6zg" Nov 01 20:30:26.772739 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:26.772718 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 20:30:26.772810 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:26.772791 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b4f17a0-efd1-4024-a201-4078069b04e7-original-pull-secret podName:6b4f17a0-efd1-4024-a201-4078069b04e7 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:42.772769399 +0000 UTC m=+34.342256971 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b4f17a0-efd1-4024-a201-4078069b04e7-original-pull-secret") pod "global-pull-secret-syncer-xh6zg" (UID: "6b4f17a0-efd1-4024-a201-4078069b04e7") : object "kube-system"/"original-pull-secret" not registered Nov 01 20:30:26.873377 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:26.873293 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btdtr\" (UniqueName: \"kubernetes.io/projected/9ee67cbd-8115-4c48-b5cd-5299ed284bce-kube-api-access-btdtr\") pod \"network-check-target-ghh5z\" (UID: \"9ee67cbd-8115-4c48-b5cd-5299ed284bce\") " pod="openshift-network-diagnostics/network-check-target-ghh5z" Nov 01 20:30:26.873523 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:26.873425 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 20:30:26.873523 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:26.873445 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 20:30:26.873523 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:26.873458 2580 projected.go:196] Error preparing data for projected volume kube-api-access-btdtr for pod openshift-network-diagnostics/network-check-target-ghh5z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 20:30:26.873523 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:26.873514 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ee67cbd-8115-4c48-b5cd-5299ed284bce-kube-api-access-btdtr podName:9ee67cbd-8115-4c48-b5cd-5299ed284bce nodeName:}" failed. No retries permitted until 2026-11-01 20:30:42.873502607 +0000 UTC m=+34.442990189 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-btdtr" (UniqueName: "kubernetes.io/projected/9ee67cbd-8115-4c48-b5cd-5299ed284bce-kube-api-access-btdtr") pod "network-check-target-ghh5z" (UID: "9ee67cbd-8115-4c48-b5cd-5299ed284bce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 20:30:26.963387 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:26.963316 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxzfk" Nov 01 20:30:26.963565 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:26.963328 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ghh5z" Nov 01 20:30:26.963565 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:26.963454 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxzfk" podUID="970d71e5-f861-4277-8fa5-6ec740418e51" Nov 01 20:30:26.963565 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:26.963491 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xh6zg" Nov 01 20:30:26.963702 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:26.963578 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ghh5z" podUID="9ee67cbd-8115-4c48-b5cd-5299ed284bce" Nov 01 20:30:26.963702 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:26.963656 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xh6zg" podUID="6b4f17a0-efd1-4024-a201-4078069b04e7" Nov 01 20:30:27.124621 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:27.124532 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fmnrq" event={"ID":"a4342ddb-e06e-4d6d-9337-651349cf133f","Type":"ContainerStarted","Data":"7735f1d5ae7e07f06ca4d4a84afe0ed7c348618bf72ae30823da9a28d1ab0896"} Nov 01 20:30:27.127694 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:27.127663 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" event={"ID":"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc","Type":"ContainerStarted","Data":"48c4106bd933e15902c8be2ab81a203e231814302684f513667a102d5f32fc96"} Nov 01 20:30:27.127857 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:27.127746 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" event={"ID":"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc","Type":"ContainerStarted","Data":"be282410a6fe881472735bcbac77fbd32220ad5dc9d8c24325596aec607acde5"} Nov 01 20:30:27.127857 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:27.127793 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" event={"ID":"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc","Type":"ContainerStarted","Data":"c4d9327ff9974499f563b0cc810285f88a7b3db210ba60e9ffd11e12ffd58f15"} Nov 01 20:30:27.127857 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:27.127834 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" event={"ID":"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc","Type":"ContainerStarted","Data":"9a7148e8c73c5bc3847185730a719ca0eb36d6af5633ac0e23914a95a780e91b"} Nov 01 20:30:27.128015 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:27.127877 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" event={"ID":"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc","Type":"ContainerStarted","Data":"7f7e54dc80adb43185862abd01ba3083c11635321c4e102da7111d9602b83d23"} Nov 01 20:30:27.128015 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:27.127923 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" event={"ID":"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc","Type":"ContainerStarted","Data":"118f2e85de906e58de484fb9e0b5829972aa4d5d09439c1dcce7e93493c8a563"} Nov 01 20:30:28.132374 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:28.132334 2580 generic.go:356] "Generic (PLEG): container finished" podID="a4342ddb-e06e-4d6d-9337-651349cf133f" containerID="7735f1d5ae7e07f06ca4d4a84afe0ed7c348618bf72ae30823da9a28d1ab0896" exitCode=0 Nov 01 20:30:28.132860 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:28.132395 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fmnrq" event={"ID":"a4342ddb-e06e-4d6d-9337-651349cf133f","Type":"ContainerDied","Data":"7735f1d5ae7e07f06ca4d4a84afe0ed7c348618bf72ae30823da9a28d1ab0896"} Nov 01 20:30:28.964099 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:28.964060 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxzfk" Nov 01 20:30:28.964299 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:28.964189 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ghh5z" Nov 01 20:30:28.964299 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:28.964198 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxzfk" podUID="970d71e5-f861-4277-8fa5-6ec740418e51" Nov 01 20:30:28.964299 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:28.964272 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xh6zg" Nov 01 20:30:28.964299 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:28.964273 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ghh5z" podUID="9ee67cbd-8115-4c48-b5cd-5299ed284bce" Nov 01 20:30:28.964508 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:28.964327 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xh6zg" podUID="6b4f17a0-efd1-4024-a201-4078069b04e7" Nov 01 20:30:29.137660 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:29.137624 2580 generic.go:356] "Generic (PLEG): container finished" podID="a4342ddb-e06e-4d6d-9337-651349cf133f" containerID="b725fc29140dc56bc40463e46694446809bfadb897cc5e5e923542cd79886078" exitCode=0 Nov 01 20:30:29.138284 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:29.137694 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fmnrq" event={"ID":"a4342ddb-e06e-4d6d-9337-651349cf133f","Type":"ContainerDied","Data":"b725fc29140dc56bc40463e46694446809bfadb897cc5e5e923542cd79886078"} Nov 01 20:30:29.141061 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:29.141041 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" event={"ID":"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc","Type":"ContainerStarted","Data":"297ef4a3bc88ab88f8f19f6bc6360c22c1f2f0614da6f27f60ccf96868ee524d"} Nov 01 20:30:30.146371 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:30.146334 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fmnrq" event={"ID":"a4342ddb-e06e-4d6d-9337-651349cf133f","Type":"ContainerStarted","Data":"5da012bcf2814c5c024fe10d5b851016a33b9dfbf775000ed0b5257fc8be77d9"} Nov 01 20:30:30.339447 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:30.339391 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-fmnrq" podStartSLOduration=8.587853219 podStartE2EDuration="21.339373915s" totalImagesPullingTime="12.751520696s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:30:09 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:30.338755734 +0000 UTC m=+21.908243335" watchObservedRunningTime="2026-11-01 20:30:30.339373915 +0000 UTC m=+21.908861508" Nov 01 20:30:30.962762 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:30.962721 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xh6zg" Nov 01 20:30:30.962762 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:30.962756 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ghh5z" Nov 01 20:30:30.962941 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:30.962766 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxzfk" Nov 01 20:30:30.962941 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:30.962849 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xh6zg" podUID="6b4f17a0-efd1-4024-a201-4078069b04e7" Nov 01 20:30:30.962941 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:30.962919 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ghh5z" podUID="9ee67cbd-8115-4c48-b5cd-5299ed284bce" Nov 01 20:30:30.963062 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:30.962988 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxzfk" podUID="970d71e5-f861-4277-8fa5-6ec740418e51" Nov 01 20:30:31.152614 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:31.152367 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" event={"ID":"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc","Type":"ContainerStarted","Data":"e23e1642550b01229c28b37ffd3313bb57b10beafb937c1935d30b4d8a2b17ed"} Nov 01 20:30:31.153278 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:31.152861 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:31.153278 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:31.152931 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:31.168061 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:31.168039 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:31.426373 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:31.426250 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" podStartSLOduration=7.34424515 podStartE2EDuration="22.426231496s" totalImagesPullingTime="15.081986346s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:30:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:31.376770064 +0000 UTC m=+22.946257656" watchObservedRunningTime="2026-11-01 20:30:31.426231496 +0000 UTC m=+22.995719078" Nov 01 20:30:32.154802 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:32.154770 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:32.169158 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:32.169133 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:32.363263 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:32.363235 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:32.549091 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:32.549057 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:32.758960 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:32.758929 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:32.962649 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:32.962564 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xh6zg" Nov 01 20:30:32.962649 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:32.962580 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxzfk" Nov 01 20:30:32.962854 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:32.962680 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xh6zg" podUID="6b4f17a0-efd1-4024-a201-4078069b04e7" Nov 01 20:30:32.962854 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:32.962708 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ghh5z" Nov 01 20:30:32.962854 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:32.962762 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxzfk" podUID="970d71e5-f861-4277-8fa5-6ec740418e51" Nov 01 20:30:32.962854 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:32.962832 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ghh5z" podUID="9ee67cbd-8115-4c48-b5cd-5299ed284bce" Nov 01 20:30:33.491428 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:33.491397 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xh6zg"] Nov 01 20:30:33.491850 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:33.491500 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xh6zg" Nov 01 20:30:33.491850 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:33.491591 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xh6zg" podUID="6b4f17a0-efd1-4024-a201-4078069b04e7" Nov 01 20:30:33.494349 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:33.494321 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vxzfk"] Nov 01 20:30:33.494445 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:33.494432 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxzfk" Nov 01 20:30:33.494563 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:33.494544 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxzfk" podUID="970d71e5-f861-4277-8fa5-6ec740418e51" Nov 01 20:30:33.494949 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:33.494928 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ghh5z"] Nov 01 20:30:33.495048 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:33.495023 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ghh5z" Nov 01 20:30:33.495132 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:33.495112 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ghh5z" podUID="9ee67cbd-8115-4c48-b5cd-5299ed284bce" Nov 01 20:30:34.963335 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:34.963300 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xh6zg" Nov 01 20:30:34.963878 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:34.963360 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxzfk" Nov 01 20:30:34.963878 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:34.963373 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ghh5z" Nov 01 20:30:34.963878 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:34.963467 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xh6zg" podUID="6b4f17a0-efd1-4024-a201-4078069b04e7" Nov 01 20:30:34.964060 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:34.963879 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxzfk" podUID="970d71e5-f861-4277-8fa5-6ec740418e51" Nov 01 20:30:34.964060 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:34.963981 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ghh5z" podUID="9ee67cbd-8115-4c48-b5cd-5299ed284bce" Nov 01 20:30:35.928050 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:35.927960 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:36.098347 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:36.098316 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:36.420870 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:36.420838 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:36.584945 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:36.584908 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:36.962723 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:36.962691 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xh6zg" Nov 01 20:30:36.962883 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:36.962694 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxzfk" Nov 01 20:30:36.962883 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:36.962831 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xh6zg" podUID="6b4f17a0-efd1-4024-a201-4078069b04e7" Nov 01 20:30:36.963018 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:36.962705 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ghh5z" Nov 01 20:30:36.963018 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:36.962887 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxzfk" podUID="970d71e5-f861-4277-8fa5-6ec740418e51" Nov 01 20:30:36.963018 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:36.963009 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ghh5z" podUID="9ee67cbd-8115-4c48-b5cd-5299ed284bce" Nov 01 20:30:38.689591 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:38.689556 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:38.964106 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:38.963868 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ghh5z" Nov 01 20:30:38.964242 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:38.964116 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ghh5z" podUID="9ee67cbd-8115-4c48-b5cd-5299ed284bce" Nov 01 20:30:38.964242 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:38.963960 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xh6zg" Nov 01 20:30:38.964242 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:38.964022 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxzfk" Nov 01 20:30:38.964352 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:38.964250 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xh6zg" podUID="6b4f17a0-efd1-4024-a201-4078069b04e7" Nov 01 20:30:38.964352 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:38.964264 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxzfk" podUID="970d71e5-f861-4277-8fa5-6ec740418e51" Nov 01 20:30:39.201899 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:39.201872 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-35.us-west-1.compute.internal" event="NodeReady" Nov 01 20:30:39.202060 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:39.202018 2580 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 01 20:30:39.369164 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:39.369125 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-gtf4c"] Nov 01 20:30:39.394166 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:39.394136 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4ssvj"] Nov 01 20:30:39.394320 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:39.394297 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gtf4c" Nov 01 20:30:39.400408 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:39.400385 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mdc46"] Nov 01 20:30:39.400532 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:39.400509 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4ssvj" Nov 01 20:30:39.406615 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:39.406588 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gtf4c"] Nov 01 20:30:39.406714 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:39.406684 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mdc46" Nov 01 20:30:39.423362 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:39.423335 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4ssvj"] Nov 01 20:30:39.461292 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:39.461263 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bbb0eff1-ed70-4783-a255-374d8bf14687-metrics-tls\") pod \"dns-default-gtf4c\" (UID: \"bbb0eff1-ed70-4783-a255-374d8bf14687\") " pod="openshift-dns/dns-default-gtf4c" Nov 01 20:30:39.461444 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:39.461317 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bbb0eff1-ed70-4783-a255-374d8bf14687-config-volume\") pod \"dns-default-gtf4c\" (UID: \"bbb0eff1-ed70-4783-a255-374d8bf14687\") " pod="openshift-dns/dns-default-gtf4c" Nov 01 20:30:39.461444 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:39.461335 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbb0eff1-ed70-4783-a255-374d8bf14687-tmp-dir\") pod \"dns-default-gtf4c\" (UID: \"bbb0eff1-ed70-4783-a255-374d8bf14687\") " pod="openshift-dns/dns-default-gtf4c" Nov 01 20:30:39.461444 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:39.461428 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74vvb\" (UniqueName: \"kubernetes.io/projected/bbb0eff1-ed70-4783-a255-374d8bf14687-kube-api-access-74vvb\") pod \"dns-default-gtf4c\" (UID: \"bbb0eff1-ed70-4783-a255-374d8bf14687\") " pod="openshift-dns/dns-default-gtf4c" Nov 01 20:30:39.469154 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:39.469134 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 01 20:30:39.469154 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:39.469148 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 01 20:30:39.469302 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:39.469184 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 01 20:30:39.469302 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:39.469238 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 01 20:30:39.469936 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:39.469920 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 01 20:30:39.469983 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:39.469951 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-t8z8c\"" Nov 01 20:30:39.469983 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:39.469965 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 01 20:30:39.470072 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:39.470042 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-n2j5c\"" Nov 01 20:30:39.473229 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:39.473214 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 01 20:30:39.484565 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:39.484530 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mdc46"] Nov 01 20:30:39.554411 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:39.554384 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:39.562182 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:39.562161 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bbb0eff1-ed70-4783-a255-374d8bf14687-metrics-tls\") pod \"dns-default-gtf4c\" (UID: \"bbb0eff1-ed70-4783-a255-374d8bf14687\") " pod="openshift-dns/dns-default-gtf4c" Nov 01 20:30:39.562272 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:39.562191 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/82ca0c95-053f-4bb0-ab8d-1d8cbd531667-data-volume\") pod \"insights-runtime-extractor-4ssvj\" (UID: \"82ca0c95-053f-4bb0-ab8d-1d8cbd531667\") " pod="openshift-insights/insights-runtime-extractor-4ssvj" Nov 01 20:30:39.562272 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:39.562231 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82ca0c95-053f-4bb0-ab8d-1d8cbd531667-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4ssvj\" (UID: \"82ca0c95-053f-4bb0-ab8d-1d8cbd531667\") " pod="openshift-insights/insights-runtime-extractor-4ssvj" Nov 01 20:30:39.562272 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:39.562256 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw4w8\" (UniqueName: \"kubernetes.io/projected/492a8e93-84b5-46d3-b6e4-4278f737249a-kube-api-access-sw4w8\") pod \"ingress-canary-mdc46\" (UID: \"492a8e93-84b5-46d3-b6e4-4278f737249a\") " pod="openshift-ingress-canary/ingress-canary-mdc46" Nov 01 20:30:39.562375 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:39.562311 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 20:30:39.562375 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:39.562319 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kplvv\" (UniqueName: \"kubernetes.io/projected/82ca0c95-053f-4bb0-ab8d-1d8cbd531667-kube-api-access-kplvv\") pod \"insights-runtime-extractor-4ssvj\" (UID: \"82ca0c95-053f-4bb0-ab8d-1d8cbd531667\") " pod="openshift-insights/insights-runtime-extractor-4ssvj" Nov 01 20:30:39.562375 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:39.562364 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/492a8e93-84b5-46d3-b6e4-4278f737249a-cert\") pod \"ingress-canary-mdc46\" (UID: \"492a8e93-84b5-46d3-b6e4-4278f737249a\") " pod="openshift-ingress-canary/ingress-canary-mdc46" Nov 01 20:30:39.562475 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:39.562384 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbb0eff1-ed70-4783-a255-374d8bf14687-metrics-tls podName:bbb0eff1-ed70-4783-a255-374d8bf14687 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:40.062367846 +0000 UTC m=+31.631855415 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bbb0eff1-ed70-4783-a255-374d8bf14687-metrics-tls") pod "dns-default-gtf4c" (UID: "bbb0eff1-ed70-4783-a255-374d8bf14687") : secret "dns-default-metrics-tls" not found Nov 01 20:30:39.562475 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:39.562449 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bbb0eff1-ed70-4783-a255-374d8bf14687-config-volume\") pod \"dns-default-gtf4c\" (UID: \"bbb0eff1-ed70-4783-a255-374d8bf14687\") " pod="openshift-dns/dns-default-gtf4c" Nov 01 20:30:39.562549 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:39.562473 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbb0eff1-ed70-4783-a255-374d8bf14687-tmp-dir\") pod \"dns-default-gtf4c\" (UID: \"bbb0eff1-ed70-4783-a255-374d8bf14687\") " pod="openshift-dns/dns-default-gtf4c" Nov 01 20:30:39.562549 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:39.562496 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74vvb\" (UniqueName: \"kubernetes.io/projected/bbb0eff1-ed70-4783-a255-374d8bf14687-kube-api-access-74vvb\") pod \"dns-default-gtf4c\" (UID: \"bbb0eff1-ed70-4783-a255-374d8bf14687\") " pod="openshift-dns/dns-default-gtf4c" Nov 01 20:30:39.562549 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:39.562517 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82ca0c95-053f-4bb0-ab8d-1d8cbd531667-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4ssvj\" (UID: \"82ca0c95-053f-4bb0-ab8d-1d8cbd531667\") " pod="openshift-insights/insights-runtime-extractor-4ssvj" Nov 01 20:30:39.562549 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:39.562544 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/82ca0c95-053f-4bb0-ab8d-1d8cbd531667-crio-socket\") pod \"insights-runtime-extractor-4ssvj\" (UID: \"82ca0c95-053f-4bb0-ab8d-1d8cbd531667\") " pod="openshift-insights/insights-runtime-extractor-4ssvj" Nov 01 20:30:39.562835 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:39.562820 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbb0eff1-ed70-4783-a255-374d8bf14687-tmp-dir\") pod \"dns-default-gtf4c\" (UID: \"bbb0eff1-ed70-4783-a255-374d8bf14687\") " pod="openshift-dns/dns-default-gtf4c" Nov 01 20:30:39.562980 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:39.562964 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bbb0eff1-ed70-4783-a255-374d8bf14687-config-volume\") pod \"dns-default-gtf4c\" (UID: \"bbb0eff1-ed70-4783-a255-374d8bf14687\") " pod="openshift-dns/dns-default-gtf4c" Nov 01 20:30:39.570628 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:39.570614 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 01 20:30:39.570694 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:39.570679 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-8jgj4\"" Nov 01 20:30:39.570737 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:39.570709 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 01 20:30:39.622123 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:39.622065 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74vvb\" (UniqueName: \"kubernetes.io/projected/bbb0eff1-ed70-4783-a255-374d8bf14687-kube-api-access-74vvb\") pod \"dns-default-gtf4c\" (UID: \"bbb0eff1-ed70-4783-a255-374d8bf14687\") " pod="openshift-dns/dns-default-gtf4c" Nov 01 20:30:39.663102 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:39.663064 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw4w8\" (UniqueName: \"kubernetes.io/projected/492a8e93-84b5-46d3-b6e4-4278f737249a-kube-api-access-sw4w8\") pod \"ingress-canary-mdc46\" (UID: \"492a8e93-84b5-46d3-b6e4-4278f737249a\") " pod="openshift-ingress-canary/ingress-canary-mdc46" Nov 01 20:30:39.663102 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:39.663102 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kplvv\" (UniqueName: \"kubernetes.io/projected/82ca0c95-053f-4bb0-ab8d-1d8cbd531667-kube-api-access-kplvv\") pod \"insights-runtime-extractor-4ssvj\" (UID: \"82ca0c95-053f-4bb0-ab8d-1d8cbd531667\") " pod="openshift-insights/insights-runtime-extractor-4ssvj" Nov 01 20:30:39.663315 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:39.663120 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/492a8e93-84b5-46d3-b6e4-4278f737249a-cert\") pod \"ingress-canary-mdc46\" (UID: \"492a8e93-84b5-46d3-b6e4-4278f737249a\") " pod="openshift-ingress-canary/ingress-canary-mdc46" Nov 01 20:30:39.663315 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:39.663192 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82ca0c95-053f-4bb0-ab8d-1d8cbd531667-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4ssvj\" (UID: \"82ca0c95-053f-4bb0-ab8d-1d8cbd531667\") " pod="openshift-insights/insights-runtime-extractor-4ssvj" Nov 01 20:30:39.663315 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:39.663231 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/82ca0c95-053f-4bb0-ab8d-1d8cbd531667-crio-socket\") pod \"insights-runtime-extractor-4ssvj\" (UID: \"82ca0c95-053f-4bb0-ab8d-1d8cbd531667\") " pod="openshift-insights/insights-runtime-extractor-4ssvj" Nov 01 20:30:39.663315 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:39.663276 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/82ca0c95-053f-4bb0-ab8d-1d8cbd531667-data-volume\") pod \"insights-runtime-extractor-4ssvj\" (UID: \"82ca0c95-053f-4bb0-ab8d-1d8cbd531667\") " pod="openshift-insights/insights-runtime-extractor-4ssvj" Nov 01 20:30:39.663509 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:39.663315 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82ca0c95-053f-4bb0-ab8d-1d8cbd531667-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4ssvj\" (UID: \"82ca0c95-053f-4bb0-ab8d-1d8cbd531667\") " pod="openshift-insights/insights-runtime-extractor-4ssvj" Nov 01 20:30:39.663509 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:39.663335 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 01 20:30:39.663509 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:39.663401 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/82ca0c95-053f-4bb0-ab8d-1d8cbd531667-kube-rbac-proxy-cm podName:82ca0c95-053f-4bb0-ab8d-1d8cbd531667 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:40.163384791 +0000 UTC m=+31.732872360 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/82ca0c95-053f-4bb0-ab8d-1d8cbd531667-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4ssvj" (UID: "82ca0c95-053f-4bb0-ab8d-1d8cbd531667") : configmap "kube-rbac-proxy" not found Nov 01 20:30:39.663509 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:39.663416 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 20:30:39.663509 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:39.663476 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82ca0c95-053f-4bb0-ab8d-1d8cbd531667-insights-runtime-extractor-tls podName:82ca0c95-053f-4bb0-ab8d-1d8cbd531667 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:40.16346066 +0000 UTC m=+31.732948233 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/82ca0c95-053f-4bb0-ab8d-1d8cbd531667-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4ssvj" (UID: "82ca0c95-053f-4bb0-ab8d-1d8cbd531667") : secret "insights-runtime-extractor-tls" not found Nov 01 20:30:39.663724 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:39.663542 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/82ca0c95-053f-4bb0-ab8d-1d8cbd531667-crio-socket\") pod \"insights-runtime-extractor-4ssvj\" (UID: \"82ca0c95-053f-4bb0-ab8d-1d8cbd531667\") " pod="openshift-insights/insights-runtime-extractor-4ssvj" Nov 01 20:30:39.663724 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:39.663645 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/82ca0c95-053f-4bb0-ab8d-1d8cbd531667-data-volume\") pod \"insights-runtime-extractor-4ssvj\" (UID: \"82ca0c95-053f-4bb0-ab8d-1d8cbd531667\") " pod="openshift-insights/insights-runtime-extractor-4ssvj" Nov 01 20:30:39.665684 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:39.665667 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/492a8e93-84b5-46d3-b6e4-4278f737249a-cert\") pod \"ingress-canary-mdc46\" (UID: \"492a8e93-84b5-46d3-b6e4-4278f737249a\") " pod="openshift-ingress-canary/ingress-canary-mdc46" Nov 01 20:30:39.713332 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:39.713295 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:39.721706 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:39.721670 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw4w8\" (UniqueName: \"kubernetes.io/projected/492a8e93-84b5-46d3-b6e4-4278f737249a-kube-api-access-sw4w8\") pod \"ingress-canary-mdc46\" (UID: \"492a8e93-84b5-46d3-b6e4-4278f737249a\") " pod="openshift-ingress-canary/ingress-canary-mdc46" Nov 01 20:30:39.724045 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:39.724020 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kplvv\" (UniqueName: \"kubernetes.io/projected/82ca0c95-053f-4bb0-ab8d-1d8cbd531667-kube-api-access-kplvv\") pod \"insights-runtime-extractor-4ssvj\" (UID: \"82ca0c95-053f-4bb0-ab8d-1d8cbd531667\") " pod="openshift-insights/insights-runtime-extractor-4ssvj" Nov 01 20:30:39.737619 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:39.737598 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mdc46" Nov 01 20:30:39.914780 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:39.914703 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:40.067518 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:40.067489 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bbb0eff1-ed70-4783-a255-374d8bf14687-metrics-tls\") pod \"dns-default-gtf4c\" (UID: \"bbb0eff1-ed70-4783-a255-374d8bf14687\") " pod="openshift-dns/dns-default-gtf4c" Nov 01 20:30:40.067682 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:40.067614 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 20:30:40.067815 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:40.067683 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbb0eff1-ed70-4783-a255-374d8bf14687-metrics-tls podName:bbb0eff1-ed70-4783-a255-374d8bf14687 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:41.067668036 +0000 UTC m=+32.637155605 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bbb0eff1-ed70-4783-a255-374d8bf14687-metrics-tls") pod "dns-default-gtf4c" (UID: "bbb0eff1-ed70-4783-a255-374d8bf14687") : secret "dns-default-metrics-tls" not found Nov 01 20:30:40.168421 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:40.168333 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82ca0c95-053f-4bb0-ab8d-1d8cbd531667-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4ssvj\" (UID: \"82ca0c95-053f-4bb0-ab8d-1d8cbd531667\") " pod="openshift-insights/insights-runtime-extractor-4ssvj" Nov 01 20:30:40.168566 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:40.168440 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82ca0c95-053f-4bb0-ab8d-1d8cbd531667-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4ssvj\" (UID: \"82ca0c95-053f-4bb0-ab8d-1d8cbd531667\") " pod="openshift-insights/insights-runtime-extractor-4ssvj" Nov 01 20:30:40.168566 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:40.168453 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 01 20:30:40.168566 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:40.168512 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/82ca0c95-053f-4bb0-ab8d-1d8cbd531667-kube-rbac-proxy-cm podName:82ca0c95-053f-4bb0-ab8d-1d8cbd531667 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:41.168495916 +0000 UTC m=+32.737983493 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/82ca0c95-053f-4bb0-ab8d-1d8cbd531667-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4ssvj" (UID: "82ca0c95-053f-4bb0-ab8d-1d8cbd531667") : configmap "kube-rbac-proxy" not found Nov 01 20:30:40.168566 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:40.168554 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 20:30:40.168682 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:40.168594 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82ca0c95-053f-4bb0-ab8d-1d8cbd531667-insights-runtime-extractor-tls podName:82ca0c95-053f-4bb0-ab8d-1d8cbd531667 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:41.16858413 +0000 UTC m=+32.738071715 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/82ca0c95-053f-4bb0-ab8d-1d8cbd531667-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4ssvj" (UID: "82ca0c95-053f-4bb0-ab8d-1d8cbd531667") : secret "insights-runtime-extractor-tls" not found Nov 01 20:30:40.184368 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:40.184268 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mdc46"] Nov 01 20:30:40.187103 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:40.187069 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod492a8e93_84b5_46d3_b6e4_4278f737249a.slice/crio-80729d64d0dbb0313f04a42bdf9b3ace58e24fb0ebf40ae93be25b504c4ce195 WatchSource:0}: Error finding container 80729d64d0dbb0313f04a42bdf9b3ace58e24fb0ebf40ae93be25b504c4ce195: Status 404 returned error can't find the container with id 80729d64d0dbb0313f04a42bdf9b3ace58e24fb0ebf40ae93be25b504c4ce195 Nov 01 20:30:40.758675 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:40.758644 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-smp9z"] Nov 01 20:30:40.789282 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:40.789255 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-smp9z" Nov 01 20:30:40.816928 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:40.816903 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-smp9z"] Nov 01 20:30:40.845314 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:40.845287 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 01 20:30:40.845467 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:40.845288 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 01 20:30:40.845467 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:40.845355 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-mv7c7\"" Nov 01 20:30:40.845467 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:40.845442 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 01 20:30:40.866961 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:40.866936 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 01 20:30:40.871081 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:40.871056 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:40.873107 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:40.873088 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f05cdcaf-b144-4c3d-b29b-2cb7d5a6f085-signing-cabundle\") pod \"service-ca-5fffc97bdf-smp9z\" (UID: \"f05cdcaf-b144-4c3d-b29b-2cb7d5a6f085\") " pod="openshift-service-ca/service-ca-5fffc97bdf-smp9z" Nov 01 20:30:40.873212 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:40.873130 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhml7\" (UniqueName: \"kubernetes.io/projected/f05cdcaf-b144-4c3d-b29b-2cb7d5a6f085-kube-api-access-dhml7\") pod \"service-ca-5fffc97bdf-smp9z\" (UID: \"f05cdcaf-b144-4c3d-b29b-2cb7d5a6f085\") " pod="openshift-service-ca/service-ca-5fffc97bdf-smp9z" Nov 01 20:30:40.873384 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:40.873254 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f05cdcaf-b144-4c3d-b29b-2cb7d5a6f085-signing-key\") pod \"service-ca-5fffc97bdf-smp9z\" (UID: \"f05cdcaf-b144-4c3d-b29b-2cb7d5a6f085\") " pod="openshift-service-ca/service-ca-5fffc97bdf-smp9z" Nov 01 20:30:40.963037 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:40.963004 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ghh5z" Nov 01 20:30:40.963037 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:40.963022 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxzfk" Nov 01 20:30:40.963232 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:40.963086 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xh6zg" Nov 01 20:30:40.974258 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:40.974236 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f05cdcaf-b144-4c3d-b29b-2cb7d5a6f085-signing-key\") pod \"service-ca-5fffc97bdf-smp9z\" (UID: \"f05cdcaf-b144-4c3d-b29b-2cb7d5a6f085\") " pod="openshift-service-ca/service-ca-5fffc97bdf-smp9z" Nov 01 20:30:40.974405 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:40.974390 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f05cdcaf-b144-4c3d-b29b-2cb7d5a6f085-signing-cabundle\") pod \"service-ca-5fffc97bdf-smp9z\" (UID: \"f05cdcaf-b144-4c3d-b29b-2cb7d5a6f085\") " pod="openshift-service-ca/service-ca-5fffc97bdf-smp9z" Nov 01 20:30:40.974447 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:40.974430 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhml7\" (UniqueName: \"kubernetes.io/projected/f05cdcaf-b144-4c3d-b29b-2cb7d5a6f085-kube-api-access-dhml7\") pod \"service-ca-5fffc97bdf-smp9z\" (UID: \"f05cdcaf-b144-4c3d-b29b-2cb7d5a6f085\") " pod="openshift-service-ca/service-ca-5fffc97bdf-smp9z" Nov 01 20:30:40.975172 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:40.975152 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f05cdcaf-b144-4c3d-b29b-2cb7d5a6f085-signing-cabundle\") pod \"service-ca-5fffc97bdf-smp9z\" (UID: \"f05cdcaf-b144-4c3d-b29b-2cb7d5a6f085\") " pod="openshift-service-ca/service-ca-5fffc97bdf-smp9z" Nov 01 20:30:40.976906 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:40.976880 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f05cdcaf-b144-4c3d-b29b-2cb7d5a6f085-signing-key\") pod \"service-ca-5fffc97bdf-smp9z\" (UID: \"f05cdcaf-b144-4c3d-b29b-2cb7d5a6f085\") " pod="openshift-service-ca/service-ca-5fffc97bdf-smp9z" Nov 01 20:30:41.018385 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:41.018318 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-kzbx4\"" Nov 01 20:30:41.018524 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:41.018512 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 20:30:41.018590 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:41.018569 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 20:30:41.018640 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:41.018592 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cg54x\"" Nov 01 20:30:41.018640 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:41.018592 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 20:30:41.018737 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:41.018665 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 20:30:41.018775 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:41.018731 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-pskvw\"" Nov 01 20:30:41.031865 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:41.031841 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhml7\" (UniqueName: \"kubernetes.io/projected/f05cdcaf-b144-4c3d-b29b-2cb7d5a6f085-kube-api-access-dhml7\") pod \"service-ca-5fffc97bdf-smp9z\" (UID: \"f05cdcaf-b144-4c3d-b29b-2cb7d5a6f085\") " pod="openshift-service-ca/service-ca-5fffc97bdf-smp9z" Nov 01 20:30:41.047293 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:41.047264 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:41.056305 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:41.056284 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-2lxrg"] Nov 01 20:30:41.065041 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:41.064975 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2lxrg" Nov 01 20:30:41.075027 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:41.074986 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bbb0eff1-ed70-4783-a255-374d8bf14687-metrics-tls\") pod \"dns-default-gtf4c\" (UID: \"bbb0eff1-ed70-4783-a255-374d8bf14687\") " pod="openshift-dns/dns-default-gtf4c" Nov 01 20:30:41.075251 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:41.075234 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 20:30:41.075322 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:41.075311 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbb0eff1-ed70-4783-a255-374d8bf14687-metrics-tls podName:bbb0eff1-ed70-4783-a255-374d8bf14687 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:43.075297754 +0000 UTC m=+34.644785337 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bbb0eff1-ed70-4783-a255-374d8bf14687-metrics-tls") pod "dns-default-gtf4c" (UID: "bbb0eff1-ed70-4783-a255-374d8bf14687") : secret "dns-default-metrics-tls" not found Nov 01 20:30:41.112281 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:41.112254 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-smp9z" Nov 01 20:30:41.117442 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:41.117410 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-2lxrg"] Nov 01 20:30:41.121164 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:41.121125 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 01 20:30:41.122035 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:41.121927 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4gdtb\"" Nov 01 20:30:41.122163 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:41.122039 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 01 20:30:41.176599 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:41.175650 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhrjv\" (UniqueName: \"kubernetes.io/projected/8f213a88-8cb5-467b-8d21-9bc86a0654c9-kube-api-access-lhrjv\") pod \"migrator-66df5d496b-2lxrg\" (UID: \"8f213a88-8cb5-467b-8d21-9bc86a0654c9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2lxrg" Nov 01 20:30:41.176599 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:41.175782 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82ca0c95-053f-4bb0-ab8d-1d8cbd531667-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4ssvj\" (UID: \"82ca0c95-053f-4bb0-ab8d-1d8cbd531667\") " pod="openshift-insights/insights-runtime-extractor-4ssvj" Nov 01 20:30:41.176599 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:41.175870 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 01 20:30:41.176599 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:41.175922 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/82ca0c95-053f-4bb0-ab8d-1d8cbd531667-kube-rbac-proxy-cm podName:82ca0c95-053f-4bb0-ab8d-1d8cbd531667 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:43.175907376 +0000 UTC m=+34.745394945 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/82ca0c95-053f-4bb0-ab8d-1d8cbd531667-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4ssvj" (UID: "82ca0c95-053f-4bb0-ab8d-1d8cbd531667") : configmap "kube-rbac-proxy" not found Nov 01 20:30:41.178979 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:41.178946 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mdc46" event={"ID":"492a8e93-84b5-46d3-b6e4-4278f737249a","Type":"ContainerStarted","Data":"80729d64d0dbb0313f04a42bdf9b3ace58e24fb0ebf40ae93be25b504c4ce195"} Nov 01 20:30:41.183125 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:41.183103 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82ca0c95-053f-4bb0-ab8d-1d8cbd531667-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4ssvj\" (UID: \"82ca0c95-053f-4bb0-ab8d-1d8cbd531667\") " pod="openshift-insights/insights-runtime-extractor-4ssvj" Nov 01 20:30:41.183237 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:41.183224 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 20:30:41.183302 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:41.183274 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82ca0c95-053f-4bb0-ab8d-1d8cbd531667-insights-runtime-extractor-tls podName:82ca0c95-053f-4bb0-ab8d-1d8cbd531667 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:43.183261808 +0000 UTC m=+34.752749389 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/82ca0c95-053f-4bb0-ab8d-1d8cbd531667-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4ssvj" (UID: "82ca0c95-053f-4bb0-ab8d-1d8cbd531667") : secret "insights-runtime-extractor-tls" not found Nov 01 20:30:41.218827 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:41.218786 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:41.284369 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:41.284288 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhrjv\" (UniqueName: \"kubernetes.io/projected/8f213a88-8cb5-467b-8d21-9bc86a0654c9-kube-api-access-lhrjv\") pod \"migrator-66df5d496b-2lxrg\" (UID: \"8f213a88-8cb5-467b-8d21-9bc86a0654c9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2lxrg" Nov 01 20:30:41.343104 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:41.343059 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhrjv\" (UniqueName: \"kubernetes.io/projected/8f213a88-8cb5-467b-8d21-9bc86a0654c9-kube-api-access-lhrjv\") pod \"migrator-66df5d496b-2lxrg\" (UID: \"8f213a88-8cb5-467b-8d21-9bc86a0654c9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2lxrg" Nov 01 20:30:41.374387 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:41.374354 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:41.381613 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:41.381581 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2lxrg" Nov 01 20:30:41.495130 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:41.495062 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-smp9z"] Nov 01 20:30:41.498052 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:41.498011 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf05cdcaf_b144_4c3d_b29b_2cb7d5a6f085.slice/crio-b23835ea8ffd51e39753cb1ba2551cabe2978b46c57b1f3ac048e4805fd5507a WatchSource:0}: Error finding container b23835ea8ffd51e39753cb1ba2551cabe2978b46c57b1f3ac048e4805fd5507a: Status 404 returned error can't find the container with id b23835ea8ffd51e39753cb1ba2551cabe2978b46c57b1f3ac048e4805fd5507a Nov 01 20:30:41.563504 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:41.563429 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:41.730174 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:41.730124 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:41.777692 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:41.777647 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-2lxrg"] Nov 01 20:30:41.786576 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:41.786536 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f213a88_8cb5_467b_8d21_9bc86a0654c9.slice/crio-9c215a148c70b1d9daea7436816efd58f428969f8cf645a832d599ccb6a95bd7 WatchSource:0}: Error finding container 9c215a148c70b1d9daea7436816efd58f428969f8cf645a832d599ccb6a95bd7: Status 404 returned error can't find the container with id 9c215a148c70b1d9daea7436816efd58f428969f8cf645a832d599ccb6a95bd7 Nov 01 20:30:42.181843 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:42.181808 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2lxrg" event={"ID":"8f213a88-8cb5-467b-8d21-9bc86a0654c9","Type":"ContainerStarted","Data":"9c215a148c70b1d9daea7436816efd58f428969f8cf645a832d599ccb6a95bd7"} Nov 01 20:30:42.182903 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:42.182879 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-smp9z" event={"ID":"f05cdcaf-b144-4c3d-b29b-2cb7d5a6f085","Type":"ContainerStarted","Data":"b23835ea8ffd51e39753cb1ba2551cabe2978b46c57b1f3ac048e4805fd5507a"} Nov 01 20:30:42.698586 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:42.698537 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/970d71e5-f861-4277-8fa5-6ec740418e51-metrics-certs\") pod \"network-metrics-daemon-vxzfk\" (UID: \"970d71e5-f861-4277-8fa5-6ec740418e51\") " pod="openshift-multus/network-metrics-daemon-vxzfk" Nov 01 20:30:42.698780 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:42.698718 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 01 20:30:42.698844 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:42.698801 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/970d71e5-f861-4277-8fa5-6ec740418e51-metrics-certs podName:970d71e5-f861-4277-8fa5-6ec740418e51 nodeName:}" failed. No retries permitted until 2026-11-01 20:31:14.698777398 +0000 UTC m=+66.268264985 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/970d71e5-f861-4277-8fa5-6ec740418e51-metrics-certs") pod "network-metrics-daemon-vxzfk" (UID: "970d71e5-f861-4277-8fa5-6ec740418e51") : secret "metrics-daemon-secret" not found Nov 01 20:30:42.799665 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:42.799626 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b4f17a0-efd1-4024-a201-4078069b04e7-original-pull-secret\") pod \"global-pull-secret-syncer-xh6zg\" (UID: \"6b4f17a0-efd1-4024-a201-4078069b04e7\") " pod="kube-system/global-pull-secret-syncer-xh6zg" Nov 01 20:30:42.802445 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:42.802415 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b4f17a0-efd1-4024-a201-4078069b04e7-original-pull-secret\") pod \"global-pull-secret-syncer-xh6zg\" (UID: \"6b4f17a0-efd1-4024-a201-4078069b04e7\") " pod="kube-system/global-pull-secret-syncer-xh6zg" Nov 01 20:30:42.804297 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:42.804271 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xh6zg" Nov 01 20:30:42.901851 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:42.900905 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btdtr\" (UniqueName: \"kubernetes.io/projected/9ee67cbd-8115-4c48-b5cd-5299ed284bce-kube-api-access-btdtr\") pod \"network-check-target-ghh5z\" (UID: \"9ee67cbd-8115-4c48-b5cd-5299ed284bce\") " pod="openshift-network-diagnostics/network-check-target-ghh5z" Nov 01 20:30:42.903690 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:42.903665 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btdtr\" (UniqueName: \"kubernetes.io/projected/9ee67cbd-8115-4c48-b5cd-5299ed284bce-kube-api-access-btdtr\") pod \"network-check-target-ghh5z\" (UID: \"9ee67cbd-8115-4c48-b5cd-5299ed284bce\") " pod="openshift-network-diagnostics/network-check-target-ghh5z" Nov 01 20:30:43.085753 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:43.085719 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ghh5z" Nov 01 20:30:43.102031 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:43.101975 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bbb0eff1-ed70-4783-a255-374d8bf14687-metrics-tls\") pod \"dns-default-gtf4c\" (UID: \"bbb0eff1-ed70-4783-a255-374d8bf14687\") " pod="openshift-dns/dns-default-gtf4c" Nov 01 20:30:43.102206 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:43.102169 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 20:30:43.102270 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:43.102225 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbb0eff1-ed70-4783-a255-374d8bf14687-metrics-tls podName:bbb0eff1-ed70-4783-a255-374d8bf14687 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:47.102212306 +0000 UTC m=+38.671699874 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bbb0eff1-ed70-4783-a255-374d8bf14687-metrics-tls") pod "dns-default-gtf4c" (UID: "bbb0eff1-ed70-4783-a255-374d8bf14687") : secret "dns-default-metrics-tls" not found Nov 01 20:30:43.201945 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:43.201811 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xh6zg"] Nov 01 20:30:43.202427 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:43.202378 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82ca0c95-053f-4bb0-ab8d-1d8cbd531667-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4ssvj\" (UID: \"82ca0c95-053f-4bb0-ab8d-1d8cbd531667\") " pod="openshift-insights/insights-runtime-extractor-4ssvj" Nov 01 20:30:43.202553 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:43.202470 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82ca0c95-053f-4bb0-ab8d-1d8cbd531667-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4ssvj\" (UID: \"82ca0c95-053f-4bb0-ab8d-1d8cbd531667\") " pod="openshift-insights/insights-runtime-extractor-4ssvj" Nov 01 20:30:43.202618 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:43.202609 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 20:30:43.202678 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:43.202672 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82ca0c95-053f-4bb0-ab8d-1d8cbd531667-insights-runtime-extractor-tls podName:82ca0c95-053f-4bb0-ab8d-1d8cbd531667 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:47.202653387 +0000 UTC m=+38.772140972 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/82ca0c95-053f-4bb0-ab8d-1d8cbd531667-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4ssvj" (UID: "82ca0c95-053f-4bb0-ab8d-1d8cbd531667") : secret "insights-runtime-extractor-tls" not found Nov 01 20:30:43.203109 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:43.203089 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 01 20:30:43.203221 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:43.203143 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/82ca0c95-053f-4bb0-ab8d-1d8cbd531667-kube-rbac-proxy-cm podName:82ca0c95-053f-4bb0-ab8d-1d8cbd531667 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:47.203129282 +0000 UTC m=+38.772616866 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/82ca0c95-053f-4bb0-ab8d-1d8cbd531667-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4ssvj" (UID: "82ca0c95-053f-4bb0-ab8d-1d8cbd531667") : configmap "kube-rbac-proxy" not found Nov 01 20:30:43.213209 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:43.213180 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b4f17a0_efd1_4024_a201_4078069b04e7.slice/crio-13b4659d2b51036b64bba0b18fb853012f3254b7cc361c202cc9ccf550b1c794 WatchSource:0}: Error finding container 13b4659d2b51036b64bba0b18fb853012f3254b7cc361c202cc9ccf550b1c794: Status 404 returned error can't find the container with id 13b4659d2b51036b64bba0b18fb853012f3254b7cc361c202cc9ccf550b1c794 Nov 01 20:30:43.519562 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:43.519526 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ghh5z"] Nov 01 20:30:43.645684 ip-10-0-141-35 kubenswrapper[2580]: W1101 20:30:43.645640 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ee67cbd_8115_4c48_b5cd_5299ed284bce.slice/crio-b2b6a5bff028eef38e8bcee10f728a6050fc3dd726d71e053b20f7e31954114c WatchSource:0}: Error finding container b2b6a5bff028eef38e8bcee10f728a6050fc3dd726d71e053b20f7e31954114c: Status 404 returned error can't find the container with id b2b6a5bff028eef38e8bcee10f728a6050fc3dd726d71e053b20f7e31954114c Nov 01 20:30:44.190633 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:44.190589 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mdc46" event={"ID":"492a8e93-84b5-46d3-b6e4-4278f737249a","Type":"ContainerStarted","Data":"4027386b76f70b86ee13e8059f9f5a8c1d0472095313b5f3a545be005220566f"} Nov 01 20:30:44.192495 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:44.192463 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xh6zg" event={"ID":"6b4f17a0-efd1-4024-a201-4078069b04e7","Type":"ContainerStarted","Data":"13b4659d2b51036b64bba0b18fb853012f3254b7cc361c202cc9ccf550b1c794"} Nov 01 20:30:44.194194 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:44.194166 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2lxrg" event={"ID":"8f213a88-8cb5-467b-8d21-9bc86a0654c9","Type":"ContainerStarted","Data":"fbaf23003c36a0f516e887b1e783045170e0cf78b00d4cb266384d3ec6f1d0f5"} Nov 01 20:30:44.195894 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:44.195870 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ghh5z" event={"ID":"9ee67cbd-8115-4c48-b5cd-5299ed284bce","Type":"ContainerStarted","Data":"b2b6a5bff028eef38e8bcee10f728a6050fc3dd726d71e053b20f7e31954114c"} Nov 01 20:30:44.437469 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:44.437373 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2lxrg" podStartSLOduration=1.173257306 podStartE2EDuration="3.43735341s" totalImagesPullingTime="2.264096104s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:30:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:44.437355476 +0000 UTC m=+36.006843066" watchObservedRunningTime="2026-11-01 20:30:44.43735341 +0000 UTC m=+36.006841000" Nov 01 20:30:44.438009 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:44.437947 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mdc46" podStartSLOduration=1.904704365 podStartE2EDuration="5.437938993s" totalImagesPullingTime="3.533234628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:30:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:44.321641393 +0000 UTC m=+35.891128983" watchObservedRunningTime="2026-11-01 20:30:44.437938993 +0000 UTC m=+36.007426583" Nov 01 20:30:44.815716 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:44.815660 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:44.970231 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:44.970164 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:46.202657 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:46.202581 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-smp9z" event={"ID":"f05cdcaf-b144-4c3d-b29b-2cb7d5a6f085","Type":"ContainerStarted","Data":"82111c92aaf76df89e48b71f0da5f42b12e48f3d183c8191ed29899465f1fc36"} Nov 01 20:30:46.317565 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:46.317498 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-smp9z" podStartSLOduration=2.660655336 podStartE2EDuration="6.317479296s" totalImagesPullingTime="3.65682396s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:30:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:46.316511562 +0000 UTC m=+37.885999177" watchObservedRunningTime="2026-11-01 20:30:46.317479296 +0000 UTC m=+37.886966888" Nov 01 20:30:46.604983 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:46.604942 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:46.765587 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:46.765478 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:46.934486 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:46.925507 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:47.135197 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:47.135162 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bbb0eff1-ed70-4783-a255-374d8bf14687-metrics-tls\") pod \"dns-default-gtf4c\" (UID: \"bbb0eff1-ed70-4783-a255-374d8bf14687\") " pod="openshift-dns/dns-default-gtf4c" Nov 01 20:30:47.135372 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:47.135333 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 20:30:47.135430 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:47.135399 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbb0eff1-ed70-4783-a255-374d8bf14687-metrics-tls podName:bbb0eff1-ed70-4783-a255-374d8bf14687 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:55.135380584 +0000 UTC m=+46.704868153 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bbb0eff1-ed70-4783-a255-374d8bf14687-metrics-tls") pod "dns-default-gtf4c" (UID: "bbb0eff1-ed70-4783-a255-374d8bf14687") : secret "dns-default-metrics-tls" not found Nov 01 20:30:47.211061 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:47.210431 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:47.236521 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:47.236488 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82ca0c95-053f-4bb0-ab8d-1d8cbd531667-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4ssvj\" (UID: \"82ca0c95-053f-4bb0-ab8d-1d8cbd531667\") " pod="openshift-insights/insights-runtime-extractor-4ssvj" Nov 01 20:30:47.236693 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:47.236571 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82ca0c95-053f-4bb0-ab8d-1d8cbd531667-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4ssvj\" (UID: \"82ca0c95-053f-4bb0-ab8d-1d8cbd531667\") " pod="openshift-insights/insights-runtime-extractor-4ssvj" Nov 01 20:30:47.236747 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:47.236697 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 20:30:47.236796 ip-10-0-141-35 kubenswrapper[2580]: E1101 20:30:47.236757 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82ca0c95-053f-4bb0-ab8d-1d8cbd531667-insights-runtime-extractor-tls podName:82ca0c95-053f-4bb0-ab8d-1d8cbd531667 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:55.236739984 +0000 UTC m=+46.806227560 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/82ca0c95-053f-4bb0-ab8d-1d8cbd531667-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4ssvj" (UID: "82ca0c95-053f-4bb0-ab8d-1d8cbd531667") : secret "insights-runtime-extractor-tls" not found Nov 01 20:30:47.237172 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:47.237109 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82ca0c95-053f-4bb0-ab8d-1d8cbd531667-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4ssvj\" (UID: \"82ca0c95-053f-4bb0-ab8d-1d8cbd531667\") " pod="openshift-insights/insights-runtime-extractor-4ssvj" Nov 01 20:30:47.366494 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:47.366450 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:47.562864 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:47.562836 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-w6nn2_04ec71b7-390e-4e40-abce-1c395a7a51ab/dns-node-resolver/0.log" Nov 01 20:30:47.630686 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:47.630655 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:47.788529 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:47.788496 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:48.090254 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:48.090177 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:48.162488 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:48.162461 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bztv7_e8ad39cd-9b6a-464c-aa75-1bfc54b23891/node-ca/0.log" Nov 01 20:30:48.209606 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:48.209574 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ghh5z" event={"ID":"9ee67cbd-8115-4c48-b5cd-5299ed284bce","Type":"ContainerStarted","Data":"23de8c762f697afcac8d12c0082aeb4f491957447e8eabe59e8328b82ceb54da"} Nov 01 20:30:48.209799 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:48.209725 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ghh5z" Nov 01 20:30:48.243889 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:48.243857 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:48.325130 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:48.325067 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ghh5z" podStartSLOduration=34.14171082 podStartE2EDuration="38.325048266s" totalImagesPullingTime="4.183337446s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:30:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:48.32411438 +0000 UTC m=+39.893601973" watchObservedRunningTime="2026-11-01 20:30:48.325048266 +0000 UTC m=+39.894535859" Nov 01 20:30:48.496867 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:48.496806 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:48.887424 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:48.887399 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:49.035032 ip-10-0-141-35 kubenswrapper[2580]: I1101 20:30:49.034962 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:49.152260 ip-10-0-141-35 systemd[1]: Stopping Kubernetes Kubelet... Nov 01 20:30:49.171470 ip-10-0-141-35 systemd[1]: kubelet.service: Deactivated successfully. Nov 01 20:30:49.171691 ip-10-0-141-35 systemd[1]: Stopped Kubernetes Kubelet. Nov 01 20:30:49.171782 ip-10-0-141-35 systemd[1]: kubelet.service: Consumed 2.540s CPU time. Nov 01 20:30:49.191776 ip-10-0-141-35 systemd[1]: Starting Kubernetes Kubelet... Nov 01 20:30:49.281524 ip-10-0-141-35 kubenswrapper[4484]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 20:30:49.281524 ip-10-0-141-35 kubenswrapper[4484]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 01 20:30:49.281524 ip-10-0-141-35 kubenswrapper[4484]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 20:30:49.281524 ip-10-0-141-35 kubenswrapper[4484]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 20:30:49.283710 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283622 4484 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:30:49.283710 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283710 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:30:49.283710 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283714 4484 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:30:49.283807 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283717 4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:30:49.283807 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283720 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:30:49.283807 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283725 4484 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:30:49.283807 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283728 4484 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:30:49.283807 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283730 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:30:49.283807 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283733 4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:30:49.283807 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283736 4484 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:30:49.283807 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283739 4484 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:30:49.283807 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283741 4484 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:30:49.283807 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283744 4484 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:30:49.283807 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283746 4484 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:30:49.283807 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283755 4484 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:30:49.283807 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283758 4484 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:30:49.283807 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283760 4484 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:30:49.283807 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283763 4484 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:30:49.283807 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283766 4484 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:30:49.283807 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283768 4484 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:30:49.283807 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283771 4484 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:30:49.283807 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283774 4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:30:49.283807 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283776 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:30:49.284331 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283779 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:30:49.284331 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283781 4484 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:30:49.284331 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283786 4484 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:30:49.284331 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283790 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:30:49.284331 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283793 4484 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:30:49.284331 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283796 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:30:49.284331 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283798 4484 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:30:49.284331 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283801 4484 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:30:49.284331 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283804 4484 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:30:49.284331 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283806 4484 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:30:49.284331 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283809 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:30:49.284331 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283812 4484 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:30:49.284331 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283815 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:30:49.284331 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283818 4484 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:30:49.284331 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283821 4484 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:30:49.284331 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283823 4484 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:30:49.284331 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283826 4484 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:30:49.284331 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283829 4484 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:30:49.284331 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283832 4484 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:30:49.284806 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283834 4484 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:30:49.284806 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283837 4484 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:30:49.284806 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283839 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:30:49.284806 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283842 4484 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:30:49.284806 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283844 4484 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:30:49.284806 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283847 4484 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:30:49.284806 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283850 4484 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:30:49.284806 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283852 4484 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:30:49.284806 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283855 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:30:49.284806 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283858 4484 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:30:49.284806 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283860 4484 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:30:49.284806 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283863 4484 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:30:49.284806 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283865 4484 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:30:49.284806 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283868 4484 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:30:49.284806 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283870 4484 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:30:49.284806 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283873 4484 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:30:49.284806 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283876 4484 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:30:49.284806 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283878 4484 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:30:49.284806 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283881 4484 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:30:49.284806 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283883 4484 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:30:49.285305 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283886 4484 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:30:49.285305 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283888 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:30:49.285305 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283891 4484 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:30:49.285305 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283894 4484 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:30:49.285305 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283897 4484 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:30:49.285305 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283899 4484 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:30:49.285305 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283902 4484 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:30:49.285305 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283905 4484 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:30:49.285305 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283910 4484 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:30:49.285305 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283913 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:30:49.285305 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283916 4484 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:30:49.285305 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283919 4484 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:30:49.285305 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283922 4484 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:30:49.285305 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283924 4484 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:30:49.285305 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283927 4484 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:30:49.285305 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283930 4484 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:30:49.285305 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283933 4484 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:30:49.285305 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283935 4484 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:30:49.285305 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283939 4484 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:30:49.285763 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283941 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:30:49.285763 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283944 4484 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:30:49.285763 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283946 4484 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:30:49.285763 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283949 4484 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:30:49.285763 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283962 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:30:49.285763 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283965 4484 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:30:49.285763 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283967 4484 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:30:49.285763 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283970 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:30:49.285763 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283973 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:30:49.285763 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283976 4484 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:30:49.285763 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283978 4484 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:30:49.285763 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283981 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:30:49.285763 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283984 4484 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:30:49.285763 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283986 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:30:49.285763 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.283989 4484 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:30:49.285763 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284004 4484 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:30:49.285763 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284008 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:30:49.285763 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284011 4484 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:30:49.285763 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284014 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:30:49.286233 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284017 4484 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:30:49.286233 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284019 4484 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:30:49.286233 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284022 4484 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:30:49.286233 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284024 4484 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:30:49.286233 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284027 4484 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:30:49.286233 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284030 4484 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:30:49.286233 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284822 4484 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:30:49.286233 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284829 4484 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:30:49.286233 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284832 4484 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:30:49.286233 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284835 4484 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:30:49.286233 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284838 4484 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:30:49.286233 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284841 4484 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:30:49.286233 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284843 4484 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:30:49.286233 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284846 4484 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:30:49.286233 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284850 4484 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:30:49.286233 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284852 4484 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:30:49.286233 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284855 4484 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:30:49.286233 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284859 4484 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:30:49.286233 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284862 4484 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:30:49.286233 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284864 4484 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:30:49.286761 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284867 4484 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:30:49.286761 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284869 4484 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:30:49.286761 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284872 4484 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:30:49.286761 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284874 4484 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:30:49.286761 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284877 4484 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:30:49.286761 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284880 4484 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:30:49.286761 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284882 4484 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:30:49.286761 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284885 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:30:49.286761 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284888 4484 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:30:49.286761 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284890 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:30:49.286761 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284893 4484 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:30:49.286761 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284896 4484 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:30:49.286761 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284898 4484 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:30:49.286761 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284901 4484 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:30:49.286761 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284903 4484 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:30:49.286761 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284906 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:30:49.286761 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284908 4484 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:30:49.286761 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284911 4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:30:49.286761 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284913 4484 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:30:49.286761 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284916 4484 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:30:49.287262 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284918 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:30:49.287262 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284921 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:30:49.287262 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284924 4484 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:30:49.287262 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284926 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:30:49.287262 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284929 4484 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:30:49.287262 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284931 4484 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:30:49.287262 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284934 4484 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:30:49.287262 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284937 4484 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:30:49.287262 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284939 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:30:49.287262 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284942 4484 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:30:49.287262 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284944 4484 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:30:49.287262 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284947 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:30:49.287262 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284949 4484 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:30:49.287262 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284952 4484 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:30:49.287262 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284954 4484 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:30:49.287262 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284957 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:30:49.287262 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284959 4484 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:30:49.287262 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284962 4484 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:30:49.287262 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284964 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:30:49.287720 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284966 4484 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:30:49.287720 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284969 4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:30:49.287720 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284973 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:30:49.287720 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284975 4484 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:30:49.287720 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284978 4484 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:30:49.287720 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284981 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:30:49.287720 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.284983 4484 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:30:49.287720 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285008 4484 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:30:49.287720 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285011 4484 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:30:49.287720 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285013 4484 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:30:49.287720 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285017 4484 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:30:49.287720 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285021 4484 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:30:49.287720 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285025 4484 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:30:49.287720 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285029 4484 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:30:49.287720 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285032 4484 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:30:49.287720 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285036 4484 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:30:49.287720 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285038 4484 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:30:49.287720 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285041 4484 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:30:49.287720 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285044 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:30:49.288242 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285046 4484 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:30:49.288242 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285052 4484 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:30:49.288242 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285055 4484 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:30:49.288242 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285058 4484 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:30:49.288242 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285061 4484 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:30:49.288242 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285063 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:30:49.288242 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285066 4484 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:30:49.288242 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285069 4484 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:30:49.288242 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285071 4484 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:30:49.288242 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285074 4484 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:30:49.288242 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285076 4484 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:30:49.288242 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285079 4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:30:49.288242 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285081 4484 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:30:49.288242 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285084 4484 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:30:49.288242 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285087 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:30:49.288242 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285090 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:30:49.288242 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285092 4484 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:30:49.288242 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285095 4484 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:30:49.288242 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285098 4484 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:30:49.288242 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285100 4484 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:30:49.288725 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285103 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:30:49.288725 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285106 4484 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:30:49.288725 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285110 4484 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:30:49.288725 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285113 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:30:49.288725 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285116 4484 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:30:49.288725 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285119 4484 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:30:49.288725 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285122 4484 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:30:49.288725 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285124 4484 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:30:49.288725 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285127 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:30:49.288725 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285129 4484 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:30:49.288725 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285132 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:30:49.288725 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285135 4484 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:30:49.288725 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285139 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:30:49.288725 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285141 4484 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:30:49.288725 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285646 4484 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:30:49.288725 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285651 4484 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:30:49.288725 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285654 4484 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:30:49.288725 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285657 4484 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:30:49.289172 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285659 4484 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:30:49.289172 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285662 4484 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:30:49.289172 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285665 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:30:49.289172 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285668 4484 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:30:49.289172 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285671 4484 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:30:49.289172 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285674 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:30:49.289172 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285676 4484 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:30:49.289172 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285679 4484 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:30:49.289172 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285685 4484 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:30:49.289172 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285688 4484 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:30:49.289172 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285690 4484 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:30:49.289172 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285693 4484 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:30:49.289172 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285696 4484 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:30:49.289172 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285699 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:30:49.289172 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285701 4484 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:30:49.289172 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285704 4484 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:30:49.289172 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285706 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:30:49.289172 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285709 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:30:49.289172 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285712 4484 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:30:49.289683 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285714 4484 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:30:49.289683 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285717 4484 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:30:49.289683 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285720 4484 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:30:49.289683 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285722 4484 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:30:49.289683 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285725 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:30:49.289683 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285727 4484 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:30:49.289683 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285730 4484 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:30:49.289683 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285733 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:30:49.289683 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285735 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:30:49.289683 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285739 4484 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:30:49.289683 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285741 4484 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:30:49.289683 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285744 4484 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:30:49.289683 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285746 4484 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:30:49.289683 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285758 4484 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:30:49.289683 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285762 4484 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:30:49.289683 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285765 4484 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:30:49.289683 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285768 4484 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:30:49.289683 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285771 4484 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:30:49.289683 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285774 4484 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:30:49.289683 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285776 4484 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:30:49.290186 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285779 4484 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:30:49.290186 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285784 4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:30:49.290186 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285787 4484 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:30:49.290186 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285790 4484 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:30:49.290186 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285792 4484 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:30:49.290186 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285795 4484 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:30:49.290186 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285798 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:30:49.290186 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285801 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:30:49.290186 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285803 4484 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:30:49.290186 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285806 4484 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:30:49.290186 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285809 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:30:49.290186 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285811 4484 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:30:49.290186 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285814 4484 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:30:49.290186 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285816 4484 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:30:49.290186 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285819 4484 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:30:49.290186 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285822 4484 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:30:49.290186 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285824 4484 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:30:49.290186 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285827 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:30:49.290186 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285830 4484 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:30:49.290186 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285832 4484 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:30:49.290752 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285835 4484 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:30:49.290752 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285837 4484 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:30:49.290752 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285840 4484 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:30:49.290752 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285844 4484 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:30:49.290752 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285848 4484 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:30:49.290752 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285850 4484 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:30:49.290752 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285854 4484 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:30:49.290752 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285859 4484 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:30:49.290752 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285863 4484 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:30:49.290752 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285866 4484 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:30:49.290752 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285869 4484 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:30:49.290752 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285872 4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:30:49.290752 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285875 4484 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:30:49.290752 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285879 4484 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:30:49.290752 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285881 4484 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:30:49.290752 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285884 4484 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:30:49.290752 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285886 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:30:49.290752 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285889 4484 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:30:49.290752 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285892 4484 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:30:49.291651 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285894 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:30:49.291651 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285897 4484 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:30:49.291651 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285900 4484 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:30:49.291651 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285902 4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:30:49.291651 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285905 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:30:49.291651 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285907 4484 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:30:49.291651 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285910 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:30:49.291651 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285912 4484 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:30:49.291651 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285915 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:30:49.291651 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285918 4484 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:30:49.291651 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285921 4484 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:30:49.291651 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285923 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:30:49.291651 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285925 4484 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:30:49.291651 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285928 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:30:49.291651 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285931 4484 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:30:49.291651 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285933 4484 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:30:49.291651 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285936 4484 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:30:49.291651 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285939 4484 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:30:49.291651 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285942 4484 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:30:49.291651 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285946 4484 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:30:49.292437 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285949 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:30:49.292437 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285951 4484 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:30:49.292437 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285954 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:30:49.292437 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.285956 4484 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:30:49.292437 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286066 4484 flags.go:64] FLAG: --address="0.0.0.0" Nov 01 20:30:49.292437 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286073 4484 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 01 20:30:49.292437 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286082 4484 flags.go:64] FLAG: --anonymous-auth="true" Nov 01 20:30:49.292437 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286087 4484 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 01 20:30:49.292437 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286092 4484 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 01 20:30:49.292437 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286096 4484 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 01 20:30:49.292437 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286100 4484 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 01 20:30:49.292437 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286105 4484 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 01 20:30:49.292437 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286108 4484 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 01 20:30:49.292437 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286111 4484 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 01 20:30:49.292437 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286115 4484 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 01 20:30:49.292437 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286119 4484 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 01 20:30:49.292437 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286122 4484 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 01 20:30:49.292437 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286125 4484 flags.go:64] FLAG: --cgroup-root="" Nov 01 20:30:49.292437 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286129 4484 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 01 20:30:49.292437 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286132 4484 flags.go:64] FLAG: --client-ca-file="" Nov 01 20:30:49.292437 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286135 4484 flags.go:64] FLAG: --cloud-provider="external" Nov 01 20:30:49.292437 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286138 4484 flags.go:64] FLAG: --cluster-dns="[]" Nov 01 20:30:49.292437 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286142 4484 flags.go:64] FLAG: --cluster-domain="" Nov 01 20:30:49.293161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286145 4484 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 01 20:30:49.293161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286148 4484 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 01 20:30:49.293161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286152 4484 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 01 20:30:49.293161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286155 4484 flags.go:64] FLAG: --container-log-max-files="5" Nov 01 20:30:49.293161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286159 4484 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 01 20:30:49.293161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286162 4484 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 01 20:30:49.293161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286166 4484 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 01 20:30:49.293161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286169 4484 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 01 20:30:49.293161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286174 4484 flags.go:64] FLAG: --contention-profiling="false" Nov 01 20:30:49.293161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286177 4484 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 01 20:30:49.293161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286180 4484 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 01 20:30:49.293161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286183 4484 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 01 20:30:49.293161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286187 4484 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 01 20:30:49.293161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286191 4484 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 01 20:30:49.293161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286195 4484 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 01 20:30:49.293161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286199 4484 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 01 20:30:49.293161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286202 4484 flags.go:64] FLAG: --enable-load-reader="false" Nov 01 20:30:49.293161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286206 4484 flags.go:64] FLAG: --enable-server="true" Nov 01 20:30:49.293161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286209 4484 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 01 20:30:49.293161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286213 4484 flags.go:64] FLAG: --event-burst="100" Nov 01 20:30:49.293161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286216 4484 flags.go:64] FLAG: --event-qps="50" Nov 01 20:30:49.293161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286219 4484 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 01 20:30:49.293161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286222 4484 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 01 20:30:49.293161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286226 4484 flags.go:64] FLAG: --eviction-hard="" Nov 01 20:30:49.293823 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286230 4484 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 01 20:30:49.293823 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286234 4484 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 01 20:30:49.293823 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286238 4484 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 01 20:30:49.293823 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286241 4484 flags.go:64] FLAG: --eviction-soft="" Nov 01 20:30:49.293823 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286244 4484 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 01 20:30:49.293823 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286248 4484 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 01 20:30:49.293823 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286251 4484 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 01 20:30:49.293823 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286254 4484 flags.go:64] FLAG: --experimental-mounter-path="" Nov 01 20:30:49.293823 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286257 4484 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 01 20:30:49.293823 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286260 4484 flags.go:64] FLAG: --fail-swap-on="true" Nov 01 20:30:49.293823 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286263 4484 flags.go:64] FLAG: --feature-gates="" Nov 01 20:30:49.293823 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286270 4484 flags.go:64] FLAG: --file-check-frequency="20s" Nov 01 20:30:49.293823 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286273 4484 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 01 20:30:49.293823 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286276 4484 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 01 20:30:49.293823 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286279 4484 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 01 20:30:49.293823 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286283 4484 flags.go:64] FLAG: --healthz-port="10248" Nov 01 20:30:49.293823 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286288 4484 flags.go:64] FLAG: --help="false" Nov 01 20:30:49.293823 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286291 4484 flags.go:64] FLAG: --hostname-override="ip-10-0-141-35.us-west-1.compute.internal" Nov 01 20:30:49.293823 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286294 4484 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 01 20:30:49.293823 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286297 4484 flags.go:64] FLAG: --http-check-frequency="20s" Nov 01 20:30:49.293823 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286300 4484 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 01 20:30:49.293823 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286304 4484 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 01 20:30:49.293823 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286308 4484 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 01 20:30:49.294419 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286313 4484 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 01 20:30:49.294419 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286316 4484 flags.go:64] FLAG: --image-service-endpoint="" Nov 01 20:30:49.294419 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286319 4484 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 01 20:30:49.294419 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286322 4484 flags.go:64] FLAG: --kube-api-burst="100" Nov 01 20:30:49.294419 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286325 4484 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 01 20:30:49.294419 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286328 4484 flags.go:64] FLAG: --kube-api-qps="50" Nov 01 20:30:49.294419 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286331 4484 flags.go:64] FLAG: --kube-reserved="" Nov 01 20:30:49.294419 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286334 4484 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 01 20:30:49.294419 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286337 4484 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 01 20:30:49.294419 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286340 4484 flags.go:64] FLAG: --kubelet-cgroups="" Nov 01 20:30:49.294419 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286343 4484 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 01 20:30:49.294419 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286346 4484 flags.go:64] FLAG: --lock-file="" Nov 01 20:30:49.294419 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286349 4484 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 01 20:30:49.294419 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286352 4484 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 01 20:30:49.294419 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286355 4484 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 01 20:30:49.294419 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286360 4484 flags.go:64] FLAG: --log-json-split-stream="false" Nov 01 20:30:49.294419 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286363 4484 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 01 20:30:49.294419 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286366 4484 flags.go:64] FLAG: --log-text-split-stream="false" Nov 01 20:30:49.294419 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286369 4484 flags.go:64] FLAG: --logging-format="text" Nov 01 20:30:49.294419 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286372 4484 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 01 20:30:49.294419 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286376 4484 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 01 20:30:49.294419 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286379 4484 flags.go:64] FLAG: --manifest-url="" Nov 01 20:30:49.294419 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286381 4484 flags.go:64] FLAG: --manifest-url-header="" Nov 01 20:30:49.294419 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286386 4484 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 01 20:30:49.294419 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286389 4484 flags.go:64] FLAG: --max-open-files="1000000" Nov 01 20:30:49.295092 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286396 4484 flags.go:64] FLAG: --max-pods="110" Nov 01 20:30:49.295092 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286400 4484 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 01 20:30:49.295092 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286403 4484 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 01 20:30:49.295092 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286406 4484 flags.go:64] FLAG: --memory-manager-policy="None" Nov 01 20:30:49.295092 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286409 4484 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 01 20:30:49.295092 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286412 4484 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 01 20:30:49.295092 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286415 4484 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 01 20:30:49.295092 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286420 4484 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 01 20:30:49.295092 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286427 4484 flags.go:64] FLAG: --node-status-max-images="50" Nov 01 20:30:49.295092 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286430 4484 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 01 20:30:49.295092 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286433 4484 flags.go:64] FLAG: --oom-score-adj="-999" Nov 01 20:30:49.295092 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286436 4484 flags.go:64] FLAG: --pod-cidr="" Nov 01 20:30:49.295092 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286440 4484 flags.go:64] FLAG: --pod-manifest-path="" Nov 01 20:30:49.295092 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286443 4484 flags.go:64] FLAG: --pod-max-pids="-1" Nov 01 20:30:49.295092 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286446 4484 flags.go:64] FLAG: --pods-per-core="0" Nov 01 20:30:49.295092 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286449 4484 flags.go:64] FLAG: --port="10250" Nov 01 20:30:49.295092 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286452 4484 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 01 20:30:49.295092 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286455 4484 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0a39cb5957849d0db" Nov 01 20:30:49.295092 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286459 4484 flags.go:64] FLAG: --qos-reserved="" Nov 01 20:30:49.295092 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286462 4484 flags.go:64] FLAG: --read-only-port="10255" Nov 01 20:30:49.295092 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286465 4484 flags.go:64] FLAG: --register-node="true" Nov 01 20:30:49.295092 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286468 4484 flags.go:64] FLAG: --register-with-taints="" Nov 01 20:30:49.295092 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286472 4484 flags.go:64] FLAG: --registry-burst="10" Nov 01 20:30:49.295092 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286475 4484 flags.go:64] FLAG: --registry-qps="5" Nov 01 20:30:49.295092 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286477 4484 flags.go:64] FLAG: --reserved-cpus="" Nov 01 20:30:49.295092 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286480 4484 flags.go:64] FLAG: --reserved-memory="" Nov 01 20:30:49.295801 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286484 4484 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 01 20:30:49.295801 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286487 4484 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 01 20:30:49.295801 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286491 4484 flags.go:64] FLAG: --rotate-certificates="false" Nov 01 20:30:49.295801 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286494 4484 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 01 20:30:49.295801 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286497 4484 flags.go:64] FLAG: --runonce="false" Nov 01 20:30:49.295801 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286500 4484 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 01 20:30:49.295801 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286504 4484 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 01 20:30:49.295801 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286508 4484 flags.go:64] FLAG: --seccomp-default="false" Nov 01 20:30:49.295801 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286511 4484 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 01 20:30:49.295801 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286514 4484 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 01 20:30:49.295801 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286517 4484 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 01 20:30:49.295801 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286521 4484 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 01 20:30:49.295801 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286524 4484 flags.go:64] FLAG: --storage-driver-password="root" Nov 01 20:30:49.295801 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286528 4484 flags.go:64] FLAG: --storage-driver-secure="false" Nov 01 20:30:49.295801 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286531 4484 flags.go:64] FLAG: --storage-driver-table="stats" Nov 01 20:30:49.295801 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286534 4484 flags.go:64] FLAG: --storage-driver-user="root" Nov 01 20:30:49.295801 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286537 4484 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 01 20:30:49.295801 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286540 4484 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 01 20:30:49.295801 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286543 4484 flags.go:64] FLAG: --system-cgroups="" Nov 01 20:30:49.295801 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286546 4484 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 01 20:30:49.295801 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286551 4484 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 01 20:30:49.295801 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286554 4484 flags.go:64] FLAG: --tls-cert-file="" Nov 01 20:30:49.295801 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286566 4484 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 01 20:30:49.295801 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286571 4484 flags.go:64] FLAG: --tls-min-version="" Nov 01 20:30:49.295801 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286574 4484 flags.go:64] FLAG: --tls-private-key-file="" Nov 01 20:30:49.296462 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286577 4484 flags.go:64] FLAG: --topology-manager-policy="none" Nov 01 20:30:49.296462 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286580 4484 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 01 20:30:49.296462 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286583 4484 flags.go:64] FLAG: --topology-manager-scope="container" Nov 01 20:30:49.296462 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286586 4484 flags.go:64] FLAG: --v="2" Nov 01 20:30:49.296462 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286591 4484 flags.go:64] FLAG: --version="false" Nov 01 20:30:49.296462 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286595 4484 flags.go:64] FLAG: --vmodule="" Nov 01 20:30:49.296462 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286599 4484 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 01 20:30:49.296462 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.286602 4484 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 01 20:30:49.296462 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286743 4484 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:30:49.296462 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286746 4484 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:30:49.296462 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286749 4484 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:30:49.296462 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286752 4484 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:30:49.296462 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286754 4484 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:30:49.296462 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286761 4484 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:30:49.296462 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286764 4484 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:30:49.296462 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286768 4484 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:30:49.296462 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286771 4484 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:30:49.296462 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286773 4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:30:49.296462 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286776 4484 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:30:49.296462 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286779 4484 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:30:49.296462 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286782 4484 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:30:49.296462 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286785 4484 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:30:49.297023 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286788 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:30:49.297023 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286790 4484 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:30:49.297023 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286793 4484 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:30:49.297023 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286796 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:30:49.297023 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286798 4484 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:30:49.297023 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286801 4484 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:30:49.297023 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286803 4484 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:30:49.297023 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286806 4484 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:30:49.297023 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286808 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:30:49.297023 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286811 4484 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:30:49.297023 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286813 4484 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:30:49.297023 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286816 4484 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:30:49.297023 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286818 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:30:49.297023 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286821 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:30:49.297023 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286823 4484 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:30:49.297023 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286826 4484 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:30:49.297023 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286829 4484 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:30:49.297023 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286831 4484 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:30:49.297023 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286834 4484 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:30:49.297023 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286837 4484 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:30:49.297647 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286840 4484 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:30:49.297647 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286844 4484 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:30:49.297647 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286847 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:30:49.297647 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286851 4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:30:49.297647 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286854 4484 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:30:49.297647 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286856 4484 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:30:49.297647 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286859 4484 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:30:49.297647 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286862 4484 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:30:49.297647 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286864 4484 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:30:49.297647 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286867 4484 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:30:49.297647 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286870 4484 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:30:49.297647 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286872 4484 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:30:49.297647 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286875 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:30:49.297647 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286878 4484 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:30:49.297647 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286880 4484 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:30:49.297647 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286883 4484 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:30:49.297647 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286885 4484 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:30:49.297647 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286888 4484 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:30:49.297647 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286890 4484 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:30:49.297647 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286893 4484 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:30:49.298535 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286895 4484 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:30:49.298535 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286898 4484 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:30:49.298535 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286900 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:30:49.298535 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286903 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:30:49.298535 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286906 4484 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:30:49.298535 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286908 4484 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:30:49.298535 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286911 4484 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:30:49.298535 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286913 4484 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:30:49.298535 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286916 4484 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:30:49.298535 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286918 4484 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:30:49.298535 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286921 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:30:49.298535 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286923 4484 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:30:49.298535 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286926 4484 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:30:49.298535 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286928 4484 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:30:49.298535 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286931 4484 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:30:49.298535 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286935 4484 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:30:49.298535 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286937 4484 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:30:49.298535 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286940 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:30:49.298535 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286943 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:30:49.299241 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286946 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:30:49.299241 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286948 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:30:49.299241 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286951 4484 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:30:49.299241 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286953 4484 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:30:49.299241 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286957 4484 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:30:49.299241 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286961 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:30:49.299241 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286964 4484 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:30:49.299241 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286967 4484 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:30:49.299241 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286969 4484 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:30:49.299241 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286972 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:30:49.299241 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286974 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:30:49.299241 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286977 4484 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:30:49.299241 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286979 4484 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:30:49.299241 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286982 4484 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:30:49.299241 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286984 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:30:49.299241 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286987 4484 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:30:49.299241 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.286989 4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:30:49.299241 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287004 4484 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:30:49.299241 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287007 4484 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:30:49.299797 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287009 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:30:49.299797 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287012 4484 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:30:49.299797 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287015 4484 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:30:49.299797 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287017 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:30:49.299797 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287020 4484 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:30:49.299797 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287022 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:30:49.299797 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287025 4484 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:30:49.299797 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287028 4484 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:30:49.299797 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287030 4484 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:30:49.299797 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287035 4484 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:30:49.299797 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287037 4484 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:30:49.299797 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287040 4484 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:30:49.299797 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287042 4484 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:30:49.299797 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287045 4484 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:30:49.299797 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.287074 4484 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 20:30:49.299797 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287112 4484 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:30:49.300226 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287117 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:30:49.300226 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287120 4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:30:49.300226 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287122 4484 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:30:49.300226 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287125 4484 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:30:49.300226 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287128 4484 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:30:49.300226 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287130 4484 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:30:49.300226 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287133 4484 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:30:49.300226 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287136 4484 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:30:49.300226 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287138 4484 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:30:49.300226 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287141 4484 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:30:49.300226 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287143 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:30:49.300226 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287146 4484 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:30:49.300226 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287148 4484 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:30:49.300226 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287151 4484 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:30:49.300226 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287153 4484 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:30:49.300226 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287156 4484 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:30:49.300226 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287159 4484 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:30:49.300226 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287161 4484 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:30:49.300226 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287164 4484 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:30:49.300226 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287166 4484 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:30:49.300702 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287169 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:30:49.300702 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287171 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:30:49.300702 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287174 4484 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:30:49.300702 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287176 4484 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:30:49.300702 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287180 4484 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:30:49.300702 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287184 4484 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:30:49.300702 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287186 4484 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:30:49.300702 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287188 4484 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:30:49.300702 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287191 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:30:49.300702 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287197 4484 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:30:49.300702 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287200 4484 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:30:49.300702 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287203 4484 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:30:49.300702 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287206 4484 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:30:49.300702 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287208 4484 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:30:49.300702 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287211 4484 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:30:49.300702 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287214 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:30:49.300702 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287216 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:30:49.300702 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287219 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:30:49.300702 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287221 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:30:49.301227 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287224 4484 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:30:49.301227 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287227 4484 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:30:49.301227 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287231 4484 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:30:49.301227 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287235 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:30:49.301227 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287238 4484 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:30:49.301227 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287241 4484 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:30:49.301227 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287244 4484 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:30:49.301227 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287247 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:30:49.301227 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287250 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:30:49.301227 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287253 4484 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:30:49.301227 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287255 4484 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:30:49.301227 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287258 4484 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:30:49.301227 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287260 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:30:49.301227 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287263 4484 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:30:49.301227 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287266 4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:30:49.301227 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287269 4484 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:30:49.301227 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287272 4484 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:30:49.301227 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287274 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:30:49.301227 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287278 4484 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:30:49.301827 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287280 4484 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:30:49.301827 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287283 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:30:49.301827 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287285 4484 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:30:49.301827 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287289 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:30:49.301827 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287292 4484 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:30:49.301827 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287294 4484 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:30:49.301827 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287297 4484 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:30:49.301827 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287300 4484 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:30:49.301827 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287302 4484 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:30:49.301827 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287305 4484 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:30:49.301827 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287308 4484 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:30:49.301827 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287310 4484 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:30:49.301827 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287313 4484 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:30:49.301827 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287315 4484 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:30:49.301827 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287318 4484 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:30:49.301827 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287320 4484 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:30:49.301827 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287323 4484 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:30:49.301827 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287326 4484 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:30:49.301827 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287328 4484 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:30:49.301827 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287331 4484 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:30:49.302358 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287333 4484 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:30:49.302358 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287336 4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:30:49.302358 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287339 4484 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:30:49.302358 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287341 4484 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:30:49.302358 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287344 4484 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:30:49.302358 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287347 4484 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:30:49.302358 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287350 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:30:49.302358 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287352 4484 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:30:49.302358 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287355 4484 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:30:49.302358 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287357 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:30:49.302358 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287360 4484 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:30:49.302358 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287362 4484 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:30:49.302358 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287365 4484 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:30:49.302358 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287368 4484 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:30:49.302358 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287370 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:30:49.302358 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287374 4484 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:30:49.302358 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287377 4484 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:30:49.302358 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287379 4484 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:30:49.302358 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287382 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:30:49.302358 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287385 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:30:49.302856 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287388 4484 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:30:49.302856 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287391 4484 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:30:49.302856 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287393 4484 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:30:49.302856 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287396 4484 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:30:49.302856 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287399 4484 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:30:49.302856 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287401 4484 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:30:49.302856 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.287406 4484 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:30:49.302856 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.292969 4484 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 01 20:30:49.302856 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.292987 4484 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 01 20:30:49.302856 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.293031 4484 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 01 20:30:49.302856 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.293037 4484 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 01 20:30:49.302856 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293097 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:30:49.302856 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293102 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:30:49.302856 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293106 4484 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:30:49.302856 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293109 4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:30:49.302856 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293112 4484 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:30:49.302856 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293115 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:30:49.302856 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293117 4484 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:30:49.302856 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293120 4484 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:30:49.303327 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293123 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:30:49.303327 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293126 4484 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:30:49.303327 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293128 4484 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:30:49.303327 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293131 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:30:49.303327 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293134 4484 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:30:49.303327 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293136 4484 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:30:49.303327 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293139 4484 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:30:49.303327 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293142 4484 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:30:49.303327 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293145 4484 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:30:49.303327 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293147 4484 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:30:49.303327 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293150 4484 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:30:49.303327 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293153 4484 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:30:49.303327 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293155 4484 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:30:49.303327 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293158 4484 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:30:49.303327 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293161 4484 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:30:49.303327 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293164 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:30:49.303327 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293168 4484 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:30:49.303327 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293172 4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:30:49.303327 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293175 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:30:49.303783 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293178 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:30:49.303783 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293182 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:30:49.303783 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293185 4484 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:30:49.303783 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293188 4484 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:30:49.303783 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293191 4484 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:30:49.303783 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293194 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:30:49.303783 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293197 4484 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:30:49.303783 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293201 4484 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:30:49.303783 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293204 4484 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:30:49.303783 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293207 4484 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:30:49.303783 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293210 4484 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:30:49.303783 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293212 4484 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:30:49.303783 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293215 4484 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:30:49.303783 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293217 4484 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:30:49.303783 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293220 4484 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:30:49.303783 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293223 4484 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:30:49.303783 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293225 4484 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:30:49.303783 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293228 4484 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:30:49.303783 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293231 4484 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:30:49.304259 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293234 4484 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:30:49.304259 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293236 4484 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:30:49.304259 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293239 4484 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:30:49.304259 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293246 4484 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:30:49.304259 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293248 4484 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:30:49.304259 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293251 4484 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:30:49.304259 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293254 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:30:49.304259 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293256 4484 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:30:49.304259 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293259 4484 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:30:49.304259 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293262 4484 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:30:49.304259 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293265 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:30:49.304259 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293268 4484 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:30:49.304259 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293270 4484 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:30:49.304259 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293273 4484 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:30:49.304259 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293275 4484 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:30:49.304259 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293278 4484 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:30:49.304259 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293281 4484 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:30:49.304259 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293283 4484 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:30:49.304259 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293286 4484 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:30:49.304259 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293289 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:30:49.304751 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293291 4484 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:30:49.304751 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293295 4484 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:30:49.304751 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293297 4484 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:30:49.304751 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293300 4484 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:30:49.304751 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293302 4484 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:30:49.304751 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293305 4484 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:30:49.304751 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293307 4484 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:30:49.304751 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293310 4484 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:30:49.304751 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293313 4484 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:30:49.304751 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293315 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:30:49.304751 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293318 4484 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:30:49.304751 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293321 4484 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:30:49.304751 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293323 4484 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:30:49.304751 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293327 4484 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:30:49.304751 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293330 4484 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:30:49.304751 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293332 4484 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:30:49.304751 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293335 4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:30:49.304751 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293338 4484 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:30:49.304751 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293340 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:30:49.304751 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293343 4484 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:30:49.305351 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293345 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:30:49.305351 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293348 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:30:49.305351 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293350 4484 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:30:49.305351 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293353 4484 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:30:49.305351 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293355 4484 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:30:49.305351 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293358 4484 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:30:49.305351 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293360 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:30:49.305351 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293363 4484 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:30:49.305351 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293365 4484 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:30:49.305351 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293368 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:30:49.305351 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293370 4484 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:30:49.305351 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293373 4484 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:30:49.305351 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293375 4484 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:30:49.305351 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293378 4484 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:30:49.305351 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293381 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:30:49.305351 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293384 4484 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:30:49.305351 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293387 4484 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:30:49.305351 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293390 4484 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:30:49.305351 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293392 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:30:49.305810 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293395 4484 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:30:49.305810 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.293425 4484 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 20:30:49.305810 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293572 4484 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:30:49.305810 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293578 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:30:49.305810 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293581 4484 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:30:49.305810 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293583 4484 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:30:49.305810 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293586 4484 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:30:49.305810 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293590 4484 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:30:49.305810 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293592 4484 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:30:49.305810 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293595 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:30:49.305810 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293598 4484 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:30:49.305810 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293600 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:30:49.305810 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293603 4484 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:30:49.305810 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293605 4484 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:30:49.305810 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293608 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:30:49.306186 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293610 4484 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:30:49.306186 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293613 4484 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:30:49.306186 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293616 4484 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:30:49.306186 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293620 4484 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:30:49.306186 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293624 4484 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:30:49.306186 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293627 4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:30:49.306186 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293629 4484 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:30:49.306186 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293632 4484 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:30:49.306186 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293634 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:30:49.306186 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293637 4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:30:49.306186 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293639 4484 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:30:49.306186 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293642 4484 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:30:49.306186 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293645 4484 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:30:49.306186 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293647 4484 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:30:49.306186 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293650 4484 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:30:49.306186 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293653 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:30:49.306186 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293655 4484 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:30:49.306186 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293658 4484 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:30:49.306186 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293660 4484 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:30:49.306186 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293663 4484 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:30:49.306702 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293666 4484 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:30:49.306702 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293668 4484 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:30:49.306702 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293671 4484 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:30:49.306702 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293674 4484 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:30:49.306702 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293676 4484 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:30:49.306702 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293680 4484 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:30:49.306702 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293682 4484 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:30:49.306702 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293685 4484 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:30:49.306702 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293688 4484 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:30:49.306702 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293691 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:30:49.306702 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293693 4484 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:30:49.306702 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293696 4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:30:49.306702 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293699 4484 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:30:49.306702 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293701 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:30:49.306702 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293704 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:30:49.306702 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293706 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:30:49.306702 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293709 4484 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:30:49.306702 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293713 4484 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:30:49.306702 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293717 4484 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:30:49.307178 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293720 4484 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:30:49.307178 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293724 4484 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:30:49.307178 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293728 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:30:49.307178 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293732 4484 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:30:49.307178 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293735 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:30:49.307178 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293738 4484 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:30:49.307178 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293741 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:30:49.307178 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293744 4484 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:30:49.307178 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293746 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:30:49.307178 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293749 4484 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:30:49.307178 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293751 4484 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:30:49.307178 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293754 4484 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:30:49.307178 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293757 4484 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:30:49.307178 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293760 4484 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:30:49.307178 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293762 4484 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:30:49.307178 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293765 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:30:49.307178 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293768 4484 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:30:49.307178 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293770 4484 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:30:49.307178 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293773 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:30:49.307623 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293777 4484 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:30:49.307623 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293779 4484 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:30:49.307623 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293782 4484 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:30:49.307623 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293785 4484 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:30:49.307623 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293788 4484 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:30:49.307623 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293790 4484 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:30:49.307623 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293792 4484 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:30:49.307623 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293795 4484 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:30:49.307623 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293798 4484 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:30:49.307623 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293800 4484 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:30:49.307623 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293803 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:30:49.307623 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293805 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:30:49.307623 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293809 4484 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:30:49.307623 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293812 4484 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:30:49.307623 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293815 4484 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:30:49.307623 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293818 4484 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:30:49.307623 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293820 4484 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:30:49.307623 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293823 4484 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:30:49.307623 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293829 4484 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:30:49.308145 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293832 4484 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:30:49.308145 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293835 4484 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:30:49.308145 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293838 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:30:49.308145 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293841 4484 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:30:49.308145 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293843 4484 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:30:49.308145 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293846 4484 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:30:49.308145 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293849 4484 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:30:49.308145 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293851 4484 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:30:49.308145 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293854 4484 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:30:49.308145 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293856 4484 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:30:49.308145 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293859 4484 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:30:49.308145 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293861 4484 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:30:49.308145 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293865 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:30:49.308145 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293867 4484 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:30:49.308145 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293871 4484 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:30:49.308145 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293874 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:30:49.308536 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.293898 4484 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 20:30:49.308536 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293932 4484 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:30:49.308536 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293936 4484 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:30:49.308536 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293939 4484 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:30:49.308536 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293942 4484 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:30:49.308536 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293945 4484 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:30:49.308536 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293948 4484 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:30:49.308536 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293951 4484 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:30:49.308536 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293953 4484 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:30:49.308536 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293956 4484 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:30:49.308536 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293958 4484 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:30:49.308536 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293961 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:30:49.308536 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293963 4484 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:30:49.308536 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293966 4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:30:49.308536 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293969 4484 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:30:49.308536 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293971 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:30:49.308913 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293975 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:30:49.308913 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293977 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:30:49.308913 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293980 4484 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:30:49.308913 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293983 4484 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:30:49.308913 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293985 4484 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:30:49.308913 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293988 4484 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:30:49.308913 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.293990 4484 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:30:49.308913 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294008 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:30:49.308913 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294012 4484 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:30:49.308913 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294015 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:30:49.308913 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294019 4484 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:30:49.308913 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294023 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:30:49.308913 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294027 4484 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:30:49.308913 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294031 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:30:49.308913 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294034 4484 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:30:49.308913 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294039 4484 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:30:49.308913 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294043 4484 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:30:49.308913 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294047 4484 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:30:49.308913 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294051 4484 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:30:49.309439 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294055 4484 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:30:49.309439 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294059 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:30:49.309439 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294062 4484 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:30:49.309439 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294067 4484 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:30:49.309439 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294071 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:30:49.309439 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294076 4484 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:30:49.309439 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294079 4484 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:30:49.309439 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294081 4484 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:30:49.309439 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294084 4484 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:30:49.309439 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294087 4484 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:30:49.309439 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294089 4484 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:30:49.309439 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294092 4484 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:30:49.309439 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294095 4484 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:30:49.309439 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294097 4484 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:30:49.309439 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294100 4484 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:30:49.309439 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294103 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:30:49.309439 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294105 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:30:49.309439 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294110 4484 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:30:49.309439 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294114 4484 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:30:49.309991 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294117 4484 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:30:49.309991 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294120 4484 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:30:49.309991 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294123 4484 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:30:49.309991 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294126 4484 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:30:49.309991 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294129 4484 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:30:49.309991 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294132 4484 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:30:49.309991 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294134 4484 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:30:49.309991 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294137 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:30:49.309991 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294140 4484 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:30:49.309991 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294142 4484 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:30:49.309991 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294145 4484 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:30:49.309991 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294149 4484 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:30:49.309991 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294151 4484 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:30:49.309991 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294154 4484 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:30:49.309991 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294156 4484 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:30:49.309991 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294158 4484 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:30:49.309991 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294161 4484 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:30:49.309991 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294164 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:30:49.309991 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294166 4484 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:30:49.309991 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294168 4484 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:30:49.310495 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294171 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:30:49.310495 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294173 4484 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:30:49.310495 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294176 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:30:49.310495 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294179 4484 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:30:49.310495 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294181 4484 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:30:49.310495 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294184 4484 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:30:49.310495 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294186 4484 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:30:49.310495 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294189 4484 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:30:49.310495 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294191 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:30:49.310495 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294194 4484 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:30:49.310495 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294197 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:30:49.310495 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294199 4484 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:30:49.310495 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294202 4484 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:30:49.310495 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294204 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:30:49.310495 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294207 4484 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:30:49.310495 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294209 4484 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:30:49.310495 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294212 4484 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:30:49.310495 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294216 4484 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:30:49.310495 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294220 4484 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:30:49.310949 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294223 4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:30:49.310949 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294226 4484 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:30:49.310949 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294229 4484 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:30:49.310949 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294231 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:30:49.310949 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294234 4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:30:49.310949 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294237 4484 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:30:49.310949 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294239 4484 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:30:49.310949 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294242 4484 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:30:49.310949 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294244 4484 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:30:49.310949 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294248 4484 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:30:49.310949 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294251 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:30:49.310949 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294253 4484 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:30:49.310949 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294256 4484 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:30:49.310949 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:49.294258 4484 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:30:49.310949 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.294315 4484 server.go:957] "Client rotation is on, will bootstrap in background" Nov 01 20:30:49.310949 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.295342 4484 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 01 20:30:49.310949 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.295407 4484 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 01 20:30:49.310949 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.295616 4484 server.go:1014] "Starting client certificate rotation" Nov 01 20:30:49.310949 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.295750 4484 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 20:30:49.311426 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.295804 4484 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 20:25:09 +0000 UTC" deadline="2028-04-13 07:51:46.174057065 +0000 UTC" Nov 01 20:30:49.311426 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.295845 4484 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12683h20m56.878214903s" Nov 01 20:30:49.311426 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.296102 4484 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 20:30:49.311426 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.297320 4484 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 20:30:49.311426 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.299194 4484 log.go:25] "Validated CRI v1 runtime API" Nov 01 20:30:49.311426 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.301116 4484 log.go:25] "Validated CRI v1 image API" Nov 01 20:30:49.311426 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.301872 4484 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 01 20:30:49.311426 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.304026 4484 fs.go:135] Filesystem UUIDs: map[4a81d1fc-210d-4b90-b516-5f74051a24f7:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 a58926c8-7ad4-4602-9128-d2a090c34e44:/dev/nvme0n1p4] Nov 01 20:30:49.311648 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.304052 4484 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/030f4716dcaaa8462c101a58de0015dbb55dca028de259c1fc7c3915330d1b7e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/030f4716dcaaa8462c101a58de0015dbb55dca028de259c1fc7c3915330d1b7e/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/13b4659d2b51036b64bba0b18fb853012f3254b7cc361c202cc9ccf550b1c794/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/13b4659d2b51036b64bba0b18fb853012f3254b7cc361c202cc9ccf550b1c794/userdata/shm major:0 minor:219 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5160691e25c668734b4c024993d5bf4c3ed08d76e846e24ca73845637ad73521/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5160691e25c668734b4c024993d5bf4c3ed08d76e846e24ca73845637ad73521/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5882d5d1fac98d98c7525baa5d1d7ea2c0071b79c1c9f344d61fd84c7f11bfaf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5882d5d1fac98d98c7525baa5d1d7ea2c0071b79c1c9f344d61fd84c7f11bfaf/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6e9ed6520b34ab71b0c03cc82e133742536fd7ed40a5a40364132cc19eef6fbc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6e9ed6520b34ab71b0c03cc82e133742536fd7ed40a5a40364132cc19eef6fbc/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7fc4de5f427b94b5aeece9302a62a6fade2769408cde35df2e0058c848f3b483/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7fc4de5f427b94b5aeece9302a62a6fade2769408cde35df2e0058c848f3b483/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/80729d64d0dbb0313f04a42bdf9b3ace58e24fb0ebf40ae93be25b504c4ce195/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/80729d64d0dbb0313f04a42bdf9b3ace58e24fb0ebf40ae93be25b504c4ce195/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8d5ba769477249f45bb9343ec9a64905df58d817e34e232a4b8b906025665541/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8d5ba769477249f45bb9343ec9a64905df58d817e34e232a4b8b906025665541/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9c215a148c70b1d9daea7436816efd58f428969f8cf645a832d599ccb6a95bd7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9c215a148c70b1d9daea7436816efd58f428969f8cf645a832d599ccb6a95bd7/userdata/shm major:0 minor:212 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b23835ea8ffd51e39753cb1ba2551cabe2978b46c57b1f3ac048e4805fd5507a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b23835ea8ffd51e39753cb1ba2551cabe2978b46c57b1f3ac048e4805fd5507a/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b2b6a5bff028eef38e8bcee10f728a6050fc3dd726d71e053b20f7e31954114c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b2b6a5bff028eef38e8bcee10f728a6050fc3dd726d71e053b20f7e31954114c/userdata/shm major:0 minor:222 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dcd1501c6ab24fec2ddbec2a6a81381d321b3ca29979d2de8e25807600c2199d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dcd1501c6ab24fec2ddbec2a6a81381d321b3ca29979d2de8e25807600c2199d/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/df6c54825fb1d72061008ce2c740924d1b7e34f39b8db1676980e72afd1a970e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/df6c54825fb1d72061008ce2c740924d1b7e34f39b8db1676980e72afd1a970e/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ee503b738885a5321cca7fa7f3c6defd4071ab42689f058bea78da5918ac14e7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ee503b738885a5321cca7fa7f3c6defd4071ab42689f058bea78da5918ac14e7/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f35238c9dff3501e77884444e067b5978d544dd583eaace87752ad74535b250a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f35238c9dff3501e77884444e067b5978d544dd583eaace87752ad74535b250a/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/04ec71b7-390e-4e40-abce-1c395a7a51ab/volumes/kubernetes.io~projected/kube-api-access-jjfv4:{mountpoint:/var/lib/kubelet/pods/04ec71b7-390e-4e40-abce-1c395a7a51ab/volumes/kubernetes.io~projected/kube-api-access-jjfv4 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2288d739-5eb3-4186-a32e-187945267de1/volumes/kubernetes.io~projected/kube-api-access-px48q:{mountpoint:/var/lib/kubelet/pods/2288d739-5eb3-4186-a32e-187945267de1/volumes/kubernetes.io~projected/kube-api-access-px48q major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3fce64bd-a955-4e2c-8844-a25eaeb73862/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/3fce64bd-a955-4e2c-8844-a25eaeb73862/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3fce64bd-a955-4e2c-8844-a25eaeb73862/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/3fce64bd-a955-4e2c-8844-a25eaeb73862/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3fce64bd-a955-4e2c-8844-a25eaeb73862/volumes/kubernetes.io~projected/kube-api-access-d5pn9:{mountpoint:/var/lib/kubelet/pods/3fce64bd-a955-4e2c-8844-a25eaeb73862/volumes/kubernetes.io~projected/kube-api-access-d5pn9 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/492a8e93-84b5-46d3-b6e4-4278f737249a/volumes/kubernetes.io~projected/kube-api-access-sw4w8:{mountpoint:/var/lib/kubelet/pods/492a8e93-84b5-46d3-b6e4-4278f737249a/volumes/kubernetes.io~projected/kube-api-access-sw4w8 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/492a8e93-84b5-46d3-b6e4-4278f737249a/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/492a8e93-84b5-46d3-b6e4-4278f737249a/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5800567f-f0e1-4cf0-8338-75b09d2d1255/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/5800567f-f0e1-4cf0-8338-75b09d2d1255/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b4f17a0-efd1-4024-a201-4078069b04e7/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/6b4f17a0-efd1-4024-a201-4078069b04e7/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/82ca0c95-053f-4bb0-ab8d-1d8cbd531667/volumes/kubernetes.io~projected/kube-api-access-kplvv:{mountpoint:/var/lib/kubelet/pods/82ca0c95-053f-4bb0-ab8d-1d8cbd531667/volumes/kubernetes.io~projected/kube-api-access-kplvv major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f213a88-8cb5-467b-8d21-9bc86a0654c9/volumes/kubernetes.io~projected/kube-api-access-lhrjv:{mountpoint:/var/lib/kubelet/pods/8f213a88-8cb5-467b-8d21-9bc86a0654c9/volumes/kubernetes.io~projected/kube-api-access-lhrjv major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/90e25e61-9761-4042-9dd4-88e1a6f8aa3c/volumes/kubernetes.io~projected/kube-api-access-gsbqz:{mountpoint:/var/lib/kubelet/pods/90e25e61-9761-4042-9dd4-88e1a6f8aa3c/volumes/kubernetes.io~projected/kube-api-access-gsbqz major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/970d71e5-f861-4277-8fa5-6ec740418e51/volumes/kubernetes.io~projected/kube-api-access-6vm67:{mountpoint:/var/lib/kubelet/pods/970d71e5-f861-4277-8fa5-6ec740418e51/volumes/kubernetes.io~projected/kube-api-access-6vm67 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9ee67cbd-8115-4c48-b5cd-5299ed284bce/volumes/kubernetes.io~projected/kube-api-access-btdtr:{mountpoint:/var/lib/kubelet/pods/9ee67cbd-8115-4c48-b5cd-5299ed284bce/volumes/kubernetes.io~projected/kube-api-access-btdtr major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4342ddb-e06e-4d6d-9337-651349cf133f/volumes/kubernetes.io~projected/kube-api-access-mm9p9:{mountpoint:/var/lib/kubelet/pods/a4342ddb-e06e-4d6d-9337-651349cf133f/volumes/kubernetes.io~projected/kube-api-access-mm9p9 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bbb0eff1-ed70-4783-a255-374d8bf14687/volumes/kubernetes.io~projected/kube-api-access-74vvb:{mountpoint:/var/lib/kubelet/pods/bbb0eff1-ed70-4783-a255-374d8bf14687/volumes/kubernetes.io~projected/kube-api-access-74vvb major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427/volumes/kubernetes.io~projected/kube-api-access-vgl7f:{mountpoint:/var/lib/kubelet/pods/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427/volumes/kubernetes.io~projected/kube-api-access-vgl7f major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc/volumes/kubernetes.io~projected/kube-api-access-qzsrc:{mountpoint:/var/lib/kubelet/pods/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc/volumes/kubernetes.io~projected/kube-api-access-qzsrc major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e8ad39cd-9b6a-464c-aa75-1bfc54b23891/volumes/kubernetes.io~projected/kube-api-access-khz9n:{mountpoint:/var/lib/kubelet/pods/e8ad39cd-9b6a-464c-aa75-1bfc54b23891/volumes/kubernetes.io~projected/kube-api-access-khz9n major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f05cdcaf-b144-4c3d-b29b-2cb7d5a6f085/volumes/kubernetes.io~projected/kube-api-access-dhml7:{mountpoint:/var/lib/kubelet/pods/f05cdcaf-b144-4c3d-b29b-2cb7d5a6f085/volumes/kubernetes.io~projected/kube-api-access-dhml7 major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f05cdcaf-b144-4c3d-b29b-2cb7d5a6f085/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/f05cdcaf-b144-4c3d-b29b-2cb7d5a6f085/volumes/kubernetes.io~secret/signing-key major:0 minor:207 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/e7d952ba7bb3dcaa1cd0ebb7bc4c5fb98c57577aaab5ff3d77797032982d5504/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/b07518e3bd236297bf04a3e975abf22f27a8610c071daec67d8eb57fb3302e34/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/b0a5398a5fef207f17d368ed188d0fcb97b0482f9b6107b3cba8e92e3c7ec0f8/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/e607453e565ffa9d0fa11644ebd120894d573cc446b523cf2d5329fcd1efb598/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/b02d63d7a6e2c8d0e8008ddefb106b3889cf595388873ab30039df78bab5afb0/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/92368477ca436876621f9a1aa40c91f697dfe92254a39eec9fc7f58cae3aa28c/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/614f9daafb78376d3e2bf394cd604756408ce4aa6ec5128c2d59101e327bbf86/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/7e824c4dd995e31afe4e7b2350adbf7573ba155f6ad74320d51672d4b97ac52b/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/8298333ebe6d9175ea04853bbe9b97cbe7af11c94e9d7e5bef6af1e6d923bd49/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/1eb7cc43c380eb90691ff2db23c861172824aa3cefa369cf41c10241ed142c07/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/50dbe7d3c97775bbb22c614f71ba0f8844025ddaa09cb1fffec17c3fb4556716/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/39de9672d5bd5a6193c302954be42f6f1212b61140595e986ee672232cdf818b/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/97dfb0872175c58df9a9f9ea03da2a1bc61cbf6c744d1a3548b62dfb77eecc15/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/bf23f45fd5e568d8463c6411d64468e21712c22aa80a798c3f84b559f68d441a/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/7809cb017adad89ac7ef93755869112fa54aa582d8874b01a6013a1631086664/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/e04fa2862420fde40d7de60ee3f84f940773552134bae9348fd9466cf6d66a97/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/235f6cf16772a4817fb92e82e18c39fcd6f1a90147297afe1e2684ea54d2b5c2/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/5655825a783f4e58d6da81e964441debdb8bf78baf4709118dd310cd0aefee2b/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/acddc1d0f463348d560a290beaad1687f86369c9063e44b99ba5b580668f9add/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/7106b84fa0142dc98a3c2002227612eed0e6ba2ab93965a62a3d95dc6162dad2/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/2d90def46b65f15c41d3278d1403a5f65008f53a638327435aa8cc85c26cd548/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-205:{mountpoint:/var/lib/containers/storage/overlay/b1fec0b5e1ec6c7022747e98ba450b99ea349bc4055aa9f759c15abe11ca52d7/merged major:0 minor:205 fsType:overlay blockSize:0} overlay_0-214:{mountpoint:/var/lib/containers/storage/overlay/e8e8838478c2c716dbcc6545ac3336df165412a5126f1a3022187dbad3d372f0/merged major:0 minor:214 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/76a1b6c529d61b9e71629e72301234fa2d3cfab373996bfb0010e3fa0d0d9ac0/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-224:{mountpoint:/var/lib/containers/storage/overlay/f4f7c1af3b37ea3a1a9cfad007e47fd92f68ff3f5a4c0a57430f6e4cda5928cc/merged major:0 minor:224 fsType:overlay blockSize:0} overlay_0-226:{mountpoint:/var/lib/containers/storage/overlay/d52e757671b03703c53e1ed6b32cf9834e5d899705d8e4d0d2a2bbb01d95cff0/merged major:0 minor:226 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/99a522aac7a29ec7be9790a1b4e9132362d5fd3f7835e822037a35dc555612df/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/fab32ca3237defe599c23d2e3dfcb35774200dfe942948d5bc8bb5fbc38c6ce4/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/90006a7b4d3d4d1a0bbfa02ed826f3b3eef62ba280f105a519de1210f8ad93fb/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-242:{mountpoint:/var/lib/containers/storage/overlay/13ab6985796747bbc83f616cef6c86af3f3cf843b48aa904f15539afc84847b2/merged major:0 minor:242 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/f25349531a90a79d2208294fc1a1f7c9466228a66c4980814435355ffffd6618/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/d7fb8f9293cdfde690ae7267a6724be466693da67602041b80a4938b63ce3b88/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/76844f5d823e95b92e8048e3ec16c8a31acd06f7e0b66db8edc8418258d8c189/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/7ac4a4a1a7c93656c8f7785fb7c1f98fd853a3bb8e2515f9a78f771a5026349d/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/581fb2cb917f917335e9a9ed4d444d49f38576d596b69df2d9632b8a0ad6b3f1/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/b32e9cd8fd06c5221809442c3eb13c2b3cefd4847d3b423bf1ce5d34f21bf9f2/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/40d444de5ff7c0e87d791f0eb9498f906623efa2c679be7c5ac8855cddc2b564/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/4a58aa807a6f2a1481d810027754ae1e97e7fe709bdf72a565b7474e8284de13/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/c20f5b99ac25992fde51da2a7bcd17d7778c653b96d68dfb6bd8c47239a08c37/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/b56cc2c3233b5ddf675a3e238ef3808062c3d85fa80f9cf366f2898f6047c7ba/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/02a56294734600144d4c1876490b65988efe2461eeef0590d4a0ee9dbe9a1556/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 01 20:30:49.314303 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.314024 4484 manager.go:217] Machine: {Timestamp:2026-11-01 20:30:49.313024393 +0000 UTC m=+0.090664523 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099998 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25db723a05217783923f1e742ba382 SystemUUID:ec25db72-3a05-2177-8392-3f1e742ba382 BootID:af0d4f6a-5d95-484a-9110-d6ce5814e71c Filesystems:[{Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5800567f-f0e1-4cf0-8338-75b09d2d1255/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-242 DeviceMajor:0 DeviceMinor:242 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5160691e25c668734b4c024993d5bf4c3ed08d76e846e24ca73845637ad73521/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b23835ea8ffd51e39753cb1ba2551cabe2978b46c57b1f3ac048e4805fd5507a/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6b4f17a0-efd1-4024-a201-4078069b04e7/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:218 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc/volumes/kubernetes.io~projected/kube-api-access-qzsrc DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/df6c54825fb1d72061008ce2c740924d1b7e34f39b8db1676980e72afd1a970e/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/492a8e93-84b5-46d3-b6e4-4278f737249a/volumes/kubernetes.io~projected/kube-api-access-sw4w8 DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/04ec71b7-390e-4e40-abce-1c395a7a51ab/volumes/kubernetes.io~projected/kube-api-access-jjfv4 DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3fce64bd-a955-4e2c-8844-a25eaeb73862/volumes/kubernetes.io~projected/kube-api-access-d5pn9 DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/970d71e5-f861-4277-8fa5-6ec740418e51/volumes/kubernetes.io~projected/kube-api-access-6vm67 DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e8ad39cd-9b6a-464c-aa75-1bfc54b23891/volumes/kubernetes.io~projected/kube-api-access-khz9n DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-214 DeviceMajor:0 DeviceMinor:214 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-224 DeviceMajor:0 DeviceMinor:224 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bbb0eff1-ed70-4783-a255-374d8bf14687/volumes/kubernetes.io~projected/kube-api-access-74vvb DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/492a8e93-84b5-46d3-b6e4-4278f737249a/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-205 DeviceMajor:0 DeviceMinor:205 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f05cdcaf-b144-4c3d-b29b-2cb7d5a6f085/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:207 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9ee67cbd-8115-4c48-b5cd-5299ed284bce/volumes/kubernetes.io~projected/kube-api-access-btdtr DeviceMajor:0 DeviceMinor:221 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427/volumes/kubernetes.io~projected/kube-api-access-vgl7f DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5882d5d1fac98d98c7525baa5d1d7ea2c0071b79c1c9f344d61fd84c7f11bfaf/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9c215a148c70b1d9daea7436816efd58f428969f8cf645a832d599ccb6a95bd7/userdata/shm DeviceMajor:0 DeviceMinor:212 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/80729d64d0dbb0313f04a42bdf9b3ace58e24fb0ebf40ae93be25b504c4ce195/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6e9ed6520b34ab71b0c03cc82e133742536fd7ed40a5a40364132cc19eef6fbc/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/030f4716dcaaa8462c101a58de0015dbb55dca028de259c1fc7c3915330d1b7e/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dcd1501c6ab24fec2ddbec2a6a81381d321b3ca29979d2de8e25807600c2199d/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/8f213a88-8cb5-467b-8d21-9bc86a0654c9/volumes/kubernetes.io~projected/kube-api-access-lhrjv DeviceMajor:0 DeviceMinor:211 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-226 DeviceMajor:0 DeviceMinor:226 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3fce64bd-a955-4e2c-8844-a25eaeb73862/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ee503b738885a5321cca7fa7f3c6defd4071ab42689f058bea78da5918ac14e7/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/90e25e61-9761-4042-9dd4-88e1a6f8aa3c/volumes/kubernetes.io~projected/kube-api-access-gsbqz DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2288d739-5eb3-4186-a32e-187945267de1/volumes/kubernetes.io~projected/kube-api-access-px48q DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3fce64bd-a955-4e2c-8844-a25eaeb73862/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8d5ba769477249f45bb9343ec9a64905df58d817e34e232a4b8b906025665541/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f35238c9dff3501e77884444e067b5978d544dd583eaace87752ad74535b250a/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/13b4659d2b51036b64bba0b18fb853012f3254b7cc361c202cc9ccf550b1c794/userdata/shm DeviceMajor:0 DeviceMinor:219 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b2b6a5bff028eef38e8bcee10f728a6050fc3dd726d71e053b20f7e31954114c/userdata/shm DeviceMajor:0 DeviceMinor:222 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/82ca0c95-053f-4bb0-ab8d-1d8cbd531667/volumes/kubernetes.io~projected/kube-api-access-kplvv DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f05cdcaf-b144-4c3d-b29b-2cb7d5a6f085/volumes/kubernetes.io~projected/kube-api-access-dhml7 DeviceMajor:0 DeviceMinor:208 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7fc4de5f427b94b5aeece9302a62a6fade2769408cde35df2e0058c848f3b483/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a4342ddb-e06e-4d6d-9337-651349cf133f/volumes/kubernetes.io~projected/kube-api-access-mm9p9 DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:13b4659d2b51036 MacAddress:da:e2:0d:82:43:dd Speed:10000 Mtu:8901} {Name:80729d64d0dbb03 MacAddress:c6:a9:cb:1d:e5:cd Speed:10000 Mtu:8901} {Name:9c215a148c70b1d MacAddress:b6:13:3b:22:b4:5f Speed:10000 Mtu:8901} {Name:b23835ea8ffd51e MacAddress:0a:38:bb:0b:90:3a Speed:10000 Mtu:8901} {Name:b2b6a5bff028eef MacAddress:f2:71:c5:87:0d:2c Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:38:0f:d6:e4:2f Speed:0 Mtu:9001} {Name:br-int MacAddress:06:4d:ef:dd:20:53 Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:38:0f:d6:e4:2f Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:d6:25:37:57:7d:64 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:e6:e2:fc:c5:b7:06 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 01 20:30:49.314303 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.314305 4484 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 01 20:30:49.314536 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.314354 4484 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 01 20:30:49.314695 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.314664 4484 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 01 20:30:49.314828 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.314696 4484 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-35.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 01 20:30:49.314873 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.314831 4484 topology_manager.go:143] "Creating topology manager with none policy" Nov 01 20:30:49.314873 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.314838 4484 container_manager_linux.go:311] "Creating device plugin manager" Nov 01 20:30:49.314873 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.314848 4484 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 20:30:49.314873 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.314869 4484 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 20:30:49.314971 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.314964 4484 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 01 20:30:49.315154 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.315144 4484 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 01 20:30:49.315274 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.315265 4484 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 01 20:30:49.315310 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.315303 4484 kubelet.go:493] "Attempting to sync node with API server" Nov 01 20:30:49.315343 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.315315 4484 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 01 20:30:49.315343 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.315325 4484 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 01 20:30:49.315343 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.315334 4484 kubelet.go:405] "Adding apiserver pod source" Nov 01 20:30:49.315343 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.315343 4484 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 01 20:30:49.315905 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.315893 4484 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 01 20:30:49.315905 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.315905 4484 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 20:30:49.316495 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.316476 4484 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 01 20:30:49.319427 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.319409 4484 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 01 20:30:49.319427 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.319426 4484 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 20:30:49.319561 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.319549 4484 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 01 20:30:49.319844 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.319831 4484 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 01 20:30:49.319880 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.319862 4484 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 01 20:30:49.319977 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.319967 4484 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 01 20:30:49.320040 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.319982 4484 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 01 20:30:49.320040 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.319989 4484 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 01 20:30:49.320040 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.320012 4484 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 01 20:30:49.320040 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.320022 4484 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 01 20:30:49.320040 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.320030 4484 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 01 20:30:49.320040 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.320036 4484 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 01 20:30:49.320040 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.320041 4484 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 01 20:30:49.320306 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.320049 4484 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 01 20:30:49.320306 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.320055 4484 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 01 20:30:49.320306 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.320063 4484 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 01 20:30:49.320306 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.320073 4484 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 01 20:30:49.320306 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.320097 4484 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 01 20:30:49.320306 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.320103 4484 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 01 20:30:49.323429 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.323410 4484 server.go:1263] "Started kubelet" Nov 01 20:30:49.323601 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.323545 4484 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 01 20:30:49.328599 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.323624 4484 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 01 20:30:49.328599 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.323493 4484 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 01 20:30:49.328599 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.324233 4484 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 01 20:30:49.328599 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.325895 4484 server.go:317] "Adding debug handlers to kubelet server" Nov 01 20:30:49.328599 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.326720 4484 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 01 20:30:49.328599 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.327299 4484 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 01 20:30:49.328599 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.327626 4484 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 20:25:11 +0000 UTC" deadline="2028-05-16 08:06:28.674336213 +0000 UTC" Nov 01 20:30:49.328599 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.327649 4484 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13475h35m39.346690768s" Nov 01 20:30:49.328599 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:30:49.327682 4484 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-35.us-west-1.compute.internal\" not found" Nov 01 20:30:49.328599 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.327717 4484 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 01 20:30:49.328599 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.327724 4484 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 01 20:30:49.328599 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.328360 4484 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 01 20:30:49.324573 ip-10-0-141-35 systemd[1]: Started Kubernetes Kubelet. Nov 01 20:30:49.330812 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.330207 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="04ec71b7-390e-4e40-abce-1c395a7a51ab" volumeName="kubernetes.io/empty-dir/04ec71b7-390e-4e40-abce-1c395a7a51ab-tmp-dir" seLinuxMountContext="" Nov 01 20:30:49.330812 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.330249 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2288d739-5eb3-4186-a32e-187945267de1" volumeName="kubernetes.io/projected/2288d739-5eb3-4186-a32e-187945267de1-kube-api-access-px48q" seLinuxMountContext="" Nov 01 20:30:49.330812 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.330265 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82ca0c95-053f-4bb0-ab8d-1d8cbd531667" volumeName="kubernetes.io/configmap/82ca0c95-053f-4bb0-ab8d-1d8cbd531667-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 01 20:30:49.330812 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.330282 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc" volumeName="kubernetes.io/configmap/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-ovnkube-config" seLinuxMountContext="" Nov 01 20:30:49.330812 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.330294 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc" volumeName="kubernetes.io/secret/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-ovn-node-metrics-cert" seLinuxMountContext="" Nov 01 20:30:49.330812 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.330311 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f05cdcaf-b144-4c3d-b29b-2cb7d5a6f085" volumeName="kubernetes.io/configmap/f05cdcaf-b144-4c3d-b29b-2cb7d5a6f085-signing-cabundle" seLinuxMountContext="" Nov 01 20:30:49.330812 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.330322 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b4f17a0-efd1-4024-a201-4078069b04e7" volumeName="kubernetes.io/secret/6b4f17a0-efd1-4024-a201-4078069b04e7-original-pull-secret" seLinuxMountContext="" Nov 01 20:30:49.330812 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.330379 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbb0eff1-ed70-4783-a255-374d8bf14687" volumeName="kubernetes.io/configmap/bbb0eff1-ed70-4783-a255-374d8bf14687-config-volume" seLinuxMountContext="" Nov 01 20:30:49.330812 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.330399 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bcc5ee5c-08d8-49ba-9d7b-04e7a684e427" volumeName="kubernetes.io/projected/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-kube-api-access-vgl7f" seLinuxMountContext="" Nov 01 20:30:49.330812 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.330410 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f05cdcaf-b144-4c3d-b29b-2cb7d5a6f085" volumeName="kubernetes.io/projected/f05cdcaf-b144-4c3d-b29b-2cb7d5a6f085-kube-api-access-dhml7" seLinuxMountContext="" Nov 01 20:30:49.330812 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.330428 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="04ec71b7-390e-4e40-abce-1c395a7a51ab" volumeName="kubernetes.io/projected/04ec71b7-390e-4e40-abce-1c395a7a51ab-kube-api-access-jjfv4" seLinuxMountContext="" Nov 01 20:30:49.330812 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.330441 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fce64bd-a955-4e2c-8844-a25eaeb73862" volumeName="kubernetes.io/empty-dir/3fce64bd-a955-4e2c-8844-a25eaeb73862-tmp" seLinuxMountContext="" Nov 01 20:30:49.330812 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.330461 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="970d71e5-f861-4277-8fa5-6ec740418e51" volumeName="kubernetes.io/projected/970d71e5-f861-4277-8fa5-6ec740418e51-kube-api-access-6vm67" seLinuxMountContext="" Nov 01 20:30:49.330812 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.330474 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4342ddb-e06e-4d6d-9337-651349cf133f" volumeName="kubernetes.io/configmap/a4342ddb-e06e-4d6d-9337-651349cf133f-cni-sysctl-allowlist" seLinuxMountContext="" Nov 01 20:30:49.330812 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.330535 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4342ddb-e06e-4d6d-9337-651349cf133f" volumeName="kubernetes.io/configmap/a4342ddb-e06e-4d6d-9337-651349cf133f-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 01 20:30:49.330812 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.330547 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4342ddb-e06e-4d6d-9337-651349cf133f" volumeName="kubernetes.io/projected/a4342ddb-e06e-4d6d-9337-651349cf133f-kube-api-access-mm9p9" seLinuxMountContext="" Nov 01 20:30:49.330812 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.330566 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8ad39cd-9b6a-464c-aa75-1bfc54b23891" volumeName="kubernetes.io/projected/e8ad39cd-9b6a-464c-aa75-1bfc54b23891-kube-api-access-khz9n" seLinuxMountContext="" Nov 01 20:30:49.330812 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.330587 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2288d739-5eb3-4186-a32e-187945267de1" volumeName="kubernetes.io/configmap/2288d739-5eb3-4186-a32e-187945267de1-iptables-alerter-script" seLinuxMountContext="" Nov 01 20:30:49.330812 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.330599 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82ca0c95-053f-4bb0-ab8d-1d8cbd531667" volumeName="kubernetes.io/empty-dir/82ca0c95-053f-4bb0-ab8d-1d8cbd531667-data-volume" seLinuxMountContext="" Nov 01 20:30:49.330812 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.330616 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bcc5ee5c-08d8-49ba-9d7b-04e7a684e427" volumeName="kubernetes.io/configmap/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-cni-binary-copy" seLinuxMountContext="" Nov 01 20:30:49.330812 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.330644 4484 factory.go:55] Registering systemd factory Nov 01 20:30:49.330812 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.330654 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="492a8e93-84b5-46d3-b6e4-4278f737249a" volumeName="kubernetes.io/secret/492a8e93-84b5-46d3-b6e4-4278f737249a-cert" seLinuxMountContext="" Nov 01 20:30:49.330812 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.330665 4484 factory.go:223] Registration of the systemd container factory successfully Nov 01 20:30:49.330812 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.330669 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbb0eff1-ed70-4783-a255-374d8bf14687" volumeName="kubernetes.io/projected/bbb0eff1-ed70-4783-a255-374d8bf14687-kube-api-access-74vvb" seLinuxMountContext="" Nov 01 20:30:49.330812 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.330684 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bcc5ee5c-08d8-49ba-9d7b-04e7a684e427" volumeName="kubernetes.io/configmap/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-multus-daemon-config" seLinuxMountContext="" Nov 01 20:30:49.330812 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.330698 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f05cdcaf-b144-4c3d-b29b-2cb7d5a6f085" volumeName="kubernetes.io/secret/f05cdcaf-b144-4c3d-b29b-2cb7d5a6f085-signing-key" seLinuxMountContext="" Nov 01 20:30:49.330812 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.330713 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fce64bd-a955-4e2c-8844-a25eaeb73862" volumeName="kubernetes.io/projected/3fce64bd-a955-4e2c-8844-a25eaeb73862-kube-api-access-d5pn9" seLinuxMountContext="" Nov 01 20:30:49.330812 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.330726 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5800567f-f0e1-4cf0-8338-75b09d2d1255" volumeName="kubernetes.io/secret/5800567f-f0e1-4cf0-8338-75b09d2d1255-agent-certs" seLinuxMountContext="" Nov 01 20:30:49.330812 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.330739 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f213a88-8cb5-467b-8d21-9bc86a0654c9" volumeName="kubernetes.io/projected/8f213a88-8cb5-467b-8d21-9bc86a0654c9-kube-api-access-lhrjv" seLinuxMountContext="" Nov 01 20:30:49.330812 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.330756 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90e25e61-9761-4042-9dd4-88e1a6f8aa3c" volumeName="kubernetes.io/projected/90e25e61-9761-4042-9dd4-88e1a6f8aa3c-kube-api-access-gsbqz" seLinuxMountContext="" Nov 01 20:30:49.330812 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.330793 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbb0eff1-ed70-4783-a255-374d8bf14687" volumeName="kubernetes.io/empty-dir/bbb0eff1-ed70-4783-a255-374d8bf14687-tmp-dir" seLinuxMountContext="" Nov 01 20:30:49.330812 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.330807 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc" volumeName="kubernetes.io/projected/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-kube-api-access-qzsrc" seLinuxMountContext="" Nov 01 20:30:49.330812 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.330824 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8ad39cd-9b6a-464c-aa75-1bfc54b23891" volumeName="kubernetes.io/configmap/e8ad39cd-9b6a-464c-aa75-1bfc54b23891-serviceca" seLinuxMountContext="" Nov 01 20:30:49.330812 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.330837 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="492a8e93-84b5-46d3-b6e4-4278f737249a" volumeName="kubernetes.io/projected/492a8e93-84b5-46d3-b6e4-4278f737249a-kube-api-access-sw4w8" seLinuxMountContext="" Nov 01 20:30:49.332615 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.330851 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ee67cbd-8115-4c48-b5cd-5299ed284bce" volumeName="kubernetes.io/projected/9ee67cbd-8115-4c48-b5cd-5299ed284bce-kube-api-access-btdtr" seLinuxMountContext="" Nov 01 20:30:49.332615 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.330870 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc" volumeName="kubernetes.io/configmap/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-env-overrides" seLinuxMountContext="" Nov 01 20:30:49.332615 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.330884 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc" volumeName="kubernetes.io/configmap/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-ovnkube-script-lib" seLinuxMountContext="" Nov 01 20:30:49.332615 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.330948 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fce64bd-a955-4e2c-8844-a25eaeb73862" volumeName="kubernetes.io/empty-dir/3fce64bd-a955-4e2c-8844-a25eaeb73862-etc-tuned" seLinuxMountContext="" Nov 01 20:30:49.332615 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.330959 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5800567f-f0e1-4cf0-8338-75b09d2d1255" volumeName="kubernetes.io/configmap/5800567f-f0e1-4cf0-8338-75b09d2d1255-konnectivity-ca" seLinuxMountContext="" Nov 01 20:30:49.332615 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.330970 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82ca0c95-053f-4bb0-ab8d-1d8cbd531667" volumeName="kubernetes.io/projected/82ca0c95-053f-4bb0-ab8d-1d8cbd531667-kube-api-access-kplvv" seLinuxMountContext="" Nov 01 20:30:49.332615 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.330986 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4342ddb-e06e-4d6d-9337-651349cf133f" volumeName="kubernetes.io/configmap/a4342ddb-e06e-4d6d-9337-651349cf133f-cni-binary-copy" seLinuxMountContext="" Nov 01 20:30:49.332615 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.331012 4484 reconstruct.go:97] "Volume reconstruction finished" Nov 01 20:30:49.332615 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.331021 4484 reconciler.go:29] "Reconciler: start to sync state" Nov 01 20:30:49.335533 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.335510 4484 factory.go:153] Registering CRI-O factory Nov 01 20:30:49.335533 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.335531 4484 factory.go:223] Registration of the crio container factory successfully Nov 01 20:30:49.335716 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.335581 4484 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 01 20:30:49.335716 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.335599 4484 factory.go:103] Registering Raw factory Nov 01 20:30:49.335716 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.335612 4484 manager.go:1199] Started watching for new ooms in manager Nov 01 20:30:49.339050 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.336200 4484 manager.go:319] Starting recovery of all containers Nov 01 20:30:49.362191 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.362134 4484 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 01 20:30:49.363559 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.363539 4484 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 01 20:30:49.363559 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.363562 4484 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 01 20:30:49.363737 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.363579 4484 kubelet.go:2521] "Starting kubelet main sync loop" Nov 01 20:30:49.363737 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:30:49.363614 4484 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 01 20:30:49.379552 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.379511 4484 generic.go:356] "Generic (PLEG): container finished" podID="a4342ddb-e06e-4d6d-9337-651349cf133f" containerID="b725fc29140dc56bc40463e46694446809bfadb897cc5e5e923542cd79886078" exitCode=0 Nov 01 20:30:49.379552 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.379548 4484 generic.go:356] "Generic (PLEG): container finished" podID="a4342ddb-e06e-4d6d-9337-651349cf133f" containerID="7735f1d5ae7e07f06ca4d4a84afe0ed7c348618bf72ae30823da9a28d1ab0896" exitCode=0 Nov 01 20:30:49.379552 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.379557 4484 generic.go:356] "Generic (PLEG): container finished" podID="a4342ddb-e06e-4d6d-9337-651349cf133f" containerID="c8a5d7fd2b1aaf93c0900511bad19aa46eb48f8c046e6e291601da629866c347" exitCode=0 Nov 01 20:30:49.379766 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.379563 4484 generic.go:356] "Generic (PLEG): container finished" podID="a4342ddb-e06e-4d6d-9337-651349cf133f" containerID="fffe5da2ab8079269a0d6709fe0379a04a5bab5f19657a39b965765763e9ccb1" exitCode=0 Nov 01 20:30:49.379766 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.379569 4484 generic.go:356] "Generic (PLEG): container finished" podID="a4342ddb-e06e-4d6d-9337-651349cf133f" containerID="11361c228fb886de0dac0d8390c5fe9112e63f6df59b51e90092f9f39bf2453a" exitCode=0 Nov 01 20:30:49.379766 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.379574 4484 generic.go:356] "Generic (PLEG): container finished" podID="a4342ddb-e06e-4d6d-9337-651349cf133f" containerID="3f64bb4b02db198c2e89007a8a297ae463ac8545263ef08c20209abe3769d61f" exitCode=0 Nov 01 20:30:49.384951 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.384923 4484 generic.go:356] "Generic (PLEG): container finished" podID="91ec11d0905cc00deef779d06bc93fe3" containerID="f8e0d927fc93d1c4dbbd959fec7aa6c7672882ba2e5c8a7226ce0c3f1b35150f" exitCode=0 Nov 01 20:30:49.405480 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.405417 4484 manager.go:324] Recovery completed Nov 01 20:30:49.418667 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.418642 4484 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 20:30:49.420585 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.420556 4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-35.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 20:30:49.420666 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.420589 4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-35.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 20:30:49.420666 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.420599 4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-35.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 20:30:49.422077 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.422059 4484 cpu_manager.go:226] "Starting" policy="none" Nov 01 20:30:49.422077 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.422075 4484 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 01 20:30:49.422177 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.422080 4484 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 20:30:49.422177 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.422096 4484 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 01 20:30:49.422296 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.422253 4484 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 01 20:30:49.422296 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.422271 4484 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 01 20:30:49.422296 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.422292 4484 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 01 20:30:49.422379 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.422298 4484 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 01 20:30:49.422379 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.422304 4484 policy_none.go:50] "Start" Nov 01 20:30:49.422379 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.422313 4484 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 01 20:30:49.422379 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.422323 4484 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 01 20:30:49.422484 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.422453 4484 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 01 20:30:49.422484 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.422462 4484 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 01 20:30:49.422484 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.422468 4484 policy_none.go:44] "Start" Nov 01 20:30:49.422953 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.422928 4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-35.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 20:30:49.423041 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.422956 4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-35.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 20:30:49.423041 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.422967 4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-35.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 20:30:49.426842 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.426809 4484 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 01 20:30:49.427734 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:30:49.427717 4484 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-35.us-west-1.compute.internal\" not found" Nov 01 20:30:49.429729 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.429717 4484 manager.go:341] "Starting Device Plugin manager" Nov 01 20:30:49.429786 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:30:49.429742 4484 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 01 20:30:49.429786 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.429751 4484 server.go:87] "Starting device plugin registration server" Nov 01 20:30:49.430023 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.430011 4484 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 01 20:30:49.430066 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.430025 4484 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 01 20:30:49.430140 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.430121 4484 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 01 20:30:49.430247 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.430218 4484 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 01 20:30:49.430299 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.430257 4484 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 01 20:30:49.430299 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.430264 4484 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 01 20:30:49.430393 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.430307 4484 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T20:30:49.430244364Z","UUID":"b8425774-110c-4e38-a244-823ca6c027db","Handler":null,"Name":"","Endpoint":""} Nov 01 20:30:49.431667 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:30:49.431612 4484 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 01 20:30:49.431667 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:30:49.431653 4484 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-35.us-west-1.compute.internal\" not found" Nov 01 20:30:49.431811 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.431708 4484 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 01 20:30:49.431811 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.431731 4484 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 01 20:30:49.464906 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.464397 4484 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-35.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-35.us-west-1.compute.internal"] Nov 01 20:30:49.465074 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.465059 4484 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 20:30:49.466136 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.466117 4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-35.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 20:30:49.466240 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.466151 4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-35.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 20:30:49.466240 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.466166 4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-35.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 20:30:49.466411 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.466401 4484 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 20:30:49.466554 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.466538 4484 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 20:30:49.467287 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.467265 4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-35.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 20:30:49.467287 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.467282 4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-35.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 20:30:49.467430 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.467299 4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-35.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 20:30:49.467430 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.467306 4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-35.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 20:30:49.467430 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.467310 4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-35.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 20:30:49.467430 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.467321 4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-35.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 20:30:49.467553 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:30:49.467475 4484 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-35.us-west-1.compute.internal\" not found" node="ip-10-0-141-35.us-west-1.compute.internal" Nov 01 20:30:49.467582 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.467537 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-35.us-west-1.compute.internal" event={"ID":"f192393bd55f8e34f5db37f9db86763a","Type":"ContainerStarted","Data":"3b894ae806f84c5a33a7700aabef954a09499835545c482ef4080d88aee3d680"} Nov 01 20:30:49.467582 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.467570 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-35.us-west-1.compute.internal" event={"ID":"f192393bd55f8e34f5db37f9db86763a","Type":"ContainerStarted","Data":"6e9ed6520b34ab71b0c03cc82e133742536fd7ed40a5a40364132cc19eef6fbc"} Nov 01 20:30:49.467661 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.467649 4484 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 20:30:49.467751 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.467734 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-35.us-west-1.compute.internal" event={"ID":"91ec11d0905cc00deef779d06bc93fe3","Type":"ContainerStarted","Data":"336ddb021d7c9688585026d477e6f3d59e5bafc1b41545387c8322fae130823a"} Nov 01 20:30:49.467806 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.467763 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-35.us-west-1.compute.internal" event={"ID":"91ec11d0905cc00deef779d06bc93fe3","Type":"ContainerDied","Data":"f8e0d927fc93d1c4dbbd959fec7aa6c7672882ba2e5c8a7226ce0c3f1b35150f"} Nov 01 20:30:49.467806 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.467777 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-35.us-west-1.compute.internal" event={"ID":"91ec11d0905cc00deef779d06bc93fe3","Type":"ContainerStarted","Data":"7fc4de5f427b94b5aeece9302a62a6fade2769408cde35df2e0058c848f3b483"} Nov 01 20:30:49.468438 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.468424 4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-35.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 20:30:49.468489 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.468447 4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-35.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 20:30:49.468489 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.468457 4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-35.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 20:30:49.468580 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:30:49.468569 4484 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-35.us-west-1.compute.internal\" not found" node="ip-10-0-141-35.us-west-1.compute.internal" Nov 01 20:30:49.530149 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.530117 4484 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 20:30:49.530373 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.530329 4484 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 01 20:30:49.530373 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.530344 4484 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 20:30:49.531293 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.531276 4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-35.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 20:30:49.531352 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.531305 4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-35.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 20:30:49.531352 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.531318 4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-35.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 20:30:49.531420 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.531356 4484 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-35.us-west-1.compute.internal" Nov 01 20:30:49.578915 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.578882 4484 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 20:30:49.578915 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.578899 4484 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 20:30:49.581423 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.581405 4484 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 20:30:49.630092 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.630058 4484 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-35.us-west-1.compute.internal" Nov 01 20:30:49.631257 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.631230 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/91ec11d0905cc00deef779d06bc93fe3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-35.us-west-1.compute.internal\" (UID: \"91ec11d0905cc00deef779d06bc93fe3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-35.us-west-1.compute.internal" Nov 01 20:30:49.631320 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.631273 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91ec11d0905cc00deef779d06bc93fe3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-35.us-west-1.compute.internal\" (UID: \"91ec11d0905cc00deef779d06bc93fe3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-35.us-west-1.compute.internal" Nov 01 20:30:49.631320 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.631306 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f192393bd55f8e34f5db37f9db86763a-config\") pod \"kube-apiserver-proxy-ip-10-0-141-35.us-west-1.compute.internal\" (UID: \"f192393bd55f8e34f5db37f9db86763a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-35.us-west-1.compute.internal" Nov 01 20:30:49.682614 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.682531 4484 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-141-35.us-west-1.compute.internal" Nov 01 20:30:49.682614 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.682614 4484 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-35.us-west-1.compute.internal" Nov 01 20:30:49.683238 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.683222 4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-35.us-west-1.compute.internal" event="NodeNotReady" Nov 01 20:30:49.683329 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.683252 4484 setters.go:546] "Node became not ready" node="ip-10-0-141-35.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-01T20:30:49Z","lastTransitionTime":"2026-11-01T20:30:49Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Nov 01 20:30:49.686160 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.686145 4484 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 20:30:49.686337 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:30:49.686189 4484 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-35.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-35.us-west-1.compute.internal" Nov 01 20:30:49.686337 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.686201 4484 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-35.us-west-1.compute.internal" Nov 01 20:30:49.693790 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.693760 4484 nodeinfomanager.go:471] existing CSINode "ip-10-0-141-35.us-west-1.compute.internal" is owned by different node (oldNodeID="daff2795-c81f-4568-8aab-dca889c6cba6", newNodeID=""), cleaning up... Nov 01 20:30:49.732190 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.732142 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/91ec11d0905cc00deef779d06bc93fe3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-35.us-west-1.compute.internal\" (UID: \"91ec11d0905cc00deef779d06bc93fe3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-35.us-west-1.compute.internal" Nov 01 20:30:49.732309 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.732193 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91ec11d0905cc00deef779d06bc93fe3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-35.us-west-1.compute.internal\" (UID: \"91ec11d0905cc00deef779d06bc93fe3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-35.us-west-1.compute.internal" Nov 01 20:30:49.732309 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.732218 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f192393bd55f8e34f5db37f9db86763a-config\") pod \"kube-apiserver-proxy-ip-10-0-141-35.us-west-1.compute.internal\" (UID: \"f192393bd55f8e34f5db37f9db86763a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-35.us-west-1.compute.internal" Nov 01 20:30:49.732309 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.732249 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/91ec11d0905cc00deef779d06bc93fe3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-35.us-west-1.compute.internal\" (UID: \"91ec11d0905cc00deef779d06bc93fe3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-35.us-west-1.compute.internal" Nov 01 20:30:49.732309 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.732261 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f192393bd55f8e34f5db37f9db86763a-config\") pod \"kube-apiserver-proxy-ip-10-0-141-35.us-west-1.compute.internal\" (UID: \"f192393bd55f8e34f5db37f9db86763a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-35.us-west-1.compute.internal" Nov 01 20:30:49.732309 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.732262 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91ec11d0905cc00deef779d06bc93fe3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-35.us-west-1.compute.internal\" (UID: \"91ec11d0905cc00deef779d06bc93fe3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-35.us-west-1.compute.internal" Nov 01 20:30:49.748382 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:49.748344 4484 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 20:30:49.748560 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:30:49.748404 4484 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-35.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-35.us-west-1.compute.internal" Nov 01 20:30:50.316012 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.315969 4484 apiserver.go:52] "Watching apiserver" Nov 01 20:30:50.401904 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.401873 4484 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-35.us-west-1.compute.internal" Nov 01 20:30:50.422876 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.422837 4484 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 01 20:30:50.424045 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.424015 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-35.us-west-1.compute.internal","openshift-ovn-kubernetes/ovnkube-node-tv7jl","openshift-kube-storage-version-migrator/migrator-66df5d496b-2lxrg","openshift-multus/multus-5k2px","openshift-network-diagnostics/network-check-target-ghh5z","openshift-cluster-node-tuning-operator/tuned-qjtnx","openshift-ingress-canary/ingress-canary-mdc46","openshift-dns/node-resolver-w6nn2","kube-system/konnectivity-agent-2f2ct","openshift-dns/dns-default-gtf4c","openshift-multus/network-metrics-daemon-vxzfk","kube-system/global-pull-secret-syncer-xh6zg","kube-system/kube-apiserver-proxy-ip-10-0-141-35.us-west-1.compute.internal","openshift-service-ca/service-ca-5fffc97bdf-smp9z","openshift-multus/multus-additional-cni-plugins-fmnrq","openshift-network-operator/iptables-alerter-w7qmz","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb94","openshift-insights/insights-runtime-extractor-4ssvj","openshift-image-registry/node-ca-bztv7"] Nov 01 20:30:50.426409 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.426383 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxzfk" Nov 01 20:30:50.429725 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.429708 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gtf4c" Nov 01 20:30:50.430133 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.430117 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4ssvj" Nov 01 20:30:50.437276 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.437243 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-etc-kubernetes\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:50.437364 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.437283 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6b4f17a0-efd1-4024-a201-4078069b04e7-dbus\") pod \"global-pull-secret-syncer-xh6zg\" (UID: \"6b4f17a0-efd1-4024-a201-4078069b04e7\") " pod="kube-system/global-pull-secret-syncer-xh6zg" Nov 01 20:30:50.437399 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.437356 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-hostroot\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:50.437444 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.437406 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-host-run-multus-certs\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:50.437444 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.437436 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/90e25e61-9761-4042-9dd4-88e1a6f8aa3c-socket-dir\") pod \"aws-ebs-csi-driver-node-ljb94\" (UID: \"90e25e61-9761-4042-9dd4-88e1a6f8aa3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb94" Nov 01 20:30:50.437505 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.437478 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a4342ddb-e06e-4d6d-9337-651349cf133f-cnibin\") pod \"multus-additional-cni-plugins-fmnrq\" (UID: \"a4342ddb-e06e-4d6d-9337-651349cf133f\") " pod="openshift-multus/multus-additional-cni-plugins-fmnrq" Nov 01 20:30:50.437505 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.437499 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-etc-systemd\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:50.437562 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.437519 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btdtr\" (UniqueName: \"kubernetes.io/projected/9ee67cbd-8115-4c48-b5cd-5299ed284bce-kube-api-access-btdtr\") pod \"network-check-target-ghh5z\" (UID: \"9ee67cbd-8115-4c48-b5cd-5299ed284bce\") " pod="openshift-network-diagnostics/network-check-target-ghh5z" Nov 01 20:30:50.437611 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.437584 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-multus-cni-dir\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:50.437645 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.437624 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-run-openvswitch\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:50.437674 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.437650 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-ovnkube-script-lib\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:50.437703 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.437671 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-var-lib-kubelet\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:50.437760 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.437738 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e8ad39cd-9b6a-464c-aa75-1bfc54b23891-host\") pod \"node-ca-bztv7\" (UID: \"e8ad39cd-9b6a-464c-aa75-1bfc54b23891\") " pod="openshift-image-registry/node-ca-bztv7" Nov 01 20:30:50.437815 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.437781 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-host-run-netns\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:50.437865 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.437837 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-node-log\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:50.437913 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.437879 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-run\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:50.437946 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.437920 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-cnibin\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:50.437979 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.437962 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-systemd-units\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:50.438042 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.438007 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-ovn-node-metrics-cert\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:50.438042 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.438030 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzsrc\" (UniqueName: \"kubernetes.io/projected/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-kube-api-access-qzsrc\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:50.438128 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.438046 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4342ddb-e06e-4d6d-9337-651349cf133f-system-cni-dir\") pod \"multus-additional-cni-plugins-fmnrq\" (UID: \"a4342ddb-e06e-4d6d-9337-651349cf133f\") " pod="openshift-multus/multus-additional-cni-plugins-fmnrq" Nov 01 20:30:50.438128 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.438063 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a4342ddb-e06e-4d6d-9337-651349cf133f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fmnrq\" (UID: \"a4342ddb-e06e-4d6d-9337-651349cf133f\") " pod="openshift-multus/multus-additional-cni-plugins-fmnrq" Nov 01 20:30:50.438261 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.438240 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a4342ddb-e06e-4d6d-9337-651349cf133f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fmnrq\" (UID: \"a4342ddb-e06e-4d6d-9337-651349cf133f\") " pod="openshift-multus/multus-additional-cni-plugins-fmnrq" Nov 01 20:30:50.438313 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.438277 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm9p9\" (UniqueName: \"kubernetes.io/projected/a4342ddb-e06e-4d6d-9337-651349cf133f-kube-api-access-mm9p9\") pod \"multus-additional-cni-plugins-fmnrq\" (UID: \"a4342ddb-e06e-4d6d-9337-651349cf133f\") " pod="openshift-multus/multus-additional-cni-plugins-fmnrq" Nov 01 20:30:50.438313 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.438301 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-etc-sysctl-d\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:50.438422 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.438406 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-run-systemd\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:50.438473 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.438440 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-etc-sysctl-conf\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:50.438473 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.438465 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fce64bd-a955-4e2c-8844-a25eaeb73862-tmp\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:50.438550 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.438526 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fce64bd-a955-4e2c-8844-a25eaeb73862-tmp\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:50.438607 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.438547 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b4f17a0-efd1-4024-a201-4078069b04e7-original-pull-secret\") pod \"global-pull-secret-syncer-xh6zg\" (UID: \"6b4f17a0-efd1-4024-a201-4078069b04e7\") " pod="kube-system/global-pull-secret-syncer-xh6zg" Nov 01 20:30:50.438607 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.438581 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-cni-binary-copy\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:50.438687 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.438614 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-host-var-lib-cni-multus\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:50.438687 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.438640 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-multus-conf-dir\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:50.438687 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.438663 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3fce64bd-a955-4e2c-8844-a25eaeb73862-etc-tuned\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:50.438818 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.438715 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-host-run-netns\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:50.438818 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.438731 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3fce64bd-a955-4e2c-8844-a25eaeb73862-etc-tuned\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:50.438818 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.438760 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgl7f\" (UniqueName: \"kubernetes.io/projected/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-kube-api-access-vgl7f\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:50.438818 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.438802 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px48q\" (UniqueName: \"kubernetes.io/projected/2288d739-5eb3-4186-a32e-187945267de1-kube-api-access-px48q\") pod \"iptables-alerter-w7qmz\" (UID: \"2288d739-5eb3-4186-a32e-187945267de1\") " pod="openshift-network-operator/iptables-alerter-w7qmz" Nov 01 20:30:50.439060 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.438889 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-etc-openvswitch\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:50.439060 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.438966 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-run-ovn\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:50.439060 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.439008 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-host-cni-netd\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:50.439198 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.439065 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-env-overrides\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:50.439198 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.439101 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a4342ddb-e06e-4d6d-9337-651349cf133f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fmnrq\" (UID: \"a4342ddb-e06e-4d6d-9337-651349cf133f\") " pod="openshift-multus/multus-additional-cni-plugins-fmnrq" Nov 01 20:30:50.439198 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.439124 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-sys\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:50.439198 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.439150 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5pn9\" (UniqueName: \"kubernetes.io/projected/3fce64bd-a955-4e2c-8844-a25eaeb73862-kube-api-access-d5pn9\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:50.439198 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.439183 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5800567f-f0e1-4cf0-8338-75b09d2d1255-konnectivity-ca\") pod \"konnectivity-agent-2f2ct\" (UID: \"5800567f-f0e1-4cf0-8338-75b09d2d1255\") " pod="kube-system/konnectivity-agent-2f2ct" Nov 01 20:30:50.439385 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.439250 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/04ec71b7-390e-4e40-abce-1c395a7a51ab-hosts-file\") pod \"node-resolver-w6nn2\" (UID: \"04ec71b7-390e-4e40-abce-1c395a7a51ab\") " pod="openshift-dns/node-resolver-w6nn2" Nov 01 20:30:50.439385 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.439315 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-host-slash\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:50.439385 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.439354 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-log-socket\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:50.439527 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.439410 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a4342ddb-e06e-4d6d-9337-651349cf133f-cni-binary-copy\") pod \"multus-additional-cni-plugins-fmnrq\" (UID: \"a4342ddb-e06e-4d6d-9337-651349cf133f\") " pod="openshift-multus/multus-additional-cni-plugins-fmnrq" Nov 01 20:30:50.439527 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.439439 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/970d71e5-f861-4277-8fa5-6ec740418e51-metrics-certs\") pod \"network-metrics-daemon-vxzfk\" (UID: \"970d71e5-f861-4277-8fa5-6ec740418e51\") " pod="openshift-multus/network-metrics-daemon-vxzfk" Nov 01 20:30:50.439527 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.439463 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-etc-sysconfig\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:50.439527 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.439489 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-host-kubelet\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:50.439527 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.439515 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-ovnkube-config\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:50.439755 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.439541 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5800567f-f0e1-4cf0-8338-75b09d2d1255-agent-certs\") pod \"konnectivity-agent-2f2ct\" (UID: \"5800567f-f0e1-4cf0-8338-75b09d2d1255\") " pod="kube-system/konnectivity-agent-2f2ct" Nov 01 20:30:50.439755 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.439606 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/90e25e61-9761-4042-9dd4-88e1a6f8aa3c-device-dir\") pod \"aws-ebs-csi-driver-node-ljb94\" (UID: \"90e25e61-9761-4042-9dd4-88e1a6f8aa3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb94" Nov 01 20:30:50.439755 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.439685 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-host-cni-bin\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:50.439755 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.439728 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-lib-modules\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:50.439898 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.439756 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-multus-daemon-config\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:50.439898 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.439826 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/90e25e61-9761-4042-9dd4-88e1a6f8aa3c-registration-dir\") pod \"aws-ebs-csi-driver-node-ljb94\" (UID: \"90e25e61-9761-4042-9dd4-88e1a6f8aa3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb94" Nov 01 20:30:50.439898 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.439858 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04ec71b7-390e-4e40-abce-1c395a7a51ab-tmp-dir\") pod \"node-resolver-w6nn2\" (UID: \"04ec71b7-390e-4e40-abce-1c395a7a51ab\") " pod="openshift-dns/node-resolver-w6nn2" Nov 01 20:30:50.440006 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.439901 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-host-run-ovn-kubernetes\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:50.440006 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.439928 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-host\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:50.440006 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.439938 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04ec71b7-390e-4e40-abce-1c395a7a51ab-tmp-dir\") pod \"node-resolver-w6nn2\" (UID: \"04ec71b7-390e-4e40-abce-1c395a7a51ab\") " pod="openshift-dns/node-resolver-w6nn2" Nov 01 20:30:50.440006 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.439954 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e8ad39cd-9b6a-464c-aa75-1bfc54b23891-serviceca\") pod \"node-ca-bztv7\" (UID: \"e8ad39cd-9b6a-464c-aa75-1bfc54b23891\") " pod="openshift-image-registry/node-ca-bztv7" Nov 01 20:30:50.440006 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.439982 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-multus-socket-dir-parent\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:50.440186 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.440031 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-host-run-k8s-cni-cncf-io\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:50.440186 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.440058 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-etc-kubernetes\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:50.440186 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.440083 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-var-lib-openvswitch\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:50.440186 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.440110 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-etc-modprobe-d\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:50.440186 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.440151 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-host-var-lib-kubelet\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:50.440186 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.440179 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2288d739-5eb3-4186-a32e-187945267de1-iptables-alerter-script\") pod \"iptables-alerter-w7qmz\" (UID: \"2288d739-5eb3-4186-a32e-187945267de1\") " pod="openshift-network-operator/iptables-alerter-w7qmz" Nov 01 20:30:50.440432 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.440195 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsbqz\" (UniqueName: \"kubernetes.io/projected/90e25e61-9761-4042-9dd4-88e1a6f8aa3c-kube-api-access-gsbqz\") pod \"aws-ebs-csi-driver-node-ljb94\" (UID: \"90e25e61-9761-4042-9dd4-88e1a6f8aa3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb94" Nov 01 20:30:50.440432 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.440244 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:50.440432 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.440277 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-host-var-lib-cni-bin\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:50.440432 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.440295 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2288d739-5eb3-4186-a32e-187945267de1-host-slash\") pod \"iptables-alerter-w7qmz\" (UID: \"2288d739-5eb3-4186-a32e-187945267de1\") " pod="openshift-network-operator/iptables-alerter-w7qmz" Nov 01 20:30:50.440432 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.440320 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/90e25e61-9761-4042-9dd4-88e1a6f8aa3c-etc-selinux\") pod \"aws-ebs-csi-driver-node-ljb94\" (UID: \"90e25e61-9761-4042-9dd4-88e1a6f8aa3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb94" Nov 01 20:30:50.440432 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.440359 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjfv4\" (UniqueName: \"kubernetes.io/projected/04ec71b7-390e-4e40-abce-1c395a7a51ab-kube-api-access-jjfv4\") pod \"node-resolver-w6nn2\" (UID: \"04ec71b7-390e-4e40-abce-1c395a7a51ab\") " pod="openshift-dns/node-resolver-w6nn2" Nov 01 20:30:50.440432 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.440388 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vm67\" (UniqueName: \"kubernetes.io/projected/970d71e5-f861-4277-8fa5-6ec740418e51-kube-api-access-6vm67\") pod \"network-metrics-daemon-vxzfk\" (UID: \"970d71e5-f861-4277-8fa5-6ec740418e51\") " pod="openshift-multus/network-metrics-daemon-vxzfk" Nov 01 20:30:50.440699 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.440494 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6b4f17a0-efd1-4024-a201-4078069b04e7-kubelet-config\") pod \"global-pull-secret-syncer-xh6zg\" (UID: \"6b4f17a0-efd1-4024-a201-4078069b04e7\") " pod="kube-system/global-pull-secret-syncer-xh6zg" Nov 01 20:30:50.440699 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.440611 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-system-cni-dir\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:50.440699 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.440629 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khz9n\" (UniqueName: \"kubernetes.io/projected/e8ad39cd-9b6a-464c-aa75-1bfc54b23891-kube-api-access-khz9n\") pod \"node-ca-bztv7\" (UID: \"e8ad39cd-9b6a-464c-aa75-1bfc54b23891\") " pod="openshift-image-registry/node-ca-bztv7" Nov 01 20:30:50.440699 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.440651 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/90e25e61-9761-4042-9dd4-88e1a6f8aa3c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ljb94\" (UID: \"90e25e61-9761-4042-9dd4-88e1a6f8aa3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb94" Nov 01 20:30:50.440699 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.440699 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/90e25e61-9761-4042-9dd4-88e1a6f8aa3c-sys-fs\") pod \"aws-ebs-csi-driver-node-ljb94\" (UID: \"90e25e61-9761-4042-9dd4-88e1a6f8aa3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb94" Nov 01 20:30:50.461264 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.461238 4484 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 20:30:50.461378 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:30:50.461288 4484 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-35.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-35.us-west-1.compute.internal" Nov 01 20:30:50.480024 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.479978 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 01 20:30:50.480024 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.480018 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 01 20:30:50.480157 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.479983 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wk8gn\"" Nov 01 20:30:50.480251 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.480237 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 01 20:30:50.480347 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.480329 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 01 20:30:50.480347 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.480342 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 01 20:30:50.480453 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.480353 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 01 20:30:50.480453 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.480368 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 01 20:30:50.480850 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.480831 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 01 20:30:50.480965 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.480953 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 01 20:30:50.481181 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.481164 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 01 20:30:50.481242 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.481201 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pjdf7\"" Nov 01 20:30:50.481242 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.481201 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ggf7k\"" Nov 01 20:30:50.481343 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.481268 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 01 20:30:50.481392 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.481365 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 01 20:30:50.481575 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.481546 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 01 20:30:50.481670 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.481640 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-j72sw\"" Nov 01 20:30:50.481737 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.481657 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 20:30:50.481773 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.481749 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cg54x\"" Nov 01 20:30:50.481824 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.481797 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 01 20:30:50.481888 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.481872 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-r4cnd\"" Nov 01 20:30:50.482523 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.482494 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 01 20:30:50.482625 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.482572 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 20:30:50.482692 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.482630 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 01 20:30:50.482692 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.482656 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 01 20:30:50.483373 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.482661 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 01 20:30:50.483373 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.483163 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 20:30:50.483509 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.483495 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 01 20:30:50.483509 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.483504 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 01 20:30:50.483596 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.483554 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 01 20:30:50.483654 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.483611 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 01 20:30:50.483838 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.483553 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 01 20:30:50.483926 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.483908 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5cgzw\"" Nov 01 20:30:50.483974 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.483926 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 01 20:30:50.484123 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.484106 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-h288r\"" Nov 01 20:30:50.484175 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.484132 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 01 20:30:50.484221 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.484187 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 20:30:50.484276 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.484241 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6bdld\"" Nov 01 20:30:50.484458 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.484439 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 01 20:30:50.484817 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.484618 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-g9xjd\"" Nov 01 20:30:50.484817 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.484695 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 01 20:30:50.484817 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.484737 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-kzbx4\"" Nov 01 20:30:50.485073 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.484941 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-pskvw\"" Nov 01 20:30:50.487989 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.487970 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-ovnkube-script-lib\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:50.488428 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.488406 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a4342ddb-e06e-4d6d-9337-651349cf133f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fmnrq\" (UID: \"a4342ddb-e06e-4d6d-9337-651349cf133f\") " pod="openshift-multus/multus-additional-cni-plugins-fmnrq" Nov 01 20:30:50.488428 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.488416 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-ovn-node-metrics-cert\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:50.488540 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.488524 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a4342ddb-e06e-4d6d-9337-651349cf133f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fmnrq\" (UID: \"a4342ddb-e06e-4d6d-9337-651349cf133f\") " pod="openshift-multus/multus-additional-cni-plugins-fmnrq" Nov 01 20:30:50.489138 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.489122 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-cni-binary-copy\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:50.489181 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.489126 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b4f17a0-efd1-4024-a201-4078069b04e7-original-pull-secret\") pod \"global-pull-secret-syncer-xh6zg\" (UID: \"6b4f17a0-efd1-4024-a201-4078069b04e7\") " pod="kube-system/global-pull-secret-syncer-xh6zg" Nov 01 20:30:50.490328 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.490236 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5800567f-f0e1-4cf0-8338-75b09d2d1255-konnectivity-ca\") pod \"konnectivity-agent-2f2ct\" (UID: \"5800567f-f0e1-4cf0-8338-75b09d2d1255\") " pod="kube-system/konnectivity-agent-2f2ct" Nov 01 20:30:50.490328 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.490287 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5800567f-f0e1-4cf0-8338-75b09d2d1255-agent-certs\") pod \"konnectivity-agent-2f2ct\" (UID: \"5800567f-f0e1-4cf0-8338-75b09d2d1255\") " pod="kube-system/konnectivity-agent-2f2ct" Nov 01 20:30:50.490328 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.490287 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-multus-daemon-config\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:50.490519 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.490332 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-env-overrides\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:50.490519 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.490405 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a4342ddb-e06e-4d6d-9337-651349cf133f-cni-binary-copy\") pod \"multus-additional-cni-plugins-fmnrq\" (UID: \"a4342ddb-e06e-4d6d-9337-651349cf133f\") " pod="openshift-multus/multus-additional-cni-plugins-fmnrq" Nov 01 20:30:50.490519 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.490442 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2288d739-5eb3-4186-a32e-187945267de1-iptables-alerter-script\") pod \"iptables-alerter-w7qmz\" (UID: \"2288d739-5eb3-4186-a32e-187945267de1\") " pod="openshift-network-operator/iptables-alerter-w7qmz" Nov 01 20:30:50.490654 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.490541 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-ovnkube-config\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:50.490723 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.490704 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e8ad39cd-9b6a-464c-aa75-1bfc54b23891-serviceca\") pod \"node-ca-bztv7\" (UID: \"e8ad39cd-9b6a-464c-aa75-1bfc54b23891\") " pod="openshift-image-registry/node-ca-bztv7" Nov 01 20:30:50.491280 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.491257 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:50.506372 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.506352 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 01 20:30:50.506693 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.506677 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 01 20:30:50.506839 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.506793 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 01 20:30:50.506954 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.506937 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-mv7c7\"" Nov 01 20:30:50.507020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.507008 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 01 20:30:50.507069 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.507020 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-8jgj4\"" Nov 01 20:30:50.507890 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.507870 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 01 20:30:50.507890 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.507870 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 01 20:30:50.508062 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.507982 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 01 20:30:50.508062 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.508044 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 01 20:30:50.508062 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.508042 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 01 20:30:50.508191 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.508071 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 01 20:30:50.508191 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.508097 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 01 20:30:50.508191 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.508100 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 01 20:30:50.508191 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.508136 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4gdtb\"" Nov 01 20:30:50.508191 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.508141 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 01 20:30:50.508191 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.508176 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-n2j5c\"" Nov 01 20:30:50.510413 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.510393 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 01 20:30:50.510517 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.510448 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-t8z8c\"" Nov 01 20:30:50.510517 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.510474 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 01 20:30:50.512659 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.512643 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5pn9\" (UniqueName: \"kubernetes.io/projected/3fce64bd-a955-4e2c-8844-a25eaeb73862-kube-api-access-d5pn9\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:50.512737 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.512717 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khz9n\" (UniqueName: \"kubernetes.io/projected/e8ad39cd-9b6a-464c-aa75-1bfc54b23891-kube-api-access-khz9n\") pod \"node-ca-bztv7\" (UID: \"e8ad39cd-9b6a-464c-aa75-1bfc54b23891\") " pod="openshift-image-registry/node-ca-bztv7" Nov 01 20:30:50.512783 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.512717 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzsrc\" (UniqueName: \"kubernetes.io/projected/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-kube-api-access-qzsrc\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:50.528478 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.528447 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vm67\" (UniqueName: \"kubernetes.io/projected/970d71e5-f861-4277-8fa5-6ec740418e51-kube-api-access-6vm67\") pod \"network-metrics-daemon-vxzfk\" (UID: \"970d71e5-f861-4277-8fa5-6ec740418e51\") " pod="openshift-multus/network-metrics-daemon-vxzfk" Nov 01 20:30:50.528606 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.528534 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsbqz\" (UniqueName: \"kubernetes.io/projected/90e25e61-9761-4042-9dd4-88e1a6f8aa3c-kube-api-access-gsbqz\") pod \"aws-ebs-csi-driver-node-ljb94\" (UID: \"90e25e61-9761-4042-9dd4-88e1a6f8aa3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb94" Nov 01 20:30:50.528855 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.528841 4484 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 01 20:30:50.530382 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.530343 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px48q\" (UniqueName: \"kubernetes.io/projected/2288d739-5eb3-4186-a32e-187945267de1-kube-api-access-px48q\") pod \"iptables-alerter-w7qmz\" (UID: \"2288d739-5eb3-4186-a32e-187945267de1\") " pod="openshift-network-operator/iptables-alerter-w7qmz" Nov 01 20:30:50.530479 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.530353 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm9p9\" (UniqueName: \"kubernetes.io/projected/a4342ddb-e06e-4d6d-9337-651349cf133f-kube-api-access-mm9p9\") pod \"multus-additional-cni-plugins-fmnrq\" (UID: \"a4342ddb-e06e-4d6d-9337-651349cf133f\") " pod="openshift-multus/multus-additional-cni-plugins-fmnrq" Nov 01 20:30:50.530479 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.530422 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjfv4\" (UniqueName: \"kubernetes.io/projected/04ec71b7-390e-4e40-abce-1c395a7a51ab-kube-api-access-jjfv4\") pod \"node-resolver-w6nn2\" (UID: \"04ec71b7-390e-4e40-abce-1c395a7a51ab\") " pod="openshift-dns/node-resolver-w6nn2" Nov 01 20:30:50.530479 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.530432 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgl7f\" (UniqueName: \"kubernetes.io/projected/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-kube-api-access-vgl7f\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:50.530594 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.530528 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btdtr\" (UniqueName: \"kubernetes.io/projected/9ee67cbd-8115-4c48-b5cd-5299ed284bce-kube-api-access-btdtr\") pod \"network-check-target-ghh5z\" (UID: \"9ee67cbd-8115-4c48-b5cd-5299ed284bce\") " pod="openshift-network-diagnostics/network-check-target-ghh5z" Nov 01 20:30:50.541215 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.541197 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-run\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:50.541311 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.541220 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-cnibin\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:50.541311 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.541235 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-systemd-units\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:50.541311 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.541251 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4342ddb-e06e-4d6d-9337-651349cf133f-system-cni-dir\") pod \"multus-additional-cni-plugins-fmnrq\" (UID: \"a4342ddb-e06e-4d6d-9337-651349cf133f\") " pod="openshift-multus/multus-additional-cni-plugins-fmnrq" Nov 01 20:30:50.541311 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.541268 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-etc-sysctl-d\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:50.541311 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.541308 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4342ddb-e06e-4d6d-9337-651349cf133f-system-cni-dir\") pod \"multus-additional-cni-plugins-fmnrq\" (UID: \"a4342ddb-e06e-4d6d-9337-651349cf133f\") " pod="openshift-multus/multus-additional-cni-plugins-fmnrq" Nov 01 20:30:50.541564 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.541315 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-systemd-units\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:50.541564 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.541323 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-run\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:50.541564 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.541319 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-cnibin\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:50.541564 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.541371 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-run-systemd\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:50.541564 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.541384 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-run-systemd\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:50.541564 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.541401 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-etc-sysctl-conf\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:50.541564 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.541419 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-etc-sysctl-d\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:50.541564 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.541432 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-host-var-lib-cni-multus\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:50.541564 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.541476 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-host-var-lib-cni-multus\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:50.541564 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.541487 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-multus-conf-dir\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:50.541564 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.541517 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82ca0c95-053f-4bb0-ab8d-1d8cbd531667-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4ssvj\" (UID: \"82ca0c95-053f-4bb0-ab8d-1d8cbd531667\") " pod="openshift-insights/insights-runtime-extractor-4ssvj" Nov 01 20:30:50.541564 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.541519 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-multus-conf-dir\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:50.541564 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.541548 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-etc-sysctl-conf\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:50.541564 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.541564 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-host-run-netns\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:50.542161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.541599 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/82ca0c95-053f-4bb0-ab8d-1d8cbd531667-crio-socket\") pod \"insights-runtime-extractor-4ssvj\" (UID: \"82ca0c95-053f-4bb0-ab8d-1d8cbd531667\") " pod="openshift-insights/insights-runtime-extractor-4ssvj" Nov 01 20:30:50.542161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.541603 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-host-run-netns\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:50.542161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.541652 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bbb0eff1-ed70-4783-a255-374d8bf14687-config-volume\") pod \"dns-default-gtf4c\" (UID: \"bbb0eff1-ed70-4783-a255-374d8bf14687\") " pod="openshift-dns/dns-default-gtf4c" Nov 01 20:30:50.542161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.541686 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-etc-openvswitch\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:50.542161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.541709 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-run-ovn\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:50.542161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.541731 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-host-cni-netd\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:50.542161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.541771 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-etc-openvswitch\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:50.542161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.541778 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-host-cni-netd\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:50.542161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.541780 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-run-ovn\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:50.542161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.541795 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a4342ddb-e06e-4d6d-9337-651349cf133f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fmnrq\" (UID: \"a4342ddb-e06e-4d6d-9337-651349cf133f\") " pod="openshift-multus/multus-additional-cni-plugins-fmnrq" Nov 01 20:30:50.542161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.541819 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-sys\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:50.542161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.541826 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a4342ddb-e06e-4d6d-9337-651349cf133f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fmnrq\" (UID: \"a4342ddb-e06e-4d6d-9337-651349cf133f\") " pod="openshift-multus/multus-additional-cni-plugins-fmnrq" Nov 01 20:30:50.542161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.541844 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/04ec71b7-390e-4e40-abce-1c395a7a51ab-hosts-file\") pod \"node-resolver-w6nn2\" (UID: \"04ec71b7-390e-4e40-abce-1c395a7a51ab\") " pod="openshift-dns/node-resolver-w6nn2" Nov 01 20:30:50.542161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.541859 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bbb0eff1-ed70-4783-a255-374d8bf14687-config-volume\") pod \"dns-default-gtf4c\" (UID: \"bbb0eff1-ed70-4783-a255-374d8bf14687\") " pod="openshift-dns/dns-default-gtf4c" Nov 01 20:30:50.542161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.541867 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-host-slash\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:50.542161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.541904 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/04ec71b7-390e-4e40-abce-1c395a7a51ab-hosts-file\") pod \"node-resolver-w6nn2\" (UID: \"04ec71b7-390e-4e40-abce-1c395a7a51ab\") " pod="openshift-dns/node-resolver-w6nn2" Nov 01 20:30:50.542161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.541910 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-host-slash\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:50.542161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.541917 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-sys\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:50.542161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.541926 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-log-socket\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:50.542161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.541949 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/970d71e5-f861-4277-8fa5-6ec740418e51-metrics-certs\") pod \"network-metrics-daemon-vxzfk\" (UID: \"970d71e5-f861-4277-8fa5-6ec740418e51\") " pod="openshift-multus/network-metrics-daemon-vxzfk" Nov 01 20:30:50.542161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.541965 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-etc-sysconfig\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:50.542161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.541972 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-log-socket\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:50.542161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.541979 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-host-kubelet\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:50.542161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.542009 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/90e25e61-9761-4042-9dd4-88e1a6f8aa3c-device-dir\") pod \"aws-ebs-csi-driver-node-ljb94\" (UID: \"90e25e61-9761-4042-9dd4-88e1a6f8aa3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb94" Nov 01 20:30:50.542161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.542029 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f05cdcaf-b144-4c3d-b29b-2cb7d5a6f085-signing-cabundle\") pod \"service-ca-5fffc97bdf-smp9z\" (UID: \"f05cdcaf-b144-4c3d-b29b-2cb7d5a6f085\") " pod="openshift-service-ca/service-ca-5fffc97bdf-smp9z" Nov 01 20:30:50.542161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.542035 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-etc-sysconfig\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:50.542161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.542043 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-host-cni-bin\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:50.542161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.542046 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-host-kubelet\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:50.542161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.542068 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-lib-modules\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:50.542161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.542067 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-host-cni-bin\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:50.542161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.542105 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/90e25e61-9761-4042-9dd4-88e1a6f8aa3c-device-dir\") pod \"aws-ebs-csi-driver-node-ljb94\" (UID: \"90e25e61-9761-4042-9dd4-88e1a6f8aa3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb94" Nov 01 20:30:50.542161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.542135 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/90e25e61-9761-4042-9dd4-88e1a6f8aa3c-registration-dir\") pod \"aws-ebs-csi-driver-node-ljb94\" (UID: \"90e25e61-9761-4042-9dd4-88e1a6f8aa3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb94" Nov 01 20:30:50.542161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.542150 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-lib-modules\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:50.542161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.542161 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw4w8\" (UniqueName: \"kubernetes.io/projected/492a8e93-84b5-46d3-b6e4-4278f737249a-kube-api-access-sw4w8\") pod \"ingress-canary-mdc46\" (UID: \"492a8e93-84b5-46d3-b6e4-4278f737249a\") " pod="openshift-ingress-canary/ingress-canary-mdc46" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.542188 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/90e25e61-9761-4042-9dd4-88e1a6f8aa3c-registration-dir\") pod \"aws-ebs-csi-driver-node-ljb94\" (UID: \"90e25e61-9761-4042-9dd4-88e1a6f8aa3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb94" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.542183 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74vvb\" (UniqueName: \"kubernetes.io/projected/bbb0eff1-ed70-4783-a255-374d8bf14687-kube-api-access-74vvb\") pod \"dns-default-gtf4c\" (UID: \"bbb0eff1-ed70-4783-a255-374d8bf14687\") " pod="openshift-dns/dns-default-gtf4c" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.542222 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-host-run-ovn-kubernetes\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.542253 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f05cdcaf-b144-4c3d-b29b-2cb7d5a6f085-signing-cabundle\") pod \"service-ca-5fffc97bdf-smp9z\" (UID: \"f05cdcaf-b144-4c3d-b29b-2cb7d5a6f085\") " pod="openshift-service-ca/service-ca-5fffc97bdf-smp9z" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.542263 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-host-run-ovn-kubernetes\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.542270 4484 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.542288 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-host\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.542314 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-multus-socket-dir-parent\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.542338 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-host\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.542336 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-host-run-k8s-cni-cncf-io\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.542378 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-etc-kubernetes\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.542394 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-multus-socket-dir-parent\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.542406 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82ca0c95-053f-4bb0-ab8d-1d8cbd531667-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4ssvj\" (UID: \"82ca0c95-053f-4bb0-ab8d-1d8cbd531667\") " pod="openshift-insights/insights-runtime-extractor-4ssvj" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.542432 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-host-run-k8s-cni-cncf-io\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.542431 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-var-lib-openvswitch\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.542465 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-var-lib-openvswitch\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.542476 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-etc-kubernetes\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.542504 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-etc-modprobe-d\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.542529 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-host-var-lib-kubelet\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.542562 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kplvv\" (UniqueName: \"kubernetes.io/projected/82ca0c95-053f-4bb0-ab8d-1d8cbd531667-kube-api-access-kplvv\") pod \"insights-runtime-extractor-4ssvj\" (UID: \"82ca0c95-053f-4bb0-ab8d-1d8cbd531667\") " pod="openshift-insights/insights-runtime-extractor-4ssvj" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.542597 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-etc-modprobe-d\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.542612 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/492a8e93-84b5-46d3-b6e4-4278f737249a-cert\") pod \"ingress-canary-mdc46\" (UID: \"492a8e93-84b5-46d3-b6e4-4278f737249a\") " pod="openshift-ingress-canary/ingress-canary-mdc46" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.542637 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.542613 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82ca0c95-053f-4bb0-ab8d-1d8cbd531667-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4ssvj\" (UID: \"82ca0c95-053f-4bb0-ab8d-1d8cbd531667\") " pod="openshift-insights/insights-runtime-extractor-4ssvj" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.542663 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-host-var-lib-cni-bin\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.542599 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-host-var-lib-kubelet\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.542686 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2288d739-5eb3-4186-a32e-187945267de1-host-slash\") pod \"iptables-alerter-w7qmz\" (UID: \"2288d739-5eb3-4186-a32e-187945267de1\") " pod="openshift-network-operator/iptables-alerter-w7qmz" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.542727 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-host-var-lib-cni-bin\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.542730 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/90e25e61-9761-4042-9dd4-88e1a6f8aa3c-etc-selinux\") pod \"aws-ebs-csi-driver-node-ljb94\" (UID: \"90e25e61-9761-4042-9dd4-88e1a6f8aa3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb94" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.542750 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2288d739-5eb3-4186-a32e-187945267de1-host-slash\") pod \"iptables-alerter-w7qmz\" (UID: \"2288d739-5eb3-4186-a32e-187945267de1\") " pod="openshift-network-operator/iptables-alerter-w7qmz" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.542704 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.542762 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bbb0eff1-ed70-4783-a255-374d8bf14687-metrics-tls\") pod \"dns-default-gtf4c\" (UID: \"bbb0eff1-ed70-4783-a255-374d8bf14687\") " pod="openshift-dns/dns-default-gtf4c" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.542775 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/90e25e61-9761-4042-9dd4-88e1a6f8aa3c-etc-selinux\") pod \"aws-ebs-csi-driver-node-ljb94\" (UID: \"90e25e61-9761-4042-9dd4-88e1a6f8aa3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb94" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.542817 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6b4f17a0-efd1-4024-a201-4078069b04e7-kubelet-config\") pod \"global-pull-secret-syncer-xh6zg\" (UID: \"6b4f17a0-efd1-4024-a201-4078069b04e7\") " pod="kube-system/global-pull-secret-syncer-xh6zg" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.542844 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-system-cni-dir\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.542856 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/492a8e93-84b5-46d3-b6e4-4278f737249a-cert\") pod \"ingress-canary-mdc46\" (UID: \"492a8e93-84b5-46d3-b6e4-4278f737249a\") " pod="openshift-ingress-canary/ingress-canary-mdc46" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.542863 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6b4f17a0-efd1-4024-a201-4078069b04e7-kubelet-config\") pod \"global-pull-secret-syncer-xh6zg\" (UID: \"6b4f17a0-efd1-4024-a201-4078069b04e7\") " pod="kube-system/global-pull-secret-syncer-xh6zg" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.542871 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/90e25e61-9761-4042-9dd4-88e1a6f8aa3c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ljb94\" (UID: \"90e25e61-9761-4042-9dd4-88e1a6f8aa3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb94" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.542904 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-system-cni-dir\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.542920 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/90e25e61-9761-4042-9dd4-88e1a6f8aa3c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ljb94\" (UID: \"90e25e61-9761-4042-9dd4-88e1a6f8aa3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb94" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.542950 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/90e25e61-9761-4042-9dd4-88e1a6f8aa3c-sys-fs\") pod \"aws-ebs-csi-driver-node-ljb94\" (UID: \"90e25e61-9761-4042-9dd4-88e1a6f8aa3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb94" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.543023 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-etc-kubernetes\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.543068 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/90e25e61-9761-4042-9dd4-88e1a6f8aa3c-sys-fs\") pod \"aws-ebs-csi-driver-node-ljb94\" (UID: \"90e25e61-9761-4042-9dd4-88e1a6f8aa3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb94" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.543075 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6b4f17a0-efd1-4024-a201-4078069b04e7-dbus\") pod \"global-pull-secret-syncer-xh6zg\" (UID: \"6b4f17a0-efd1-4024-a201-4078069b04e7\") " pod="kube-system/global-pull-secret-syncer-xh6zg" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.543108 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-hostroot\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.543120 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6b4f17a0-efd1-4024-a201-4078069b04e7-dbus\") pod \"global-pull-secret-syncer-xh6zg\" (UID: \"6b4f17a0-efd1-4024-a201-4078069b04e7\") " pod="kube-system/global-pull-secret-syncer-xh6zg" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.543070 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-etc-kubernetes\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.543134 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-host-run-multus-certs\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.543153 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-hostroot\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.543161 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/90e25e61-9761-4042-9dd4-88e1a6f8aa3c-socket-dir\") pod \"aws-ebs-csi-driver-node-ljb94\" (UID: \"90e25e61-9761-4042-9dd4-88e1a6f8aa3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb94" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.543194 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-host-run-multus-certs\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.543198 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a4342ddb-e06e-4d6d-9337-651349cf133f-cnibin\") pod \"multus-additional-cni-plugins-fmnrq\" (UID: \"a4342ddb-e06e-4d6d-9337-651349cf133f\") " pod="openshift-multus/multus-additional-cni-plugins-fmnrq" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.543222 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-etc-systemd\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.543243 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/90e25e61-9761-4042-9dd4-88e1a6f8aa3c-socket-dir\") pod \"aws-ebs-csi-driver-node-ljb94\" (UID: \"90e25e61-9761-4042-9dd4-88e1a6f8aa3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljb94" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.543245 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-multus-cni-dir\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.543283 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a4342ddb-e06e-4d6d-9337-651349cf133f-cnibin\") pod \"multus-additional-cni-plugins-fmnrq\" (UID: \"a4342ddb-e06e-4d6d-9337-651349cf133f\") " pod="openshift-multus/multus-additional-cni-plugins-fmnrq" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.543292 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-etc-systemd\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.543300 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bcc5ee5c-08d8-49ba-9d7b-04e7a684e427-multus-cni-dir\") pod \"multus-5k2px\" (UID: \"bcc5ee5c-08d8-49ba-9d7b-04e7a684e427\") " pod="openshift-multus/multus-5k2px" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.543300 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhml7\" (UniqueName: \"kubernetes.io/projected/f05cdcaf-b144-4c3d-b29b-2cb7d5a6f085-kube-api-access-dhml7\") pod \"service-ca-5fffc97bdf-smp9z\" (UID: \"f05cdcaf-b144-4c3d-b29b-2cb7d5a6f085\") " pod="openshift-service-ca/service-ca-5fffc97bdf-smp9z" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.543339 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbb0eff1-ed70-4783-a255-374d8bf14687-tmp-dir\") pod \"dns-default-gtf4c\" (UID: \"bbb0eff1-ed70-4783-a255-374d8bf14687\") " pod="openshift-dns/dns-default-gtf4c" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.543387 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhrjv\" (UniqueName: \"kubernetes.io/projected/8f213a88-8cb5-467b-8d21-9bc86a0654c9-kube-api-access-lhrjv\") pod \"migrator-66df5d496b-2lxrg\" (UID: \"8f213a88-8cb5-467b-8d21-9bc86a0654c9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2lxrg" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.543420 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbb0eff1-ed70-4783-a255-374d8bf14687-tmp-dir\") pod \"dns-default-gtf4c\" (UID: \"bbb0eff1-ed70-4783-a255-374d8bf14687\") " pod="openshift-dns/dns-default-gtf4c" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.543426 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-run-openvswitch\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.543457 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-var-lib-kubelet\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.543473 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-run-openvswitch\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.543483 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e8ad39cd-9b6a-464c-aa75-1bfc54b23891-host\") pod \"node-ca-bztv7\" (UID: \"e8ad39cd-9b6a-464c-aa75-1bfc54b23891\") " pod="openshift-image-registry/node-ca-bztv7" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.543514 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/82ca0c95-053f-4bb0-ab8d-1d8cbd531667-data-volume\") pod \"insights-runtime-extractor-4ssvj\" (UID: \"82ca0c95-053f-4bb0-ab8d-1d8cbd531667\") " pod="openshift-insights/insights-runtime-extractor-4ssvj" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.543523 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fce64bd-a955-4e2c-8844-a25eaeb73862-var-lib-kubelet\") pod \"tuned-qjtnx\" (UID: \"3fce64bd-a955-4e2c-8844-a25eaeb73862\") " pod="openshift-cluster-node-tuning-operator/tuned-qjtnx" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.543538 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f05cdcaf-b144-4c3d-b29b-2cb7d5a6f085-signing-key\") pod \"service-ca-5fffc97bdf-smp9z\" (UID: \"f05cdcaf-b144-4c3d-b29b-2cb7d5a6f085\") " pod="openshift-service-ca/service-ca-5fffc97bdf-smp9z" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.543574 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e8ad39cd-9b6a-464c-aa75-1bfc54b23891-host\") pod \"node-ca-bztv7\" (UID: \"e8ad39cd-9b6a-464c-aa75-1bfc54b23891\") " pod="openshift-image-registry/node-ca-bztv7" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.543594 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-host-run-netns\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.543609 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-node-log\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.543600 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/82ca0c95-053f-4bb0-ab8d-1d8cbd531667-data-volume\") pod \"insights-runtime-extractor-4ssvj\" (UID: \"82ca0c95-053f-4bb0-ab8d-1d8cbd531667\") " pod="openshift-insights/insights-runtime-extractor-4ssvj" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.543640 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-node-log\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:50.543616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.543645 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc-host-run-netns\") pod \"ovnkube-node-tv7jl\" (UID: \"cad48e5e-da24-4aa6-bbb4-3f24c9fd4bfc\") " pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:50.546788 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.543751 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f05cdcaf-b144-4c3d-b29b-2cb7d5a6f085-signing-key\") pod \"service-ca-5fffc97bdf-smp9z\" (UID: \"f05cdcaf-b144-4c3d-b29b-2cb7d5a6f085\") " pod="openshift-service-ca/service-ca-5fffc97bdf-smp9z" Nov 01 20:30:50.546788 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.544569 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/970d71e5-f861-4277-8fa5-6ec740418e51-metrics-certs\") pod \"network-metrics-daemon-vxzfk\" (UID: \"970d71e5-f861-4277-8fa5-6ec740418e51\") " pod="openshift-multus/network-metrics-daemon-vxzfk" Nov 01 20:30:50.598625 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.598533 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74vvb\" (UniqueName: \"kubernetes.io/projected/bbb0eff1-ed70-4783-a255-374d8bf14687-kube-api-access-74vvb\") pod \"dns-default-gtf4c\" (UID: \"bbb0eff1-ed70-4783-a255-374d8bf14687\") " pod="openshift-dns/dns-default-gtf4c" Nov 01 20:30:50.598973 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.598952 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw4w8\" (UniqueName: \"kubernetes.io/projected/492a8e93-84b5-46d3-b6e4-4278f737249a-kube-api-access-sw4w8\") pod \"ingress-canary-mdc46\" (UID: \"492a8e93-84b5-46d3-b6e4-4278f737249a\") " pod="openshift-ingress-canary/ingress-canary-mdc46" Nov 01 20:30:50.599062 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.598956 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kplvv\" (UniqueName: \"kubernetes.io/projected/82ca0c95-053f-4bb0-ab8d-1d8cbd531667-kube-api-access-kplvv\") pod \"insights-runtime-extractor-4ssvj\" (UID: \"82ca0c95-053f-4bb0-ab8d-1d8cbd531667\") " pod="openshift-insights/insights-runtime-extractor-4ssvj" Nov 01 20:30:50.599315 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.599295 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhml7\" (UniqueName: \"kubernetes.io/projected/f05cdcaf-b144-4c3d-b29b-2cb7d5a6f085-kube-api-access-dhml7\") pod \"service-ca-5fffc97bdf-smp9z\" (UID: \"f05cdcaf-b144-4c3d-b29b-2cb7d5a6f085\") " pod="openshift-service-ca/service-ca-5fffc97bdf-smp9z" Nov 01 20:30:50.599513 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.599496 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhrjv\" (UniqueName: \"kubernetes.io/projected/8f213a88-8cb5-467b-8d21-9bc86a0654c9-kube-api-access-lhrjv\") pod \"migrator-66df5d496b-2lxrg\" (UID: \"8f213a88-8cb5-467b-8d21-9bc86a0654c9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2lxrg" Nov 01 20:30:50.644097 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.644063 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82ca0c95-053f-4bb0-ab8d-1d8cbd531667-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4ssvj\" (UID: \"82ca0c95-053f-4bb0-ab8d-1d8cbd531667\") " pod="openshift-insights/insights-runtime-extractor-4ssvj" Nov 01 20:30:50.644097 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.644101 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/82ca0c95-053f-4bb0-ab8d-1d8cbd531667-crio-socket\") pod \"insights-runtime-extractor-4ssvj\" (UID: \"82ca0c95-053f-4bb0-ab8d-1d8cbd531667\") " pod="openshift-insights/insights-runtime-extractor-4ssvj" Nov 01 20:30:50.644320 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.644140 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bbb0eff1-ed70-4783-a255-374d8bf14687-metrics-tls\") pod \"dns-default-gtf4c\" (UID: \"bbb0eff1-ed70-4783-a255-374d8bf14687\") " pod="openshift-dns/dns-default-gtf4c" Nov 01 20:30:50.644320 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.644258 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/82ca0c95-053f-4bb0-ab8d-1d8cbd531667-crio-socket\") pod \"insights-runtime-extractor-4ssvj\" (UID: \"82ca0c95-053f-4bb0-ab8d-1d8cbd531667\") " pod="openshift-insights/insights-runtime-extractor-4ssvj" Nov 01 20:30:50.646589 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.646554 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82ca0c95-053f-4bb0-ab8d-1d8cbd531667-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4ssvj\" (UID: \"82ca0c95-053f-4bb0-ab8d-1d8cbd531667\") " pod="openshift-insights/insights-runtime-extractor-4ssvj" Nov 01 20:30:50.646708 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.646618 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bbb0eff1-ed70-4783-a255-374d8bf14687-metrics-tls\") pod \"dns-default-gtf4c\" (UID: \"bbb0eff1-ed70-4783-a255-374d8bf14687\") " pod="openshift-dns/dns-default-gtf4c" Nov 01 20:30:50.726656 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.726624 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxzfk" Nov 01 20:30:50.730740 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.730709 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gtf4c" Nov 01 20:30:50.730740 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.730723 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4ssvj" Nov 01 20:30:50.819450 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.819427 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ghh5z" Nov 01 20:30:50.883425 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:50.883345 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:51.098156 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:51.098119 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:51.138449 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:51.138380 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vxzfk"] Nov 01 20:30:51.141353 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:51.141324 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod970d71e5_f861_4277_8fa5_6ec740418e51.slice/crio-a5e04c686c30e9b1f448d851c437b7ab3c6fe45f841b893fd3d7cbfd2c97dd23 WatchSource:0}: Error finding container a5e04c686c30e9b1f448d851c437b7ab3c6fe45f841b893fd3d7cbfd2c97dd23: Status 404 returned error can't find the container with id a5e04c686c30e9b1f448d851c437b7ab3c6fe45f841b893fd3d7cbfd2c97dd23 Nov 01 20:30:51.146381 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:51.146361 4484 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 20:30:51.191178 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:51.191145 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gtf4c"] Nov 01 20:30:51.194149 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:51.194122 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbb0eff1_ed70_4783_a255_374d8bf14687.slice/crio-95fbd5b5b401d7ad49702ad222ac59c7763e760ef71d7d24f68d3b0bbdcea5e7 WatchSource:0}: Error finding container 95fbd5b5b401d7ad49702ad222ac59c7763e760ef71d7d24f68d3b0bbdcea5e7: Status 404 returned error can't find the container with id 95fbd5b5b401d7ad49702ad222ac59c7763e760ef71d7d24f68d3b0bbdcea5e7 Nov 01 20:30:51.220960 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:51.220934 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4ssvj"] Nov 01 20:30:51.224579 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:30:51.224544 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82ca0c95_053f_4bb0_ab8d_1d8cbd531667.slice/crio-6ce57fdc1b9308c728beaf11b22893481d7ff1dc9ca81d083d8739c48572e00d WatchSource:0}: Error finding container 6ce57fdc1b9308c728beaf11b22893481d7ff1dc9ca81d083d8739c48572e00d: Status 404 returned error can't find the container with id 6ce57fdc1b9308c728beaf11b22893481d7ff1dc9ca81d083d8739c48572e00d Nov 01 20:30:51.293216 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:51.293176 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:51.405290 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:51.405194 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gtf4c" event={"ID":"bbb0eff1-ed70-4783-a255-374d8bf14687","Type":"ContainerStarted","Data":"95fbd5b5b401d7ad49702ad222ac59c7763e760ef71d7d24f68d3b0bbdcea5e7"} Nov 01 20:30:51.406641 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:51.406608 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vxzfk" event={"ID":"970d71e5-f861-4277-8fa5-6ec740418e51","Type":"ContainerStarted","Data":"a5e04c686c30e9b1f448d851c437b7ab3c6fe45f841b893fd3d7cbfd2c97dd23"} Nov 01 20:30:51.408260 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:51.408236 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4ssvj" event={"ID":"82ca0c95-053f-4bb0-ab8d-1d8cbd531667","Type":"ContainerStarted","Data":"91361ec5da06b04b463ef0492ed8bc2ef50750203973720ab79839654984f229"} Nov 01 20:30:51.408395 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:51.408284 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4ssvj" event={"ID":"82ca0c95-053f-4bb0-ab8d-1d8cbd531667","Type":"ContainerStarted","Data":"6ce57fdc1b9308c728beaf11b22893481d7ff1dc9ca81d083d8739c48572e00d"} Nov 01 20:30:51.492841 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:51.492810 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:51.581492 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:51.581464 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:52.712607 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:52.712568 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:52.959320 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:52.959275 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:53.119111 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:53.119084 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:53.291989 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:53.291887 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:53.416723 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:53.416676 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4ssvj" event={"ID":"82ca0c95-053f-4bb0-ab8d-1d8cbd531667","Type":"ContainerStarted","Data":"6b5e11ed723c03d69dfdff3871262bbeafd76b7283fc65bbae6887c0f6920171"} Nov 01 20:30:53.420464 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:53.420435 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vxzfk" event={"ID":"970d71e5-f861-4277-8fa5-6ec740418e51","Type":"ContainerStarted","Data":"636c8b3791205ca936f8734975d11b97f9e9adf77185cdf7b327d34d7698cd8c"} Nov 01 20:30:53.420613 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:53.420487 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vxzfk" event={"ID":"970d71e5-f861-4277-8fa5-6ec740418e51","Type":"ContainerStarted","Data":"ffd3c04f8a3c5e1c7079e542a3f62eee83fcb07f957ddfe09a787e42e4bd70e2"} Nov 01 20:30:53.488572 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:53.488525 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:54.426489 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:54.426445 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gtf4c" event={"ID":"bbb0eff1-ed70-4783-a255-374d8bf14687","Type":"ContainerStarted","Data":"e458b516f213e3a530761b38aa1d86f15a21594a37d385984b773cc8a1105b13"} Nov 01 20:30:54.426962 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:54.426509 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gtf4c" event={"ID":"bbb0eff1-ed70-4783-a255-374d8bf14687","Type":"ContainerStarted","Data":"562709ec87f3fdf6a02abe5a90503b86a0205926a63fc0126a5481b30f1964ff"} Nov 01 20:30:54.426962 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:54.426807 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-gtf4c" Nov 01 20:30:54.487412 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:54.487375 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:54.884363 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:54.884337 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:55.095265 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:55.095214 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:55.289597 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:55.289536 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:55.431380 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:55.431334 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4ssvj" event={"ID":"82ca0c95-053f-4bb0-ab8d-1d8cbd531667","Type":"ContainerStarted","Data":"22d4881bd72facf878fded4a7bf9a1dc4b3cc6e69d4bf79058d4c273576b84fc"} Nov 01 20:30:55.683839 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:55.683753 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:56.492965 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:56.492918 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:56.886793 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:56.886721 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:56.891156 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:56.891128 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:30:56.891327 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:56.891316 4484 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 20:30:57.124035 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:57.123956 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:57.287415 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:57.287384 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:57.519555 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:57.519525 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:57.693407 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:57.693306 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:57.940606 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:57.940536 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:58.102079 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:58.102031 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:58.559902 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:58.559852 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:59.016034 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:59.015960 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:59.164487 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:59.164451 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:59.814005 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:30:59.813973 4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-35.us-west-1.compute.internal" event="NodeReady" Nov 01 20:31:00.855321 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:00.855290 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2f2ct" Nov 01 20:31:01.990327 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:01.990287 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-ngxhs"] Nov 01 20:31:01.992308 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:01.992287 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-ngxhs" Nov 01 20:31:02.022715 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:02.022684 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs9wl\" (UniqueName: \"kubernetes.io/projected/65896401-c413-42f3-af9f-4fcf04cacadb-kube-api-access-rs9wl\") pod \"downloads-6dd789c8b7-ngxhs\" (UID: \"65896401-c413-42f3-af9f-4fcf04cacadb\") " pod="openshift-console/downloads-6dd789c8b7-ngxhs" Nov 01 20:31:02.022892 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:02.022726 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65896401-c413-42f3-af9f-4fcf04cacadb-tmp\") pod \"downloads-6dd789c8b7-ngxhs\" (UID: \"65896401-c413-42f3-af9f-4fcf04cacadb\") " pod="openshift-console/downloads-6dd789c8b7-ngxhs" Nov 01 20:31:02.046817 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:02.046776 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 01 20:31:02.046817 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:02.046793 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 01 20:31:02.081123 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:02.081086 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-njpqp\"" Nov 01 20:31:02.098758 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:02.098726 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-fdf47dfdc-r8c58"] Nov 01 20:31:02.100636 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:02.100607 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-ngxhs"] Nov 01 20:31:02.100785 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:02.100701 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fdf47dfdc-r8c58" Nov 01 20:31:02.123422 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:02.123388 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs9wl\" (UniqueName: \"kubernetes.io/projected/65896401-c413-42f3-af9f-4fcf04cacadb-kube-api-access-rs9wl\") pod \"downloads-6dd789c8b7-ngxhs\" (UID: \"65896401-c413-42f3-af9f-4fcf04cacadb\") " pod="openshift-console/downloads-6dd789c8b7-ngxhs" Nov 01 20:31:02.123422 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:02.123429 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65896401-c413-42f3-af9f-4fcf04cacadb-tmp\") pod \"downloads-6dd789c8b7-ngxhs\" (UID: \"65896401-c413-42f3-af9f-4fcf04cacadb\") " pod="openshift-console/downloads-6dd789c8b7-ngxhs" Nov 01 20:31:02.123649 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:02.123449 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/99ebc016-9398-4982-97c7-54eab90ffa5f-trusted-ca\") pod \"image-registry-fdf47dfdc-r8c58\" (UID: \"99ebc016-9398-4982-97c7-54eab90ffa5f\") " pod="openshift-image-registry/image-registry-fdf47dfdc-r8c58" Nov 01 20:31:02.123649 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:02.123469 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/99ebc016-9398-4982-97c7-54eab90ffa5f-registry-tls\") pod \"image-registry-fdf47dfdc-r8c58\" (UID: \"99ebc016-9398-4982-97c7-54eab90ffa5f\") " pod="openshift-image-registry/image-registry-fdf47dfdc-r8c58" Nov 01 20:31:02.123649 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:02.123549 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/99ebc016-9398-4982-97c7-54eab90ffa5f-installation-pull-secrets\") pod \"image-registry-fdf47dfdc-r8c58\" (UID: \"99ebc016-9398-4982-97c7-54eab90ffa5f\") " pod="openshift-image-registry/image-registry-fdf47dfdc-r8c58" Nov 01 20:31:02.123790 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:02.123645 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkhsm\" (UniqueName: \"kubernetes.io/projected/99ebc016-9398-4982-97c7-54eab90ffa5f-kube-api-access-pkhsm\") pod \"image-registry-fdf47dfdc-r8c58\" (UID: \"99ebc016-9398-4982-97c7-54eab90ffa5f\") " pod="openshift-image-registry/image-registry-fdf47dfdc-r8c58" Nov 01 20:31:02.123790 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:02.123715 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/99ebc016-9398-4982-97c7-54eab90ffa5f-image-registry-private-configuration\") pod \"image-registry-fdf47dfdc-r8c58\" (UID: \"99ebc016-9398-4982-97c7-54eab90ffa5f\") " pod="openshift-image-registry/image-registry-fdf47dfdc-r8c58" Nov 01 20:31:02.123790 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:02.123778 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/99ebc016-9398-4982-97c7-54eab90ffa5f-ca-trust-extracted\") pod \"image-registry-fdf47dfdc-r8c58\" (UID: \"99ebc016-9398-4982-97c7-54eab90ffa5f\") " pod="openshift-image-registry/image-registry-fdf47dfdc-r8c58" Nov 01 20:31:02.123900 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:02.123806 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/99ebc016-9398-4982-97c7-54eab90ffa5f-bound-sa-token\") pod \"image-registry-fdf47dfdc-r8c58\" (UID: \"99ebc016-9398-4982-97c7-54eab90ffa5f\") " pod="openshift-image-registry/image-registry-fdf47dfdc-r8c58" Nov 01 20:31:02.123900 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:02.123815 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65896401-c413-42f3-af9f-4fcf04cacadb-tmp\") pod \"downloads-6dd789c8b7-ngxhs\" (UID: \"65896401-c413-42f3-af9f-4fcf04cacadb\") " pod="openshift-console/downloads-6dd789c8b7-ngxhs" Nov 01 20:31:02.123900 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:02.123826 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/99ebc016-9398-4982-97c7-54eab90ffa5f-registry-certificates\") pod \"image-registry-fdf47dfdc-r8c58\" (UID: \"99ebc016-9398-4982-97c7-54eab90ffa5f\") " pod="openshift-image-registry/image-registry-fdf47dfdc-r8c58" Nov 01 20:31:02.151750 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:02.150721 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-fdf47dfdc-r8c58"] Nov 01 20:31:02.155702 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:02.155676 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 01 20:31:02.156577 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:02.156559 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 01 20:31:02.156635 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:02.156566 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 01 20:31:02.156670 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:02.156657 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9chrm\"" Nov 01 20:31:02.213291 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:02.213258 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:02.219607 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:02.219583 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 01 20:31:02.222122 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:02.222095 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs9wl\" (UniqueName: \"kubernetes.io/projected/65896401-c413-42f3-af9f-4fcf04cacadb-kube-api-access-rs9wl\") pod \"downloads-6dd789c8b7-ngxhs\" (UID: \"65896401-c413-42f3-af9f-4fcf04cacadb\") " pod="openshift-console/downloads-6dd789c8b7-ngxhs" Nov 01 20:31:02.224848 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:02.224828 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/99ebc016-9398-4982-97c7-54eab90ffa5f-trusted-ca\") pod \"image-registry-fdf47dfdc-r8c58\" (UID: \"99ebc016-9398-4982-97c7-54eab90ffa5f\") " pod="openshift-image-registry/image-registry-fdf47dfdc-r8c58" Nov 01 20:31:02.224910 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:02.224857 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/99ebc016-9398-4982-97c7-54eab90ffa5f-registry-tls\") pod \"image-registry-fdf47dfdc-r8c58\" (UID: \"99ebc016-9398-4982-97c7-54eab90ffa5f\") " pod="openshift-image-registry/image-registry-fdf47dfdc-r8c58" Nov 01 20:31:02.224910 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:02.224881 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/99ebc016-9398-4982-97c7-54eab90ffa5f-installation-pull-secrets\") pod \"image-registry-fdf47dfdc-r8c58\" (UID: \"99ebc016-9398-4982-97c7-54eab90ffa5f\") " pod="openshift-image-registry/image-registry-fdf47dfdc-r8c58" Nov 01 20:31:02.224910 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:02.224896 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkhsm\" (UniqueName: \"kubernetes.io/projected/99ebc016-9398-4982-97c7-54eab90ffa5f-kube-api-access-pkhsm\") pod \"image-registry-fdf47dfdc-r8c58\" (UID: \"99ebc016-9398-4982-97c7-54eab90ffa5f\") " pod="openshift-image-registry/image-registry-fdf47dfdc-r8c58" Nov 01 20:31:02.225120 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:02.225033 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/99ebc016-9398-4982-97c7-54eab90ffa5f-image-registry-private-configuration\") pod \"image-registry-fdf47dfdc-r8c58\" (UID: \"99ebc016-9398-4982-97c7-54eab90ffa5f\") " pod="openshift-image-registry/image-registry-fdf47dfdc-r8c58" Nov 01 20:31:02.225120 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:02.225063 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/99ebc016-9398-4982-97c7-54eab90ffa5f-ca-trust-extracted\") pod \"image-registry-fdf47dfdc-r8c58\" (UID: \"99ebc016-9398-4982-97c7-54eab90ffa5f\") " pod="openshift-image-registry/image-registry-fdf47dfdc-r8c58" Nov 01 20:31:02.225120 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:02.225105 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/99ebc016-9398-4982-97c7-54eab90ffa5f-bound-sa-token\") pod \"image-registry-fdf47dfdc-r8c58\" (UID: \"99ebc016-9398-4982-97c7-54eab90ffa5f\") " pod="openshift-image-registry/image-registry-fdf47dfdc-r8c58" Nov 01 20:31:02.225273 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:02.225133 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/99ebc016-9398-4982-97c7-54eab90ffa5f-registry-certificates\") pod \"image-registry-fdf47dfdc-r8c58\" (UID: \"99ebc016-9398-4982-97c7-54eab90ffa5f\") " pod="openshift-image-registry/image-registry-fdf47dfdc-r8c58" Nov 01 20:31:02.225575 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:02.225536 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/99ebc016-9398-4982-97c7-54eab90ffa5f-ca-trust-extracted\") pod \"image-registry-fdf47dfdc-r8c58\" (UID: \"99ebc016-9398-4982-97c7-54eab90ffa5f\") " pod="openshift-image-registry/image-registry-fdf47dfdc-r8c58" Nov 01 20:31:02.225926 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:02.225900 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/99ebc016-9398-4982-97c7-54eab90ffa5f-registry-certificates\") pod \"image-registry-fdf47dfdc-r8c58\" (UID: \"99ebc016-9398-4982-97c7-54eab90ffa5f\") " pod="openshift-image-registry/image-registry-fdf47dfdc-r8c58" Nov 01 20:31:02.226147 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:02.226122 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/99ebc016-9398-4982-97c7-54eab90ffa5f-trusted-ca\") pod \"image-registry-fdf47dfdc-r8c58\" (UID: \"99ebc016-9398-4982-97c7-54eab90ffa5f\") " pod="openshift-image-registry/image-registry-fdf47dfdc-r8c58" Nov 01 20:31:02.227651 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:02.227626 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/99ebc016-9398-4982-97c7-54eab90ffa5f-installation-pull-secrets\") pod \"image-registry-fdf47dfdc-r8c58\" (UID: \"99ebc016-9398-4982-97c7-54eab90ffa5f\") " pod="openshift-image-registry/image-registry-fdf47dfdc-r8c58" Nov 01 20:31:02.227725 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:02.227629 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/99ebc016-9398-4982-97c7-54eab90ffa5f-registry-tls\") pod \"image-registry-fdf47dfdc-r8c58\" (UID: \"99ebc016-9398-4982-97c7-54eab90ffa5f\") " pod="openshift-image-registry/image-registry-fdf47dfdc-r8c58" Nov 01 20:31:02.227769 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:02.227735 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/99ebc016-9398-4982-97c7-54eab90ffa5f-image-registry-private-configuration\") pod \"image-registry-fdf47dfdc-r8c58\" (UID: \"99ebc016-9398-4982-97c7-54eab90ffa5f\") " pod="openshift-image-registry/image-registry-fdf47dfdc-r8c58" Nov 01 20:31:02.289210 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:02.289144 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/99ebc016-9398-4982-97c7-54eab90ffa5f-bound-sa-token\") pod \"image-registry-fdf47dfdc-r8c58\" (UID: \"99ebc016-9398-4982-97c7-54eab90ffa5f\") " pod="openshift-image-registry/image-registry-fdf47dfdc-r8c58" Nov 01 20:31:02.289339 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:02.289298 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkhsm\" (UniqueName: \"kubernetes.io/projected/99ebc016-9398-4982-97c7-54eab90ffa5f-kube-api-access-pkhsm\") pod \"image-registry-fdf47dfdc-r8c58\" (UID: \"99ebc016-9398-4982-97c7-54eab90ffa5f\") " pod="openshift-image-registry/image-registry-fdf47dfdc-r8c58" Nov 01 20:31:02.302305 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:02.302275 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-ngxhs" Nov 01 20:31:02.373906 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:02.373877 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:02.410588 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:02.410546 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fdf47dfdc-r8c58" Nov 01 20:31:02.529790 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:02.529756 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:02.690382 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:02.690303 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:02.714423 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:02.714390 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-ngxhs"] Nov 01 20:31:02.717538 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:31:02.717510 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65896401_c413_42f3_af9f_4fcf04cacadb.slice/crio-aab8f88c7fea6b6238937eae6e29c40147f0f23ebd63ea5d6554f258559e5e0e WatchSource:0}: Error finding container aab8f88c7fea6b6238937eae6e29c40147f0f23ebd63ea5d6554f258559e5e0e: Status 404 returned error can't find the container with id aab8f88c7fea6b6238937eae6e29c40147f0f23ebd63ea5d6554f258559e5e0e Nov 01 20:31:02.779873 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:02.779845 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-fdf47dfdc-r8c58"] Nov 01 20:31:02.782608 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:31:02.782583 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99ebc016_9398_4982_97c7_54eab90ffa5f.slice/crio-51076304f2249ee5032c8f1d27e94b769a7d31323a3cf56e3b8fa9e4ce30cc75 WatchSource:0}: Error finding container 51076304f2249ee5032c8f1d27e94b769a7d31323a3cf56e3b8fa9e4ce30cc75: Status 404 returned error can't find the container with id 51076304f2249ee5032c8f1d27e94b769a7d31323a3cf56e3b8fa9e4ce30cc75 Nov 01 20:31:02.857866 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:02.857833 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:03.458408 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:03.458368 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-ngxhs" event={"ID":"65896401-c413-42f3-af9f-4fcf04cacadb","Type":"ContainerStarted","Data":"aab8f88c7fea6b6238937eae6e29c40147f0f23ebd63ea5d6554f258559e5e0e"} Nov 01 20:31:03.459862 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:03.459832 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fdf47dfdc-r8c58" event={"ID":"99ebc016-9398-4982-97c7-54eab90ffa5f","Type":"ContainerStarted","Data":"770cc1db7f237531474e89355738e66b8b8d686a3c2fcda7ab1a4ae21db3bee2"} Nov 01 20:31:03.459988 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:03.459912 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fdf47dfdc-r8c58" event={"ID":"99ebc016-9398-4982-97c7-54eab90ffa5f","Type":"ContainerStarted","Data":"51076304f2249ee5032c8f1d27e94b769a7d31323a3cf56e3b8fa9e4ce30cc75"} Nov 01 20:31:03.459988 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:03.459983 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-fdf47dfdc-r8c58" Nov 01 20:31:03.578632 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:03.578580 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-fdf47dfdc-r8c58" podStartSLOduration=1.5785611899999998 podStartE2EDuration="1.57856119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:31:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:31:03.578211344 +0000 UTC m=+14.355851479" watchObservedRunningTime="2026-11-01 20:31:03.57856119 +0000 UTC m=+14.356201319" Nov 01 20:31:04.210750 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:04.210714 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:04.378301 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:04.378264 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:04.434501 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:04.434474 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gtf4c" Nov 01 20:31:04.531956 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:04.531913 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:05.257017 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:05.256948 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:05.316842 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:05.316812 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-j9k6m"] Nov 01 20:31:05.318667 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:05.318645 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-j9k6m" Nov 01 20:31:05.347290 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:05.347259 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a7dc6f05-bd61-4286-b8ee-078fa119068e-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-j9k6m\" (UID: \"a7dc6f05-bd61-4286-b8ee-078fa119068e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-j9k6m" Nov 01 20:31:05.375747 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:05.375713 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-j9k6m"] Nov 01 20:31:05.406552 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:05.406516 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:05.422554 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:05.422520 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-ltp6m\"" Nov 01 20:31:05.422709 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:05.422615 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 01 20:31:05.448561 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:05.448529 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a7dc6f05-bd61-4286-b8ee-078fa119068e-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-j9k6m\" (UID: \"a7dc6f05-bd61-4286-b8ee-078fa119068e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-j9k6m" Nov 01 20:31:05.451454 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:05.451417 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a7dc6f05-bd61-4286-b8ee-078fa119068e-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-j9k6m\" (UID: \"a7dc6f05-bd61-4286-b8ee-078fa119068e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-j9k6m" Nov 01 20:31:05.631639 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:05.631552 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-j9k6m" Nov 01 20:31:06.007328 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:06.007299 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-j9k6m"] Nov 01 20:31:06.010342 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:31:06.010317 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7dc6f05_bd61_4286_b8ee_078fa119068e.slice/crio-d5a887292aff40b95d6949461d9cad13e439d33d8f23896b70ddcbfe61d656e9 WatchSource:0}: Error finding container d5a887292aff40b95d6949461d9cad13e439d33d8f23896b70ddcbfe61d656e9: Status 404 returned error can't find the container with id d5a887292aff40b95d6949461d9cad13e439d33d8f23896b70ddcbfe61d656e9 Nov 01 20:31:06.479007 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:06.478914 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-j9k6m" event={"ID":"a7dc6f05-bd61-4286-b8ee-078fa119068e","Type":"ContainerStarted","Data":"d5a887292aff40b95d6949461d9cad13e439d33d8f23896b70ddcbfe61d656e9"} Nov 01 20:31:07.544757 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:07.544720 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:07.749519 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:07.749491 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:07.912679 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:07.912592 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:08.488014 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:08.487960 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-j9k6m" event={"ID":"a7dc6f05-bd61-4286-b8ee-078fa119068e","Type":"ContainerStarted","Data":"5ff2308c7b13c002016847b0e3158f4c13c2ed7a77c4b3fdb829a13a60f64de5"} Nov 01 20:31:08.488319 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:08.488293 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-j9k6m" Nov 01 20:31:08.494508 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:08.494483 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-j9k6m" Nov 01 20:31:08.626739 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:08.626694 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:08.716182 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:08.716116 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-j9k6m" podStartSLOduration=2.039792444 podStartE2EDuration="3.716101132s" totalImagesPullingTime="1.676308688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:31:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:31:08.602130311 +0000 UTC m=+19.379770447" watchObservedRunningTime="2026-11-01 20:31:08.716101132 +0000 UTC m=+19.493741267" Nov 01 20:31:08.805852 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:08.805773 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:09.536674 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:09.536631 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:09.697512 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:09.697475 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:09.849797 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:09.849716 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:10.048907 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:10.048871 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:10.629625 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:10.629593 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tv7jl" Nov 01 20:31:14.439688 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:14.439643 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:14.631151 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:14.631109 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:15.294462 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:15.294425 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:15.772905 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:31:15.772867 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 20:31:15.777270 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:15.777236 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:15.995594 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:15.995559 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:16.178451 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.178321 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:16.369357 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.369320 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:16.443967 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.443885 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-dk6sz"] Nov 01 20:31:16.446696 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.446670 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dk6sz" Nov 01 20:31:16.529288 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.529250 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3635e179-1be9-4756-9594-7e29268eb6cc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-dk6sz\" (UID: \"3635e179-1be9-4756-9594-7e29268eb6cc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dk6sz" Nov 01 20:31:16.529465 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.529314 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3635e179-1be9-4756-9594-7e29268eb6cc-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-dk6sz\" (UID: \"3635e179-1be9-4756-9594-7e29268eb6cc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dk6sz" Nov 01 20:31:16.529518 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.529472 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3635e179-1be9-4756-9594-7e29268eb6cc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-dk6sz\" (UID: \"3635e179-1be9-4756-9594-7e29268eb6cc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dk6sz" Nov 01 20:31:16.529576 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.529553 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlp9k\" (UniqueName: \"kubernetes.io/projected/3635e179-1be9-4756-9594-7e29268eb6cc-kube-api-access-jlp9k\") pod \"openshift-state-metrics-7bd7489957-dk6sz\" (UID: \"3635e179-1be9-4756-9594-7e29268eb6cc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dk6sz" Nov 01 20:31:16.554189 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.554132 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 01 20:31:16.554189 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.554156 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 01 20:31:16.554189 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.554169 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 01 20:31:16.554488 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.554245 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 01 20:31:16.554579 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.554561 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-59dtr\"" Nov 01 20:31:16.554718 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.554705 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 01 20:31:16.555785 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.555763 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-dk6sz"] Nov 01 20:31:16.604414 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.604371 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:16.606439 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.606410 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-p6znz"] Nov 01 20:31:16.608514 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.608492 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-p6znz" Nov 01 20:31:16.630276 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.630249 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlp9k\" (UniqueName: \"kubernetes.io/projected/3635e179-1be9-4756-9594-7e29268eb6cc-kube-api-access-jlp9k\") pod \"openshift-state-metrics-7bd7489957-dk6sz\" (UID: \"3635e179-1be9-4756-9594-7e29268eb6cc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dk6sz" Nov 01 20:31:16.630276 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.630286 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3635e179-1be9-4756-9594-7e29268eb6cc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-dk6sz\" (UID: \"3635e179-1be9-4756-9594-7e29268eb6cc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dk6sz" Nov 01 20:31:16.630499 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.630323 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3635e179-1be9-4756-9594-7e29268eb6cc-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-dk6sz\" (UID: \"3635e179-1be9-4756-9594-7e29268eb6cc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dk6sz" Nov 01 20:31:16.630499 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.630368 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3635e179-1be9-4756-9594-7e29268eb6cc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-dk6sz\" (UID: \"3635e179-1be9-4756-9594-7e29268eb6cc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dk6sz" Nov 01 20:31:16.631224 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.631195 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3635e179-1be9-4756-9594-7e29268eb6cc-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-dk6sz\" (UID: \"3635e179-1be9-4756-9594-7e29268eb6cc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dk6sz" Nov 01 20:31:16.633058 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.633037 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3635e179-1be9-4756-9594-7e29268eb6cc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-dk6sz\" (UID: \"3635e179-1be9-4756-9594-7e29268eb6cc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dk6sz" Nov 01 20:31:16.633404 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.633377 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3635e179-1be9-4756-9594-7e29268eb6cc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-dk6sz\" (UID: \"3635e179-1be9-4756-9594-7e29268eb6cc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dk6sz" Nov 01 20:31:16.690926 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.690891 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlp9k\" (UniqueName: \"kubernetes.io/projected/3635e179-1be9-4756-9594-7e29268eb6cc-kube-api-access-jlp9k\") pod \"openshift-state-metrics-7bd7489957-dk6sz\" (UID: \"3635e179-1be9-4756-9594-7e29268eb6cc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dk6sz" Nov 01 20:31:16.703946 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.703877 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 01 20:31:16.704178 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.704161 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rcjlq\"" Nov 01 20:31:16.704285 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.704272 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 01 20:31:16.706610 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.706574 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 01 20:31:16.731147 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.731104 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d858ea89-1f2a-4b06-b72a-2b43733a253d-sys\") pod \"node-exporter-p6znz\" (UID: \"d858ea89-1f2a-4b06-b72a-2b43733a253d\") " pod="openshift-monitoring/node-exporter-p6znz" Nov 01 20:31:16.731147 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.731146 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d858ea89-1f2a-4b06-b72a-2b43733a253d-metrics-client-ca\") pod \"node-exporter-p6znz\" (UID: \"d858ea89-1f2a-4b06-b72a-2b43733a253d\") " pod="openshift-monitoring/node-exporter-p6znz" Nov 01 20:31:16.731437 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.731179 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d858ea89-1f2a-4b06-b72a-2b43733a253d-node-exporter-wtmp\") pod \"node-exporter-p6znz\" (UID: \"d858ea89-1f2a-4b06-b72a-2b43733a253d\") " pod="openshift-monitoring/node-exporter-p6znz" Nov 01 20:31:16.731437 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.731200 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d858ea89-1f2a-4b06-b72a-2b43733a253d-node-exporter-tls\") pod \"node-exporter-p6znz\" (UID: \"d858ea89-1f2a-4b06-b72a-2b43733a253d\") " pod="openshift-monitoring/node-exporter-p6znz" Nov 01 20:31:16.731437 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.731223 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d858ea89-1f2a-4b06-b72a-2b43733a253d-node-exporter-textfile\") pod \"node-exporter-p6znz\" (UID: \"d858ea89-1f2a-4b06-b72a-2b43733a253d\") " pod="openshift-monitoring/node-exporter-p6znz" Nov 01 20:31:16.731437 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.731248 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d858ea89-1f2a-4b06-b72a-2b43733a253d-root\") pod \"node-exporter-p6znz\" (UID: \"d858ea89-1f2a-4b06-b72a-2b43733a253d\") " pod="openshift-monitoring/node-exporter-p6znz" Nov 01 20:31:16.731437 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.731266 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2lpt\" (UniqueName: \"kubernetes.io/projected/d858ea89-1f2a-4b06-b72a-2b43733a253d-kube-api-access-j2lpt\") pod \"node-exporter-p6znz\" (UID: \"d858ea89-1f2a-4b06-b72a-2b43733a253d\") " pod="openshift-monitoring/node-exporter-p6znz" Nov 01 20:31:16.731437 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.731288 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d858ea89-1f2a-4b06-b72a-2b43733a253d-node-exporter-accelerators-collector-config\") pod \"node-exporter-p6znz\" (UID: \"d858ea89-1f2a-4b06-b72a-2b43733a253d\") " pod="openshift-monitoring/node-exporter-p6znz" Nov 01 20:31:16.731437 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.731328 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d858ea89-1f2a-4b06-b72a-2b43733a253d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p6znz\" (UID: \"d858ea89-1f2a-4b06-b72a-2b43733a253d\") " pod="openshift-monitoring/node-exporter-p6znz" Nov 01 20:31:16.761454 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.761421 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dk6sz" Nov 01 20:31:16.808472 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.808422 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:16.836753 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.834407 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d858ea89-1f2a-4b06-b72a-2b43733a253d-root\") pod \"node-exporter-p6znz\" (UID: \"d858ea89-1f2a-4b06-b72a-2b43733a253d\") " pod="openshift-monitoring/node-exporter-p6znz" Nov 01 20:31:16.836753 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.834462 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2lpt\" (UniqueName: \"kubernetes.io/projected/d858ea89-1f2a-4b06-b72a-2b43733a253d-kube-api-access-j2lpt\") pod \"node-exporter-p6znz\" (UID: \"d858ea89-1f2a-4b06-b72a-2b43733a253d\") " pod="openshift-monitoring/node-exporter-p6znz" Nov 01 20:31:16.836753 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.834499 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d858ea89-1f2a-4b06-b72a-2b43733a253d-node-exporter-accelerators-collector-config\") pod \"node-exporter-p6znz\" (UID: \"d858ea89-1f2a-4b06-b72a-2b43733a253d\") " pod="openshift-monitoring/node-exporter-p6znz" Nov 01 20:31:16.836753 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.834540 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d858ea89-1f2a-4b06-b72a-2b43733a253d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p6znz\" (UID: \"d858ea89-1f2a-4b06-b72a-2b43733a253d\") " pod="openshift-monitoring/node-exporter-p6znz" Nov 01 20:31:16.836753 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.834585 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d858ea89-1f2a-4b06-b72a-2b43733a253d-sys\") pod \"node-exporter-p6znz\" (UID: \"d858ea89-1f2a-4b06-b72a-2b43733a253d\") " pod="openshift-monitoring/node-exporter-p6znz" Nov 01 20:31:16.836753 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.834623 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d858ea89-1f2a-4b06-b72a-2b43733a253d-metrics-client-ca\") pod \"node-exporter-p6znz\" (UID: \"d858ea89-1f2a-4b06-b72a-2b43733a253d\") " pod="openshift-monitoring/node-exporter-p6znz" Nov 01 20:31:16.836753 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.834655 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d858ea89-1f2a-4b06-b72a-2b43733a253d-node-exporter-wtmp\") pod \"node-exporter-p6znz\" (UID: \"d858ea89-1f2a-4b06-b72a-2b43733a253d\") " pod="openshift-monitoring/node-exporter-p6znz" Nov 01 20:31:16.836753 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.834685 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d858ea89-1f2a-4b06-b72a-2b43733a253d-node-exporter-tls\") pod \"node-exporter-p6znz\" (UID: \"d858ea89-1f2a-4b06-b72a-2b43733a253d\") " pod="openshift-monitoring/node-exporter-p6znz" Nov 01 20:31:16.836753 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.834713 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d858ea89-1f2a-4b06-b72a-2b43733a253d-node-exporter-textfile\") pod \"node-exporter-p6znz\" (UID: \"d858ea89-1f2a-4b06-b72a-2b43733a253d\") " pod="openshift-monitoring/node-exporter-p6znz" Nov 01 20:31:16.836753 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.835112 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d858ea89-1f2a-4b06-b72a-2b43733a253d-node-exporter-textfile\") pod \"node-exporter-p6znz\" (UID: \"d858ea89-1f2a-4b06-b72a-2b43733a253d\") " pod="openshift-monitoring/node-exporter-p6znz" Nov 01 20:31:16.836753 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.835200 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d858ea89-1f2a-4b06-b72a-2b43733a253d-root\") pod \"node-exporter-p6znz\" (UID: \"d858ea89-1f2a-4b06-b72a-2b43733a253d\") " pod="openshift-monitoring/node-exporter-p6znz" Nov 01 20:31:16.836753 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.836098 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d858ea89-1f2a-4b06-b72a-2b43733a253d-node-exporter-accelerators-collector-config\") pod \"node-exporter-p6znz\" (UID: \"d858ea89-1f2a-4b06-b72a-2b43733a253d\") " pod="openshift-monitoring/node-exporter-p6znz" Nov 01 20:31:16.839034 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.838502 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d858ea89-1f2a-4b06-b72a-2b43733a253d-metrics-client-ca\") pod \"node-exporter-p6znz\" (UID: \"d858ea89-1f2a-4b06-b72a-2b43733a253d\") " pod="openshift-monitoring/node-exporter-p6znz" Nov 01 20:31:16.839034 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.838577 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d858ea89-1f2a-4b06-b72a-2b43733a253d-sys\") pod \"node-exporter-p6znz\" (UID: \"d858ea89-1f2a-4b06-b72a-2b43733a253d\") " pod="openshift-monitoring/node-exporter-p6znz" Nov 01 20:31:16.839034 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.838709 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d858ea89-1f2a-4b06-b72a-2b43733a253d-node-exporter-wtmp\") pod \"node-exporter-p6znz\" (UID: \"d858ea89-1f2a-4b06-b72a-2b43733a253d\") " pod="openshift-monitoring/node-exporter-p6znz" Nov 01 20:31:16.841983 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.841937 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d858ea89-1f2a-4b06-b72a-2b43733a253d-node-exporter-tls\") pod \"node-exporter-p6znz\" (UID: \"d858ea89-1f2a-4b06-b72a-2b43733a253d\") " pod="openshift-monitoring/node-exporter-p6znz" Nov 01 20:31:16.850117 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.850056 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d858ea89-1f2a-4b06-b72a-2b43733a253d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p6znz\" (UID: \"d858ea89-1f2a-4b06-b72a-2b43733a253d\") " pod="openshift-monitoring/node-exporter-p6znz" Nov 01 20:31:16.895629 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.895592 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2lpt\" (UniqueName: \"kubernetes.io/projected/d858ea89-1f2a-4b06-b72a-2b43733a253d-kube-api-access-j2lpt\") pod \"node-exporter-p6znz\" (UID: \"d858ea89-1f2a-4b06-b72a-2b43733a253d\") " pod="openshift-monitoring/node-exporter-p6znz" Nov 01 20:31:16.922136 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.922100 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-p6znz" Nov 01 20:31:16.933081 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:31:16.933025 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd858ea89_1f2a_4b06_b72a_2b43733a253d.slice/crio-92de807320a3e1df05203053df2cffbe3fcb6f9f81086b36b13c242f3c88dd49 WatchSource:0}: Error finding container 92de807320a3e1df05203053df2cffbe3fcb6f9f81086b36b13c242f3c88dd49: Status 404 returned error can't find the container with id 92de807320a3e1df05203053df2cffbe3fcb6f9f81086b36b13c242f3c88dd49 Nov 01 20:31:16.994610 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:16.994562 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:17.192090 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:17.192064 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-dk6sz"] Nov 01 20:31:17.195162 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:31:17.195135 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3635e179_1be9_4756_9594_7e29268eb6cc.slice/crio-bfe4bd8aa131317dd268a2c3b7dd13dcf57ac468603c2a38dad0981d78bed893 WatchSource:0}: Error finding container bfe4bd8aa131317dd268a2c3b7dd13dcf57ac468603c2a38dad0981d78bed893: Status 404 returned error can't find the container with id bfe4bd8aa131317dd268a2c3b7dd13dcf57ac468603c2a38dad0981d78bed893 Nov 01 20:31:17.196130 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:17.196097 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:17.410778 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:17.410744 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:17.522499 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:17.522404 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dk6sz" event={"ID":"3635e179-1be9-4756-9594-7e29268eb6cc","Type":"ContainerStarted","Data":"98319a78de33d233641b026d233bbf37a7aeb298c69cad0ce52ca405ea73f4af"} Nov 01 20:31:17.522499 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:17.522474 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dk6sz" event={"ID":"3635e179-1be9-4756-9594-7e29268eb6cc","Type":"ContainerStarted","Data":"9c3600368f798cdb84286f951241958a3aa26a22bd229f102fc44e3ab1cd567d"} Nov 01 20:31:17.522718 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:17.522505 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dk6sz" event={"ID":"3635e179-1be9-4756-9594-7e29268eb6cc","Type":"ContainerStarted","Data":"bfe4bd8aa131317dd268a2c3b7dd13dcf57ac468603c2a38dad0981d78bed893"} Nov 01 20:31:17.523817 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:17.523776 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p6znz" event={"ID":"d858ea89-1f2a-4b06-b72a-2b43733a253d","Type":"ContainerStarted","Data":"92de807320a3e1df05203053df2cffbe3fcb6f9f81086b36b13c242f3c88dd49"} Nov 01 20:31:17.618197 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:17.618158 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:17.909525 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:17.909434 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:18.103651 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:18.103608 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:18.725596 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:18.725550 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:18.922763 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:18.922732 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:19.196324 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:19.196260 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6bc9c45967-46rtx"] Nov 01 20:31:19.199712 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:19.199694 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6bc9c45967-46rtx" Nov 01 20:31:19.253793 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:19.253768 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/635876cb-613a-4691-a980-255f7df1bcef-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6bc9c45967-46rtx\" (UID: \"635876cb-613a-4691-a980-255f7df1bcef\") " pod="openshift-monitoring/thanos-querier-6bc9c45967-46rtx" Nov 01 20:31:19.253891 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:19.253816 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/635876cb-613a-4691-a980-255f7df1bcef-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6bc9c45967-46rtx\" (UID: \"635876cb-613a-4691-a980-255f7df1bcef\") " pod="openshift-monitoring/thanos-querier-6bc9c45967-46rtx" Nov 01 20:31:19.253952 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:19.253889 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/635876cb-613a-4691-a980-255f7df1bcef-metrics-client-ca\") pod \"thanos-querier-6bc9c45967-46rtx\" (UID: \"635876cb-613a-4691-a980-255f7df1bcef\") " pod="openshift-monitoring/thanos-querier-6bc9c45967-46rtx" Nov 01 20:31:19.253952 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:19.253936 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ftxm\" (UniqueName: \"kubernetes.io/projected/635876cb-613a-4691-a980-255f7df1bcef-kube-api-access-8ftxm\") pod \"thanos-querier-6bc9c45967-46rtx\" (UID: \"635876cb-613a-4691-a980-255f7df1bcef\") " pod="openshift-monitoring/thanos-querier-6bc9c45967-46rtx" Nov 01 20:31:19.254106 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:19.253977 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/635876cb-613a-4691-a980-255f7df1bcef-tmp\") pod \"thanos-querier-6bc9c45967-46rtx\" (UID: \"635876cb-613a-4691-a980-255f7df1bcef\") " pod="openshift-monitoring/thanos-querier-6bc9c45967-46rtx" Nov 01 20:31:19.254106 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:19.254094 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/635876cb-613a-4691-a980-255f7df1bcef-secret-thanos-querier-tls\") pod \"thanos-querier-6bc9c45967-46rtx\" (UID: \"635876cb-613a-4691-a980-255f7df1bcef\") " pod="openshift-monitoring/thanos-querier-6bc9c45967-46rtx" Nov 01 20:31:19.254204 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:19.254125 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/635876cb-613a-4691-a980-255f7df1bcef-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6bc9c45967-46rtx\" (UID: \"635876cb-613a-4691-a980-255f7df1bcef\") " pod="openshift-monitoring/thanos-querier-6bc9c45967-46rtx" Nov 01 20:31:19.254204 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:19.254193 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/635876cb-613a-4691-a980-255f7df1bcef-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6bc9c45967-46rtx\" (UID: \"635876cb-613a-4691-a980-255f7df1bcef\") " pod="openshift-monitoring/thanos-querier-6bc9c45967-46rtx" Nov 01 20:31:19.254297 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:19.254214 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/635876cb-613a-4691-a980-255f7df1bcef-secret-grpc-tls\") pod \"thanos-querier-6bc9c45967-46rtx\" (UID: \"635876cb-613a-4691-a980-255f7df1bcef\") " pod="openshift-monitoring/thanos-querier-6bc9c45967-46rtx" Nov 01 20:31:19.305045 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:19.305017 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 01 20:31:19.305045 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:19.305014 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 01 20:31:19.305272 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:19.305057 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 01 20:31:19.305272 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:19.305062 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 01 20:31:19.305272 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:19.305117 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 01 20:31:19.305272 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:19.305206 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-ts8kw\"" Nov 01 20:31:19.305481 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:19.305278 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-5lah1os031phu\"" Nov 01 20:31:19.306098 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:19.306080 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6bc9c45967-46rtx"] Nov 01 20:31:19.355486 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:19.355455 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/635876cb-613a-4691-a980-255f7df1bcef-tmp\") pod \"thanos-querier-6bc9c45967-46rtx\" (UID: \"635876cb-613a-4691-a980-255f7df1bcef\") " pod="openshift-monitoring/thanos-querier-6bc9c45967-46rtx" Nov 01 20:31:19.355652 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:19.355500 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/635876cb-613a-4691-a980-255f7df1bcef-secret-thanos-querier-tls\") pod \"thanos-querier-6bc9c45967-46rtx\" (UID: \"635876cb-613a-4691-a980-255f7df1bcef\") " pod="openshift-monitoring/thanos-querier-6bc9c45967-46rtx" Nov 01 20:31:19.355652 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:19.355518 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/635876cb-613a-4691-a980-255f7df1bcef-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6bc9c45967-46rtx\" (UID: \"635876cb-613a-4691-a980-255f7df1bcef\") " pod="openshift-monitoring/thanos-querier-6bc9c45967-46rtx" Nov 01 20:31:19.355652 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:19.355535 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/635876cb-613a-4691-a980-255f7df1bcef-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6bc9c45967-46rtx\" (UID: \"635876cb-613a-4691-a980-255f7df1bcef\") " pod="openshift-monitoring/thanos-querier-6bc9c45967-46rtx" Nov 01 20:31:19.355652 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:19.355551 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/635876cb-613a-4691-a980-255f7df1bcef-secret-grpc-tls\") pod \"thanos-querier-6bc9c45967-46rtx\" (UID: \"635876cb-613a-4691-a980-255f7df1bcef\") " pod="openshift-monitoring/thanos-querier-6bc9c45967-46rtx" Nov 01 20:31:19.355652 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:19.355589 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/635876cb-613a-4691-a980-255f7df1bcef-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6bc9c45967-46rtx\" (UID: \"635876cb-613a-4691-a980-255f7df1bcef\") " pod="openshift-monitoring/thanos-querier-6bc9c45967-46rtx" Nov 01 20:31:19.355652 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:19.355619 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/635876cb-613a-4691-a980-255f7df1bcef-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6bc9c45967-46rtx\" (UID: \"635876cb-613a-4691-a980-255f7df1bcef\") " pod="openshift-monitoring/thanos-querier-6bc9c45967-46rtx" Nov 01 20:31:19.355652 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:19.355647 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/635876cb-613a-4691-a980-255f7df1bcef-metrics-client-ca\") pod \"thanos-querier-6bc9c45967-46rtx\" (UID: \"635876cb-613a-4691-a980-255f7df1bcef\") " pod="openshift-monitoring/thanos-querier-6bc9c45967-46rtx" Nov 01 20:31:19.356162 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:19.355674 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ftxm\" (UniqueName: \"kubernetes.io/projected/635876cb-613a-4691-a980-255f7df1bcef-kube-api-access-8ftxm\") pod \"thanos-querier-6bc9c45967-46rtx\" (UID: \"635876cb-613a-4691-a980-255f7df1bcef\") " pod="openshift-monitoring/thanos-querier-6bc9c45967-46rtx" Nov 01 20:31:19.356162 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:19.356009 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/635876cb-613a-4691-a980-255f7df1bcef-tmp\") pod \"thanos-querier-6bc9c45967-46rtx\" (UID: \"635876cb-613a-4691-a980-255f7df1bcef\") " pod="openshift-monitoring/thanos-querier-6bc9c45967-46rtx" Nov 01 20:31:19.357472 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:19.357438 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/635876cb-613a-4691-a980-255f7df1bcef-metrics-client-ca\") pod \"thanos-querier-6bc9c45967-46rtx\" (UID: \"635876cb-613a-4691-a980-255f7df1bcef\") " pod="openshift-monitoring/thanos-querier-6bc9c45967-46rtx" Nov 01 20:31:19.359501 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:19.359368 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/635876cb-613a-4691-a980-255f7df1bcef-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6bc9c45967-46rtx\" (UID: \"635876cb-613a-4691-a980-255f7df1bcef\") " pod="openshift-monitoring/thanos-querier-6bc9c45967-46rtx" Nov 01 20:31:19.361033 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:19.360811 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/635876cb-613a-4691-a980-255f7df1bcef-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6bc9c45967-46rtx\" (UID: \"635876cb-613a-4691-a980-255f7df1bcef\") " pod="openshift-monitoring/thanos-querier-6bc9c45967-46rtx" Nov 01 20:31:19.361033 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:19.360891 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/635876cb-613a-4691-a980-255f7df1bcef-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6bc9c45967-46rtx\" (UID: \"635876cb-613a-4691-a980-255f7df1bcef\") " pod="openshift-monitoring/thanos-querier-6bc9c45967-46rtx" Nov 01 20:31:19.362066 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:19.361313 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/635876cb-613a-4691-a980-255f7df1bcef-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6bc9c45967-46rtx\" (UID: \"635876cb-613a-4691-a980-255f7df1bcef\") " pod="openshift-monitoring/thanos-querier-6bc9c45967-46rtx" Nov 01 20:31:19.362066 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:19.361393 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/635876cb-613a-4691-a980-255f7df1bcef-secret-thanos-querier-tls\") pod \"thanos-querier-6bc9c45967-46rtx\" (UID: \"635876cb-613a-4691-a980-255f7df1bcef\") " pod="openshift-monitoring/thanos-querier-6bc9c45967-46rtx" Nov 01 20:31:19.362066 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:19.361692 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/635876cb-613a-4691-a980-255f7df1bcef-secret-grpc-tls\") pod \"thanos-querier-6bc9c45967-46rtx\" (UID: \"635876cb-613a-4691-a980-255f7df1bcef\") " pod="openshift-monitoring/thanos-querier-6bc9c45967-46rtx" Nov 01 20:31:19.418165 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:19.418135 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ftxm\" (UniqueName: \"kubernetes.io/projected/635876cb-613a-4691-a980-255f7df1bcef-kube-api-access-8ftxm\") pod \"thanos-querier-6bc9c45967-46rtx\" (UID: \"635876cb-613a-4691-a980-255f7df1bcef\") " pod="openshift-monitoring/thanos-querier-6bc9c45967-46rtx" Nov 01 20:31:19.522737 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:19.522709 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6bc9c45967-46rtx" Nov 01 20:31:19.532497 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:19.532463 4484 generic.go:356] "Generic (PLEG): container finished" podID="d858ea89-1f2a-4b06-b72a-2b43733a253d" containerID="6315a62d9636819ce1331c76926d1f74b4f22e04575ef32a2206adaaad820fdc" exitCode=0 Nov 01 20:31:19.532655 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:19.532569 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p6znz" event={"ID":"d858ea89-1f2a-4b06-b72a-2b43733a253d","Type":"ContainerDied","Data":"6315a62d9636819ce1331c76926d1f74b4f22e04575ef32a2206adaaad820fdc"} Nov 01 20:31:19.535431 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:19.535403 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-ngxhs" event={"ID":"65896401-c413-42f3-af9f-4fcf04cacadb","Type":"ContainerStarted","Data":"cd43549e1c946968b5379823ba95990d4375dc94f996309d9c8083e7e87003f1"} Nov 01 20:31:19.537086 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:19.537060 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-ngxhs" Nov 01 20:31:19.539978 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:19.539953 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dk6sz" event={"ID":"3635e179-1be9-4756-9594-7e29268eb6cc","Type":"ContainerStarted","Data":"cceff124d45cc367adba0dfc80b6ec074b87ba6714592d4cdb1d156364a74add"} Nov 01 20:31:19.551314 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:19.551245 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-ngxhs" Nov 01 20:31:19.782245 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:19.782144 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-ngxhs" podStartSLOduration=2.253478123 podStartE2EDuration="18.782127146s" totalImagesPullingTime="16.528649023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:31:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:31:19.781707815 +0000 UTC m=+30.559347950" watchObservedRunningTime="2026-11-01 20:31:19.782127146 +0000 UTC m=+30.559767281" Nov 01 20:31:19.901608 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:19.901546 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dk6sz" podStartSLOduration=2.338944901 podStartE2EDuration="3.90153023s" totalImagesPullingTime="1.562585329s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:31:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:31:19.900719325 +0000 UTC m=+30.678359460" watchObservedRunningTime="2026-11-01 20:31:19.90153023 +0000 UTC m=+30.679170366" Nov 01 20:31:19.973827 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:19.973682 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6bc9c45967-46rtx"] Nov 01 20:31:19.976890 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:31:19.976860 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod635876cb_613a_4691_a980_255f7df1bcef.slice/crio-8b9ea22ef61e4c7da2e50dcb10cfff1766cd5f27647357af00f99d9c9a14bfd7 WatchSource:0}: Error finding container 8b9ea22ef61e4c7da2e50dcb10cfff1766cd5f27647357af00f99d9c9a14bfd7: Status 404 returned error can't find the container with id 8b9ea22ef61e4c7da2e50dcb10cfff1766cd5f27647357af00f99d9c9a14bfd7 Nov 01 20:31:20.122907 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:20.122822 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:20.339555 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:20.339520 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:20.546849 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:20.546811 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p6znz" event={"ID":"d858ea89-1f2a-4b06-b72a-2b43733a253d","Type":"ContainerStarted","Data":"9ccc978eef66db4f5cc63e5c90fff3531f9367d19f4e290515af248623c751ae"} Nov 01 20:31:20.547067 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:20.546884 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p6znz" event={"ID":"d858ea89-1f2a-4b06-b72a-2b43733a253d","Type":"ContainerStarted","Data":"0aa15b65b010217fad7357318089f4167da0e02002287f5681e557dcabdd2bae"} Nov 01 20:31:20.554915 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:20.549680 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bc9c45967-46rtx" event={"ID":"635876cb-613a-4691-a980-255f7df1bcef","Type":"ContainerStarted","Data":"8b9ea22ef61e4c7da2e50dcb10cfff1766cd5f27647357af00f99d9c9a14bfd7"} Nov 01 20:31:20.676280 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:20.676224 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-p6znz" podStartSLOduration=3.022968829 podStartE2EDuration="4.676210158s" totalImagesPullingTime="1.573833814s" totalInitContainerRuntime="79.407515ms" isStatefulPod=true podCreationTimestamp="2026-11-01 20:31:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:31:20.672852813 +0000 UTC m=+31.450492959" watchObservedRunningTime="2026-11-01 20:31:20.676210158 +0000 UTC m=+31.453850295" Nov 01 20:31:20.830315 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:20.830230 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:21.022755 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:21.022718 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:21.203736 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:21.203624 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:21.236723 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:21.236689 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-hk84s"] Nov 01 20:31:21.256806 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:21.256773 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hk84s" Nov 01 20:31:21.298295 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:21.298259 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-hk84s"] Nov 01 20:31:21.315103 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:21.314978 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-mf6z9\"" Nov 01 20:31:21.315103 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:21.315074 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 01 20:31:21.373206 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:21.373156 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx8jb\" (UniqueName: \"kubernetes.io/projected/c693b7c8-323f-4c08-b95f-bc2dca1ba6e3-kube-api-access-vx8jb\") pod \"monitoring-plugin-77f4f76594-hk84s\" (UID: \"c693b7c8-323f-4c08-b95f-bc2dca1ba6e3\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hk84s" Nov 01 20:31:21.373206 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:21.373278 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c693b7c8-323f-4c08-b95f-bc2dca1ba6e3-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-hk84s\" (UID: \"c693b7c8-323f-4c08-b95f-bc2dca1ba6e3\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hk84s" Nov 01 20:31:21.389757 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:21.389724 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:21.411352 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:21.411321 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ghh5z" Nov 01 20:31:21.417801 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:21.417769 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-84f999b6b9-g8p4q"] Nov 01 20:31:21.443767 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:21.443733 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-84f999b6b9-g8p4q" Nov 01 20:31:21.474117 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:21.474034 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx8jb\" (UniqueName: \"kubernetes.io/projected/c693b7c8-323f-4c08-b95f-bc2dca1ba6e3-kube-api-access-vx8jb\") pod \"monitoring-plugin-77f4f76594-hk84s\" (UID: \"c693b7c8-323f-4c08-b95f-bc2dca1ba6e3\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hk84s" Nov 01 20:31:21.474117 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:21.474102 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c693b7c8-323f-4c08-b95f-bc2dca1ba6e3-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-hk84s\" (UID: \"c693b7c8-323f-4c08-b95f-bc2dca1ba6e3\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hk84s" Nov 01 20:31:21.477329 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:21.477297 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c693b7c8-323f-4c08-b95f-bc2dca1ba6e3-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-hk84s\" (UID: \"c693b7c8-323f-4c08-b95f-bc2dca1ba6e3\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hk84s" Nov 01 20:31:21.481350 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:21.481205 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-84f999b6b9-g8p4q"] Nov 01 20:31:21.497749 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:21.497704 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 01 20:31:21.497932 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:21.497793 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 01 20:31:21.513859 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:21.513819 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 01 20:31:21.514098 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:21.513975 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-2d8g4\"" Nov 01 20:31:21.514098 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:21.514046 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 01 20:31:21.514255 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:21.514105 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-bb33anul2a0bv\"" Nov 01 20:31:21.537620 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:21.537555 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx8jb\" (UniqueName: \"kubernetes.io/projected/c693b7c8-323f-4c08-b95f-bc2dca1ba6e3-kube-api-access-vx8jb\") pod \"monitoring-plugin-77f4f76594-hk84s\" (UID: \"c693b7c8-323f-4c08-b95f-bc2dca1ba6e3\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hk84s" Nov 01 20:31:21.574892 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:21.574856 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8995aa4c-ac0d-4f43-aa03-bebced1e08e0-audit-log\") pod \"metrics-server-84f999b6b9-g8p4q\" (UID: \"8995aa4c-ac0d-4f43-aa03-bebced1e08e0\") " pod="openshift-monitoring/metrics-server-84f999b6b9-g8p4q" Nov 01 20:31:21.575078 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:21.574925 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7826m\" (UniqueName: \"kubernetes.io/projected/8995aa4c-ac0d-4f43-aa03-bebced1e08e0-kube-api-access-7826m\") pod \"metrics-server-84f999b6b9-g8p4q\" (UID: \"8995aa4c-ac0d-4f43-aa03-bebced1e08e0\") " pod="openshift-monitoring/metrics-server-84f999b6b9-g8p4q" Nov 01 20:31:21.575154 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:21.575068 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8995aa4c-ac0d-4f43-aa03-bebced1e08e0-secret-metrics-server-client-certs\") pod \"metrics-server-84f999b6b9-g8p4q\" (UID: \"8995aa4c-ac0d-4f43-aa03-bebced1e08e0\") " pod="openshift-monitoring/metrics-server-84f999b6b9-g8p4q" Nov 01 20:31:21.575154 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:21.575111 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8995aa4c-ac0d-4f43-aa03-bebced1e08e0-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-84f999b6b9-g8p4q\" (UID: \"8995aa4c-ac0d-4f43-aa03-bebced1e08e0\") " pod="openshift-monitoring/metrics-server-84f999b6b9-g8p4q" Nov 01 20:31:21.575154 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:21.575145 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8995aa4c-ac0d-4f43-aa03-bebced1e08e0-metrics-server-audit-profiles\") pod \"metrics-server-84f999b6b9-g8p4q\" (UID: \"8995aa4c-ac0d-4f43-aa03-bebced1e08e0\") " pod="openshift-monitoring/metrics-server-84f999b6b9-g8p4q" Nov 01 20:31:21.575315 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:21.575183 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8995aa4c-ac0d-4f43-aa03-bebced1e08e0-client-ca-bundle\") pod \"metrics-server-84f999b6b9-g8p4q\" (UID: \"8995aa4c-ac0d-4f43-aa03-bebced1e08e0\") " pod="openshift-monitoring/metrics-server-84f999b6b9-g8p4q" Nov 01 20:31:21.575315 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:21.575232 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8995aa4c-ac0d-4f43-aa03-bebced1e08e0-secret-metrics-server-tls\") pod \"metrics-server-84f999b6b9-g8p4q\" (UID: \"8995aa4c-ac0d-4f43-aa03-bebced1e08e0\") " pod="openshift-monitoring/metrics-server-84f999b6b9-g8p4q" Nov 01 20:31:21.589198 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:21.589163 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hk84s" Nov 01 20:31:21.676544 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:21.676426 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7826m\" (UniqueName: \"kubernetes.io/projected/8995aa4c-ac0d-4f43-aa03-bebced1e08e0-kube-api-access-7826m\") pod \"metrics-server-84f999b6b9-g8p4q\" (UID: \"8995aa4c-ac0d-4f43-aa03-bebced1e08e0\") " pod="openshift-monitoring/metrics-server-84f999b6b9-g8p4q" Nov 01 20:31:21.676544 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:21.676476 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8995aa4c-ac0d-4f43-aa03-bebced1e08e0-secret-metrics-server-client-certs\") pod \"metrics-server-84f999b6b9-g8p4q\" (UID: \"8995aa4c-ac0d-4f43-aa03-bebced1e08e0\") " pod="openshift-monitoring/metrics-server-84f999b6b9-g8p4q" Nov 01 20:31:21.676544 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:21.676508 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8995aa4c-ac0d-4f43-aa03-bebced1e08e0-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-84f999b6b9-g8p4q\" (UID: \"8995aa4c-ac0d-4f43-aa03-bebced1e08e0\") " pod="openshift-monitoring/metrics-server-84f999b6b9-g8p4q" Nov 01 20:31:21.676544 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:21.676535 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8995aa4c-ac0d-4f43-aa03-bebced1e08e0-metrics-server-audit-profiles\") pod \"metrics-server-84f999b6b9-g8p4q\" (UID: \"8995aa4c-ac0d-4f43-aa03-bebced1e08e0\") " pod="openshift-monitoring/metrics-server-84f999b6b9-g8p4q" Nov 01 20:31:21.676772 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:21.676600 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8995aa4c-ac0d-4f43-aa03-bebced1e08e0-client-ca-bundle\") pod \"metrics-server-84f999b6b9-g8p4q\" (UID: \"8995aa4c-ac0d-4f43-aa03-bebced1e08e0\") " pod="openshift-monitoring/metrics-server-84f999b6b9-g8p4q" Nov 01 20:31:21.676772 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:21.676669 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8995aa4c-ac0d-4f43-aa03-bebced1e08e0-secret-metrics-server-tls\") pod \"metrics-server-84f999b6b9-g8p4q\" (UID: \"8995aa4c-ac0d-4f43-aa03-bebced1e08e0\") " pod="openshift-monitoring/metrics-server-84f999b6b9-g8p4q" Nov 01 20:31:21.676772 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:21.676727 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8995aa4c-ac0d-4f43-aa03-bebced1e08e0-audit-log\") pod \"metrics-server-84f999b6b9-g8p4q\" (UID: \"8995aa4c-ac0d-4f43-aa03-bebced1e08e0\") " pod="openshift-monitoring/metrics-server-84f999b6b9-g8p4q" Nov 01 20:31:21.679981 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:21.679957 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8995aa4c-ac0d-4f43-aa03-bebced1e08e0-metrics-server-audit-profiles\") pod \"metrics-server-84f999b6b9-g8p4q\" (UID: \"8995aa4c-ac0d-4f43-aa03-bebced1e08e0\") " pod="openshift-monitoring/metrics-server-84f999b6b9-g8p4q" Nov 01 20:31:21.680461 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:21.680413 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8995aa4c-ac0d-4f43-aa03-bebced1e08e0-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-84f999b6b9-g8p4q\" (UID: \"8995aa4c-ac0d-4f43-aa03-bebced1e08e0\") " pod="openshift-monitoring/metrics-server-84f999b6b9-g8p4q" Nov 01 20:31:21.680578 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:21.680555 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8995aa4c-ac0d-4f43-aa03-bebced1e08e0-audit-log\") pod \"metrics-server-84f999b6b9-g8p4q\" (UID: \"8995aa4c-ac0d-4f43-aa03-bebced1e08e0\") " pod="openshift-monitoring/metrics-server-84f999b6b9-g8p4q" Nov 01 20:31:21.683326 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:21.683273 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8995aa4c-ac0d-4f43-aa03-bebced1e08e0-secret-metrics-server-tls\") pod \"metrics-server-84f999b6b9-g8p4q\" (UID: \"8995aa4c-ac0d-4f43-aa03-bebced1e08e0\") " pod="openshift-monitoring/metrics-server-84f999b6b9-g8p4q" Nov 01 20:31:21.684746 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:21.684719 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8995aa4c-ac0d-4f43-aa03-bebced1e08e0-secret-metrics-server-client-certs\") pod \"metrics-server-84f999b6b9-g8p4q\" (UID: \"8995aa4c-ac0d-4f43-aa03-bebced1e08e0\") " pod="openshift-monitoring/metrics-server-84f999b6b9-g8p4q" Nov 01 20:31:21.686335 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:21.686115 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8995aa4c-ac0d-4f43-aa03-bebced1e08e0-client-ca-bundle\") pod \"metrics-server-84f999b6b9-g8p4q\" (UID: \"8995aa4c-ac0d-4f43-aa03-bebced1e08e0\") " pod="openshift-monitoring/metrics-server-84f999b6b9-g8p4q" Nov 01 20:31:21.737848 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:21.737765 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7826m\" (UniqueName: \"kubernetes.io/projected/8995aa4c-ac0d-4f43-aa03-bebced1e08e0-kube-api-access-7826m\") pod \"metrics-server-84f999b6b9-g8p4q\" (UID: \"8995aa4c-ac0d-4f43-aa03-bebced1e08e0\") " pod="openshift-monitoring/metrics-server-84f999b6b9-g8p4q" Nov 01 20:31:21.784467 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:21.784429 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-84f999b6b9-g8p4q" Nov 01 20:31:22.005510 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:22.005327 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-hk84s"] Nov 01 20:31:22.009523 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:31:22.009490 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc693b7c8_323f_4c08_b95f_bc2dca1ba6e3.slice/crio-e69783c2688db4a1841a8d083583c6b37bc605290978ce22aa59ff75e93f128e WatchSource:0}: Error finding container e69783c2688db4a1841a8d083583c6b37bc605290978ce22aa59ff75e93f128e: Status 404 returned error can't find the container with id e69783c2688db4a1841a8d083583c6b37bc605290978ce22aa59ff75e93f128e Nov 01 20:31:22.201275 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:22.201237 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-84f999b6b9-g8p4q"] Nov 01 20:31:22.206643 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:31:22.206609 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8995aa4c_ac0d_4f43_aa03_bebced1e08e0.slice/crio-e3c15d488f984ca4d0366e70f8f00d6702586d5ffc71863716744ea79a8ded42 WatchSource:0}: Error finding container e3c15d488f984ca4d0366e70f8f00d6702586d5ffc71863716744ea79a8ded42: Status 404 returned error can't find the container with id e3c15d488f984ca4d0366e70f8f00d6702586d5ffc71863716744ea79a8ded42 Nov 01 20:31:22.256340 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:22.256265 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-dfb96dff-4xpmq"] Nov 01 20:31:22.279073 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:22.279042 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-dfb96dff-4xpmq" Nov 01 20:31:22.329070 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:22.328970 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-dfb96dff-4xpmq"] Nov 01 20:31:22.333637 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:22.333606 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 01 20:31:22.333781 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:22.333691 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 01 20:31:22.333781 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:22.333774 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 01 20:31:22.376663 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:22.376634 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 01 20:31:22.376663 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:22.376637 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-kph4h\"" Nov 01 20:31:22.376878 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:22.376671 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 01 20:31:22.386051 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:22.386017 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0e381e93-ad4a-4c65-b7fe-1482e80d87b4-secret-telemeter-client\") pod \"telemeter-client-dfb96dff-4xpmq\" (UID: \"0e381e93-ad4a-4c65-b7fe-1482e80d87b4\") " pod="openshift-monitoring/telemeter-client-dfb96dff-4xpmq" Nov 01 20:31:22.386384 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:22.386073 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e381e93-ad4a-4c65-b7fe-1482e80d87b4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-dfb96dff-4xpmq\" (UID: \"0e381e93-ad4a-4c65-b7fe-1482e80d87b4\") " pod="openshift-monitoring/telemeter-client-dfb96dff-4xpmq" Nov 01 20:31:22.386384 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:22.386159 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0e381e93-ad4a-4c65-b7fe-1482e80d87b4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-dfb96dff-4xpmq\" (UID: \"0e381e93-ad4a-4c65-b7fe-1482e80d87b4\") " pod="openshift-monitoring/telemeter-client-dfb96dff-4xpmq" Nov 01 20:31:22.386384 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:22.386212 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e381e93-ad4a-4c65-b7fe-1482e80d87b4-serving-certs-ca-bundle\") pod \"telemeter-client-dfb96dff-4xpmq\" (UID: \"0e381e93-ad4a-4c65-b7fe-1482e80d87b4\") " pod="openshift-monitoring/telemeter-client-dfb96dff-4xpmq" Nov 01 20:31:22.386384 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:22.386287 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7xcx\" (UniqueName: \"kubernetes.io/projected/0e381e93-ad4a-4c65-b7fe-1482e80d87b4-kube-api-access-m7xcx\") pod \"telemeter-client-dfb96dff-4xpmq\" (UID: \"0e381e93-ad4a-4c65-b7fe-1482e80d87b4\") " pod="openshift-monitoring/telemeter-client-dfb96dff-4xpmq" Nov 01 20:31:22.386384 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:22.386381 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0e381e93-ad4a-4c65-b7fe-1482e80d87b4-telemeter-client-tls\") pod \"telemeter-client-dfb96dff-4xpmq\" (UID: \"0e381e93-ad4a-4c65-b7fe-1482e80d87b4\") " pod="openshift-monitoring/telemeter-client-dfb96dff-4xpmq" Nov 01 20:31:22.386639 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:22.386428 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e381e93-ad4a-4c65-b7fe-1482e80d87b4-metrics-client-ca\") pod \"telemeter-client-dfb96dff-4xpmq\" (UID: \"0e381e93-ad4a-4c65-b7fe-1482e80d87b4\") " pod="openshift-monitoring/telemeter-client-dfb96dff-4xpmq" Nov 01 20:31:22.386639 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:22.386458 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0e381e93-ad4a-4c65-b7fe-1482e80d87b4-federate-client-tls\") pod \"telemeter-client-dfb96dff-4xpmq\" (UID: \"0e381e93-ad4a-4c65-b7fe-1482e80d87b4\") " pod="openshift-monitoring/telemeter-client-dfb96dff-4xpmq" Nov 01 20:31:22.429347 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:22.429307 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 01 20:31:22.487502 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:22.487310 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e381e93-ad4a-4c65-b7fe-1482e80d87b4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-dfb96dff-4xpmq\" (UID: \"0e381e93-ad4a-4c65-b7fe-1482e80d87b4\") " pod="openshift-monitoring/telemeter-client-dfb96dff-4xpmq" Nov 01 20:31:22.488423 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:22.488394 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0e381e93-ad4a-4c65-b7fe-1482e80d87b4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-dfb96dff-4xpmq\" (UID: \"0e381e93-ad4a-4c65-b7fe-1482e80d87b4\") " pod="openshift-monitoring/telemeter-client-dfb96dff-4xpmq" Nov 01 20:31:22.488774 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:22.488312 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e381e93-ad4a-4c65-b7fe-1482e80d87b4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-dfb96dff-4xpmq\" (UID: \"0e381e93-ad4a-4c65-b7fe-1482e80d87b4\") " pod="openshift-monitoring/telemeter-client-dfb96dff-4xpmq" Nov 01 20:31:22.488774 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:22.488448 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e381e93-ad4a-4c65-b7fe-1482e80d87b4-serving-certs-ca-bundle\") pod \"telemeter-client-dfb96dff-4xpmq\" (UID: \"0e381e93-ad4a-4c65-b7fe-1482e80d87b4\") " pod="openshift-monitoring/telemeter-client-dfb96dff-4xpmq" Nov 01 20:31:22.488774 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:22.488476 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7xcx\" (UniqueName: \"kubernetes.io/projected/0e381e93-ad4a-4c65-b7fe-1482e80d87b4-kube-api-access-m7xcx\") pod \"telemeter-client-dfb96dff-4xpmq\" (UID: \"0e381e93-ad4a-4c65-b7fe-1482e80d87b4\") " pod="openshift-monitoring/telemeter-client-dfb96dff-4xpmq" Nov 01 20:31:22.488774 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:22.488516 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0e381e93-ad4a-4c65-b7fe-1482e80d87b4-telemeter-client-tls\") pod \"telemeter-client-dfb96dff-4xpmq\" (UID: \"0e381e93-ad4a-4c65-b7fe-1482e80d87b4\") " pod="openshift-monitoring/telemeter-client-dfb96dff-4xpmq" Nov 01 20:31:22.489355 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:22.489191 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e381e93-ad4a-4c65-b7fe-1482e80d87b4-serving-certs-ca-bundle\") pod \"telemeter-client-dfb96dff-4xpmq\" (UID: \"0e381e93-ad4a-4c65-b7fe-1482e80d87b4\") " pod="openshift-monitoring/telemeter-client-dfb96dff-4xpmq" Nov 01 20:31:22.489355 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:22.489252 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e381e93-ad4a-4c65-b7fe-1482e80d87b4-metrics-client-ca\") pod \"telemeter-client-dfb96dff-4xpmq\" (UID: \"0e381e93-ad4a-4c65-b7fe-1482e80d87b4\") " pod="openshift-monitoring/telemeter-client-dfb96dff-4xpmq" Nov 01 20:31:22.489355 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:22.489285 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0e381e93-ad4a-4c65-b7fe-1482e80d87b4-federate-client-tls\") pod \"telemeter-client-dfb96dff-4xpmq\" (UID: \"0e381e93-ad4a-4c65-b7fe-1482e80d87b4\") " pod="openshift-monitoring/telemeter-client-dfb96dff-4xpmq" Nov 01 20:31:22.489511 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:22.489364 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0e381e93-ad4a-4c65-b7fe-1482e80d87b4-secret-telemeter-client\") pod \"telemeter-client-dfb96dff-4xpmq\" (UID: \"0e381e93-ad4a-4c65-b7fe-1482e80d87b4\") " pod="openshift-monitoring/telemeter-client-dfb96dff-4xpmq" Nov 01 20:31:22.489925 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:22.489900 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e381e93-ad4a-4c65-b7fe-1482e80d87b4-metrics-client-ca\") pod \"telemeter-client-dfb96dff-4xpmq\" (UID: \"0e381e93-ad4a-4c65-b7fe-1482e80d87b4\") " pod="openshift-monitoring/telemeter-client-dfb96dff-4xpmq" Nov 01 20:31:22.492458 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:22.492435 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0e381e93-ad4a-4c65-b7fe-1482e80d87b4-federate-client-tls\") pod \"telemeter-client-dfb96dff-4xpmq\" (UID: \"0e381e93-ad4a-4c65-b7fe-1482e80d87b4\") " pod="openshift-monitoring/telemeter-client-dfb96dff-4xpmq" Nov 01 20:31:22.492558 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:22.492536 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0e381e93-ad4a-4c65-b7fe-1482e80d87b4-secret-telemeter-client\") pod \"telemeter-client-dfb96dff-4xpmq\" (UID: \"0e381e93-ad4a-4c65-b7fe-1482e80d87b4\") " pod="openshift-monitoring/telemeter-client-dfb96dff-4xpmq" Nov 01 20:31:22.493789 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:22.493765 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0e381e93-ad4a-4c65-b7fe-1482e80d87b4-telemeter-client-tls\") pod \"telemeter-client-dfb96dff-4xpmq\" (UID: \"0e381e93-ad4a-4c65-b7fe-1482e80d87b4\") " pod="openshift-monitoring/telemeter-client-dfb96dff-4xpmq" Nov 01 20:31:22.494466 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:22.494440 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0e381e93-ad4a-4c65-b7fe-1482e80d87b4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-dfb96dff-4xpmq\" (UID: \"0e381e93-ad4a-4c65-b7fe-1482e80d87b4\") " pod="openshift-monitoring/telemeter-client-dfb96dff-4xpmq" Nov 01 20:31:22.547333 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:22.547256 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7xcx\" (UniqueName: \"kubernetes.io/projected/0e381e93-ad4a-4c65-b7fe-1482e80d87b4-kube-api-access-m7xcx\") pod \"telemeter-client-dfb96dff-4xpmq\" (UID: \"0e381e93-ad4a-4c65-b7fe-1482e80d87b4\") " pod="openshift-monitoring/telemeter-client-dfb96dff-4xpmq" Nov 01 20:31:22.565842 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:22.565791 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hk84s" event={"ID":"c693b7c8-323f-4c08-b95f-bc2dca1ba6e3","Type":"ContainerStarted","Data":"e69783c2688db4a1841a8d083583c6b37bc605290978ce22aa59ff75e93f128e"} Nov 01 20:31:22.567405 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:22.567374 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-84f999b6b9-g8p4q" event={"ID":"8995aa4c-ac0d-4f43-aa03-bebced1e08e0","Type":"ContainerStarted","Data":"e3c15d488f984ca4d0366e70f8f00d6702586d5ffc71863716744ea79a8ded42"} Nov 01 20:31:22.641949 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:22.641907 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-dfb96dff-4xpmq" Nov 01 20:31:23.156404 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:23.156369 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-dfb96dff-4xpmq"] Nov 01 20:31:23.208625 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:31:23.208556 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e381e93_ad4a_4c65_b7fe_1482e80d87b4.slice/crio-404ffc26765b7f0df7cc0570b3aaedd5dfcbad023ce6c047283c53b3ecca5cf0 WatchSource:0}: Error finding container 404ffc26765b7f0df7cc0570b3aaedd5dfcbad023ce6c047283c53b3ecca5cf0: Status 404 returned error can't find the container with id 404ffc26765b7f0df7cc0570b3aaedd5dfcbad023ce6c047283c53b3ecca5cf0 Nov 01 20:31:23.433376 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:23.433258 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:23.576171 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:23.576089 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-dfb96dff-4xpmq" event={"ID":"0e381e93-ad4a-4c65-b7fe-1482e80d87b4","Type":"ContainerStarted","Data":"404ffc26765b7f0df7cc0570b3aaedd5dfcbad023ce6c047283c53b3ecca5cf0"} Nov 01 20:31:23.640366 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:23.640326 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:24.476264 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:24.476236 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-fdf47dfdc-r8c58" Nov 01 20:31:24.909767 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:24.909693 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:25.082840 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:25.082812 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:25.249895 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:25.249810 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:25.590659 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:25.590400 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bc9c45967-46rtx" event={"ID":"635876cb-613a-4691-a980-255f7df1bcef","Type":"ContainerStarted","Data":"6ca3966b7da52b64845188f7b1b77df5696912638eb0cc4caca6cb08d04debe3"} Nov 01 20:31:25.590659 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:25.590492 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bc9c45967-46rtx" event={"ID":"635876cb-613a-4691-a980-255f7df1bcef","Type":"ContainerStarted","Data":"04267718a395112f06b47ca1d8a6e73808351e889e61b1db358c0f096810ec98"} Nov 01 20:31:25.590659 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:25.590541 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bc9c45967-46rtx" event={"ID":"635876cb-613a-4691-a980-255f7df1bcef","Type":"ContainerStarted","Data":"d995c26d8d5d96e0e6a17dd1a3f9d57721abb556a170863dd95a7ac6b7204570"} Nov 01 20:31:25.593547 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:25.593473 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-84f999b6b9-g8p4q" event={"ID":"8995aa4c-ac0d-4f43-aa03-bebced1e08e0","Type":"ContainerStarted","Data":"4672bc9db082730fe09a5710936f223a6cce3fb634a2dcaadd24407d29d40b32"} Nov 01 20:31:25.777510 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:25.777437 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-84f999b6b9-g8p4q" podStartSLOduration=2.000335379 podStartE2EDuration="4.777417243s" totalImagesPullingTime="2.777081864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:31:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:31:25.712859565 +0000 UTC m=+36.490499699" watchObservedRunningTime="2026-11-01 20:31:25.777417243 +0000 UTC m=+36.555057377" Nov 01 20:31:25.778025 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:25.777854 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 20:31:25.779256 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:25.779224 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:25.798267 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:25.798241 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:25.853213 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:25.853121 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 01 20:31:25.853213 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:25.853144 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 01 20:31:25.853213 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:25.853137 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-2f24m\"" Nov 01 20:31:25.853213 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:25.853123 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 01 20:31:25.853213 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:25.853187 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1m0kfcr09p10u\"" Nov 01 20:31:25.853513 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:25.853306 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 01 20:31:25.906908 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:25.906874 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 20:31:25.922138 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:25.921974 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:25.922408 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:25.922336 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/04386401-6c41-42c3-ad9f-73702962c4a2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:25.922408 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:25.922365 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/04386401-6c41-42c3-ad9f-73702962c4a2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:25.922408 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:25.922388 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:25.922592 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:25.922488 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/04386401-6c41-42c3-ad9f-73702962c4a2-config-out\") pod \"prometheus-k8s-0\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:25.922592 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:25.922522 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-config\") pod \"prometheus-k8s-0\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:25.922592 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:25.922549 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:25.922592 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:25.922581 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/04386401-6c41-42c3-ad9f-73702962c4a2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:25.922799 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:25.922605 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6h69\" (UniqueName: \"kubernetes.io/projected/04386401-6c41-42c3-ad9f-73702962c4a2-kube-api-access-g6h69\") pod \"prometheus-k8s-0\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:25.922799 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:25.922672 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04386401-6c41-42c3-ad9f-73702962c4a2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:25.922799 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:25.922702 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-web-config\") pod \"prometheus-k8s-0\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:25.922799 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:25.922729 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/04386401-6c41-42c3-ad9f-73702962c4a2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:25.922799 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:25.922757 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:25.923059 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:25.922804 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:25.923059 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:25.922835 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04386401-6c41-42c3-ad9f-73702962c4a2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:25.923059 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:25.922865 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/04386401-6c41-42c3-ad9f-73702962c4a2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:25.923059 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:25.922901 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04386401-6c41-42c3-ad9f-73702962c4a2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:25.923059 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:25.922936 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/04386401-6c41-42c3-ad9f-73702962c4a2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:25.923059 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:25.922962 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:25.923059 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:25.922990 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:25.953552 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:25.953357 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 01 20:31:25.953552 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:25.953436 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 01 20:31:25.953552 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:25.953499 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 01 20:31:25.953552 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:25.953499 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 01 20:31:25.954155 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:25.953611 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 01 20:31:25.954155 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:25.953571 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 01 20:31:25.954155 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:25.953718 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 01 20:31:25.954155 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:25.953761 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 01 20:31:25.959241 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:25.959206 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:26.007033 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:26.006985 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 01 20:31:26.007217 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:26.007102 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 01 20:31:26.024433 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:26.024389 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:26.024739 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:26.024438 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04386401-6c41-42c3-ad9f-73702962c4a2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:26.024739 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:26.024474 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/04386401-6c41-42c3-ad9f-73702962c4a2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:26.024739 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:26.024498 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04386401-6c41-42c3-ad9f-73702962c4a2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:26.024739 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:26.024711 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/04386401-6c41-42c3-ad9f-73702962c4a2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:26.025036 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:26.024759 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:26.025036 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:26.024796 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:26.025036 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:26.024828 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:26.025036 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:26.024853 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/04386401-6c41-42c3-ad9f-73702962c4a2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:26.025036 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:26.024946 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/04386401-6c41-42c3-ad9f-73702962c4a2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:26.025036 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:26.024975 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:26.025366 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:26.025040 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/04386401-6c41-42c3-ad9f-73702962c4a2-config-out\") pod \"prometheus-k8s-0\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:26.025366 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:26.025079 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-config\") pod \"prometheus-k8s-0\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:26.025366 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:26.025103 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:26.025366 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:26.025137 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/04386401-6c41-42c3-ad9f-73702962c4a2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:26.025366 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:26.025162 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6h69\" (UniqueName: \"kubernetes.io/projected/04386401-6c41-42c3-ad9f-73702962c4a2-kube-api-access-g6h69\") pod \"prometheus-k8s-0\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:26.025366 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:26.025232 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04386401-6c41-42c3-ad9f-73702962c4a2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:26.025366 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:26.025265 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-web-config\") pod \"prometheus-k8s-0\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:26.025366 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:26.025292 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/04386401-6c41-42c3-ad9f-73702962c4a2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:26.025366 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:26.025321 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:26.025366 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:26.025336 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04386401-6c41-42c3-ad9f-73702962c4a2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:26.025904 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:26.025853 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/04386401-6c41-42c3-ad9f-73702962c4a2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:26.026450 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:26.026420 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04386401-6c41-42c3-ad9f-73702962c4a2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:26.026569 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:26.026425 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04386401-6c41-42c3-ad9f-73702962c4a2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:26.027109 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:26.027081 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/04386401-6c41-42c3-ad9f-73702962c4a2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:26.029136 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:26.029110 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:26.037239 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:26.029388 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/04386401-6c41-42c3-ad9f-73702962c4a2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:26.037239 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:26.030225 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/04386401-6c41-42c3-ad9f-73702962c4a2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:26.037239 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:26.030632 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/04386401-6c41-42c3-ad9f-73702962c4a2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:26.037239 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:26.030719 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-config\") pod \"prometheus-k8s-0\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:26.037239 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:26.031204 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-web-config\") pod \"prometheus-k8s-0\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:26.037239 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:26.032485 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:26.037239 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:26.033428 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:26.037239 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:26.033706 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:26.037239 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:26.035303 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:26.038941 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:26.038911 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:26.040212 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:26.040185 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/04386401-6c41-42c3-ad9f-73702962c4a2-config-out\") pod \"prometheus-k8s-0\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:26.041322 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:26.041298 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:26.045698 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:26.045672 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/04386401-6c41-42c3-ad9f-73702962c4a2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:26.084943 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:26.084911 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6h69\" (UniqueName: \"kubernetes.io/projected/04386401-6c41-42c3-ad9f-73702962c4a2-kube-api-access-g6h69\") pod \"prometheus-k8s-0\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:26.120827 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:26.120678 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:26.270414 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:26.270380 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:26.604360 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:26.604295 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hk84s" event={"ID":"c693b7c8-323f-4c08-b95f-bc2dca1ba6e3","Type":"ContainerStarted","Data":"3502fe94c62f24a347564b3ec6ed8ee8f45069142e1b10666bc77f8b2f47b857"} Nov 01 20:31:26.604829 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:26.604673 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hk84s" Nov 01 20:31:26.610991 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:26.610967 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hk84s" Nov 01 20:31:26.719864 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:26.719806 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hk84s" podStartSLOduration=2.027969706 podStartE2EDuration="5.719788501s" totalImagesPullingTime="3.691818795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:31:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:31:26.719031167 +0000 UTC m=+37.496671304" watchObservedRunningTime="2026-11-01 20:31:26.719788501 +0000 UTC m=+37.497428637" Nov 01 20:31:26.787376 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:26.787340 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 20:31:26.900538 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:31:26.900423 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04386401_6c41_42c3_ad9f_73702962c4a2.slice/crio-6d8adbf9b13c2a9dfcf25359acaafc32e5b144d4098136c859b6dfd4ee058582 WatchSource:0}: Error finding container 6d8adbf9b13c2a9dfcf25359acaafc32e5b144d4098136c859b6dfd4ee058582: Status 404 returned error can't find the container with id 6d8adbf9b13c2a9dfcf25359acaafc32e5b144d4098136c859b6dfd4ee058582 Nov 01 20:31:27.612061 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:27.612021 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bc9c45967-46rtx" event={"ID":"635876cb-613a-4691-a980-255f7df1bcef","Type":"ContainerStarted","Data":"560354b91090da6a663989dba3237b3f4c5fdcd9bd2631a7eda60169a03fa300"} Nov 01 20:31:27.612744 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:27.612716 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6bc9c45967-46rtx" Nov 01 20:31:27.612856 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:27.612776 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bc9c45967-46rtx" event={"ID":"635876cb-613a-4691-a980-255f7df1bcef","Type":"ContainerStarted","Data":"42e5df056c1b45f5796158170d1eed88f5b6042b40ac4483d735839f63709ae4"} Nov 01 20:31:27.612856 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:27.612820 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bc9c45967-46rtx" event={"ID":"635876cb-613a-4691-a980-255f7df1bcef","Type":"ContainerStarted","Data":"6f7c49aef281e3466a9bc2893785cae729a7daada233a0f4fea298297cdecc45"} Nov 01 20:31:27.613535 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:27.613503 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"04386401-6c41-42c3-ad9f-73702962c4a2","Type":"ContainerStarted","Data":"6d8adbf9b13c2a9dfcf25359acaafc32e5b144d4098136c859b6dfd4ee058582"} Nov 01 20:31:27.615331 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:27.615303 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-dfb96dff-4xpmq" event={"ID":"0e381e93-ad4a-4c65-b7fe-1482e80d87b4","Type":"ContainerStarted","Data":"ab5714a4f54c998015b7875552eeaebbeef8c69f78784fcbc0f8458d8055a5ed"} Nov 01 20:31:27.733490 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:27.733402 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6bc9c45967-46rtx" podStartSLOduration=2.180949237 podStartE2EDuration="8.733388121s" totalImagesPullingTime="6.552438884s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:31:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:31:27.732336987 +0000 UTC m=+38.509977122" watchObservedRunningTime="2026-11-01 20:31:27.733388121 +0000 UTC m=+38.511028250" Nov 01 20:31:29.627833 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:29.627796 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-dfb96dff-4xpmq" event={"ID":"0e381e93-ad4a-4c65-b7fe-1482e80d87b4","Type":"ContainerStarted","Data":"23c3f8ab5dda3f4d2597adf69941b62acbef96373f04181b9dddb208c9a7481c"} Nov 01 20:31:29.628368 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:29.627893 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-dfb96dff-4xpmq" event={"ID":"0e381e93-ad4a-4c65-b7fe-1482e80d87b4","Type":"ContainerStarted","Data":"8084a3c1189f50789dfac181bb5e16c8603660dd43d4ab56406d8366cb97f318"} Nov 01 20:31:29.629394 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:29.629368 4484 generic.go:356] "Generic (PLEG): container finished" podID="04386401-6c41-42c3-ad9f-73702962c4a2" containerID="fd9263fae24a75ce31837067678f2dd05979705ba5917ddf8195e2abaa0c3e3e" exitCode=0 Nov 01 20:31:29.629508 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:29.629401 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"04386401-6c41-42c3-ad9f-73702962c4a2","Type":"ContainerDied","Data":"fd9263fae24a75ce31837067678f2dd05979705ba5917ddf8195e2abaa0c3e3e"} Nov 01 20:31:29.779954 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:29.779904 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-dfb96dff-4xpmq" podStartSLOduration=1.876126416 podStartE2EDuration="7.779892498s" totalImagesPullingTime="5.903766082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:31:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:31:29.77866119 +0000 UTC m=+40.556301325" watchObservedRunningTime="2026-11-01 20:31:29.779892498 +0000 UTC m=+40.557532633" Nov 01 20:31:30.427846 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:30.427813 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:33.066157 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:33.066052 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:33.252379 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:33.252341 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:33.628981 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:33.628954 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6bc9c45967-46rtx" Nov 01 20:31:34.444974 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:34.444867 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:34.607060 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:34.607025 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:34.652892 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:34.652854 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"04386401-6c41-42c3-ad9f-73702962c4a2","Type":"ContainerStarted","Data":"4d400d325b72cea886c224352b3e0ac77faa4d54d0230195bb4a8565956dd020"} Nov 01 20:31:34.653107 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:34.652930 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"04386401-6c41-42c3-ad9f-73702962c4a2","Type":"ContainerStarted","Data":"761987ac7dee4c796185aa989dd1dcdf41a13667a650644cb9e7a643601e12c1"} Nov 01 20:31:34.653107 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:34.652973 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"04386401-6c41-42c3-ad9f-73702962c4a2","Type":"ContainerStarted","Data":"3b2ec0b592dd5236a454da217a88c714374c7bca27ee4fa2f0274fb1eb70a68a"} Nov 01 20:31:34.653107 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:34.653023 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"04386401-6c41-42c3-ad9f-73702962c4a2","Type":"ContainerStarted","Data":"12af7378aec2923ddae356156bc6b219d813668b82743a6de54b104072562f17"} Nov 01 20:31:34.653107 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:34.653069 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"04386401-6c41-42c3-ad9f-73702962c4a2","Type":"ContainerStarted","Data":"f21818b0e5208bb61b0859d1d0e7defaf5f31b66f09234809f226959427ace1a"} Nov 01 20:31:34.653107 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:34.653105 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"04386401-6c41-42c3-ad9f-73702962c4a2","Type":"ContainerStarted","Data":"8daec41b36d5a569ffc2b6bb641246bda6d21f82e1006a18e55fda5a9133ef21"} Nov 01 20:31:34.782505 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:34.782458 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.631752676 podStartE2EDuration="9.78244414s" totalImagesPullingTime="6.128790258s" totalInitContainerRuntime="21.901206ms" isStatefulPod=false podCreationTimestamp="2026-11-01 20:31:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:31:34.780991249 +0000 UTC m=+45.558631386" watchObservedRunningTime="2026-11-01 20:31:34.78244414 +0000 UTC m=+45.560084330" Nov 01 20:31:36.270949 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:36.270911 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:31:37.835318 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:37.835278 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:37.991808 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:37.991773 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:41.784661 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:41.784619 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-84f999b6b9-g8p4q" Nov 01 20:31:41.785171 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:41.784701 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-84f999b6b9-g8p4q" Nov 01 20:31:45.765132 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:31:45.765095 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 20:31:47.397822 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:47.397791 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:47.572101 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:47.572063 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:55.054927 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:55.054886 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:55.206803 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:31:55.206763 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:32:01.790082 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:01.790052 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-84f999b6b9-g8p4q" Nov 01 20:32:01.794030 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:01.794009 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-84f999b6b9-g8p4q" Nov 01 20:32:02.394326 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:02.394288 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:32:02.554952 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:02.554903 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:32:08.765626 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:08.765582 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:08.816863 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:08.816835 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:08.832254 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:08.832226 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:10.612653 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:10.612618 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:32:15.489787 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:15.489754 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:32:15.764062 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:32:15.763970 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 20:32:16.559790 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:16.559751 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:32:16.857671 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:16.857591 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:32:17.012395 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:17.012360 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:32:21.610260 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:21.610205 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:32:28.122535 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:28.122496 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:32:28.270843 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:28.270799 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:32:28.851560 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:28.851520 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 20:32:28.851990 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:28.851957 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="04386401-6c41-42c3-ad9f-73702962c4a2" containerName="prometheus" containerID="cri-o://8daec41b36d5a569ffc2b6bb641246bda6d21f82e1006a18e55fda5a9133ef21" gracePeriod=600 Nov 01 20:32:28.852135 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:28.851969 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="04386401-6c41-42c3-ad9f-73702962c4a2" containerName="kube-rbac-proxy" containerID="cri-o://761987ac7dee4c796185aa989dd1dcdf41a13667a650644cb9e7a643601e12c1" gracePeriod=600 Nov 01 20:32:28.852135 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:28.851975 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="04386401-6c41-42c3-ad9f-73702962c4a2" containerName="thanos-sidecar" containerID="cri-o://12af7378aec2923ddae356156bc6b219d813668b82743a6de54b104072562f17" gracePeriod=600 Nov 01 20:32:28.852135 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:28.852032 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="04386401-6c41-42c3-ad9f-73702962c4a2" containerName="config-reloader" containerID="cri-o://f21818b0e5208bb61b0859d1d0e7defaf5f31b66f09234809f226959427ace1a" gracePeriod=600 Nov 01 20:32:28.852135 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:28.852082 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="04386401-6c41-42c3-ad9f-73702962c4a2" containerName="kube-rbac-proxy-thanos" containerID="cri-o://4d400d325b72cea886c224352b3e0ac77faa4d54d0230195bb4a8565956dd020" gracePeriod=600 Nov 01 20:32:28.852135 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:28.852029 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="04386401-6c41-42c3-ad9f-73702962c4a2" containerName="kube-rbac-proxy-web" containerID="cri-o://3b2ec0b592dd5236a454da217a88c714374c7bca27ee4fa2f0274fb1eb70a68a" gracePeriod=600 Nov 01 20:32:29.097413 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.097387 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:29.295185 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.295149 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04386401-6c41-42c3-ad9f-73702962c4a2-config-out\" (UniqueName: \"kubernetes.io/empty-dir/04386401-6c41-42c3-ad9f-73702962c4a2-config-out\") pod \"04386401-6c41-42c3-ad9f-73702962c4a2\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " Nov 01 20:32:29.295185 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.295188 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04386401-6c41-42c3-ad9f-73702962c4a2-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/04386401-6c41-42c3-ad9f-73702962c4a2-prometheus-k8s-db\") pod \"04386401-6c41-42c3-ad9f-73702962c4a2\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " Nov 01 20:32:29.295722 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.295211 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-secret-metrics-client-certs\") pod \"04386401-6c41-42c3-ad9f-73702962c4a2\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " Nov 01 20:32:29.295722 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.295257 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04386401-6c41-42c3-ad9f-73702962c4a2-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04386401-6c41-42c3-ad9f-73702962c4a2-configmap-serving-certs-ca-bundle\") pod \"04386401-6c41-42c3-ad9f-73702962c4a2\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " Nov 01 20:32:29.295722 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.295287 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-secret-kube-rbac-proxy\") pod \"04386401-6c41-42c3-ad9f-73702962c4a2\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " Nov 01 20:32:29.295722 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.295327 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04386401-6c41-42c3-ad9f-73702962c4a2-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/04386401-6c41-42c3-ad9f-73702962c4a2-prometheus-k8s-rulefiles-1\") pod \"04386401-6c41-42c3-ad9f-73702962c4a2\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " Nov 01 20:32:29.295722 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.295506 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04386401-6c41-42c3-ad9f-73702962c4a2-tls-assets\" (UniqueName: \"kubernetes.io/projected/04386401-6c41-42c3-ad9f-73702962c4a2-tls-assets\") pod \"04386401-6c41-42c3-ad9f-73702962c4a2\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " Nov 01 20:32:29.295722 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.295569 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"04386401-6c41-42c3-ad9f-73702962c4a2\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " Nov 01 20:32:29.295722 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.295613 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"04386401-6c41-42c3-ad9f-73702962c4a2\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " Nov 01 20:32:29.295722 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.295644 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04386401-6c41-42c3-ad9f-73702962c4a2-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04386401-6c41-42c3-ad9f-73702962c4a2-prometheus-trusted-ca-bundle\") pod \"04386401-6c41-42c3-ad9f-73702962c4a2\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " Nov 01 20:32:29.295722 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.295650 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04386401-6c41-42c3-ad9f-73702962c4a2-prometheus-k8s-rulefiles-1" pod "04386401-6c41-42c3-ad9f-73702962c4a2" (UID: "04386401-6c41-42c3-ad9f-73702962c4a2"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:32:29.295722 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.295676 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-secret-prometheus-k8s-tls\") pod \"04386401-6c41-42c3-ad9f-73702962c4a2\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " Nov 01 20:32:29.295722 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.295703 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-config\" (UniqueName: \"kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-config\") pod \"04386401-6c41-42c3-ad9f-73702962c4a2\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " Nov 01 20:32:29.295722 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.295716 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04386401-6c41-42c3-ad9f-73702962c4a2-configmap-serving-certs-ca-bundle" pod "04386401-6c41-42c3-ad9f-73702962c4a2" (UID: "04386401-6c41-42c3-ad9f-73702962c4a2"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:32:29.295722 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.295732 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04386401-6c41-42c3-ad9f-73702962c4a2-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04386401-6c41-42c3-ad9f-73702962c4a2-configmap-kubelet-serving-ca-bundle\") pod \"04386401-6c41-42c3-ad9f-73702962c4a2\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " Nov 01 20:32:29.296424 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.295758 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04386401-6c41-42c3-ad9f-73702962c4a2-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/04386401-6c41-42c3-ad9f-73702962c4a2-prometheus-k8s-rulefiles-2\") pod \"04386401-6c41-42c3-ad9f-73702962c4a2\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " Nov 01 20:32:29.296424 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.295803 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04386401-6c41-42c3-ad9f-73702962c4a2-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/04386401-6c41-42c3-ad9f-73702962c4a2-prometheus-k8s-rulefiles-0\") pod \"04386401-6c41-42c3-ad9f-73702962c4a2\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " Nov 01 20:32:29.296424 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.295831 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-secret-grpc-tls\") pod \"04386401-6c41-42c3-ad9f-73702962c4a2\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " Nov 01 20:32:29.296424 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.295865 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04386401-6c41-42c3-ad9f-73702962c4a2-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/04386401-6c41-42c3-ad9f-73702962c4a2-configmap-metrics-client-ca\") pod \"04386401-6c41-42c3-ad9f-73702962c4a2\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " Nov 01 20:32:29.296424 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.295901 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-thanos-prometheus-http-client-file\") pod \"04386401-6c41-42c3-ad9f-73702962c4a2\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " Nov 01 20:32:29.296424 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.295953 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04386401-6c41-42c3-ad9f-73702962c4a2-kube-api-access-g6h69\" (UniqueName: \"kubernetes.io/projected/04386401-6c41-42c3-ad9f-73702962c4a2-kube-api-access-g6h69\") pod \"04386401-6c41-42c3-ad9f-73702962c4a2\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " Nov 01 20:32:29.296424 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.296015 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-web-config\" (UniqueName: \"kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-web-config\") pod \"04386401-6c41-42c3-ad9f-73702962c4a2\" (UID: \"04386401-6c41-42c3-ad9f-73702962c4a2\") " Nov 01 20:32:29.296424 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.296304 4484 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/04386401-6c41-42c3-ad9f-73702962c4a2-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:29.296424 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.296324 4484 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04386401-6c41-42c3-ad9f-73702962c4a2-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:29.296876 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.296693 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04386401-6c41-42c3-ad9f-73702962c4a2-prometheus-k8s-db" pod "04386401-6c41-42c3-ad9f-73702962c4a2" (UID: "04386401-6c41-42c3-ad9f-73702962c4a2"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:32:29.296937 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.296895 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04386401-6c41-42c3-ad9f-73702962c4a2-prometheus-k8s-rulefiles-2" pod "04386401-6c41-42c3-ad9f-73702962c4a2" (UID: "04386401-6c41-42c3-ad9f-73702962c4a2"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:32:29.297816 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.297508 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04386401-6c41-42c3-ad9f-73702962c4a2-prometheus-trusted-ca-bundle" pod "04386401-6c41-42c3-ad9f-73702962c4a2" (UID: "04386401-6c41-42c3-ad9f-73702962c4a2"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:32:29.297816 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.297576 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04386401-6c41-42c3-ad9f-73702962c4a2-configmap-metrics-client-ca" pod "04386401-6c41-42c3-ad9f-73702962c4a2" (UID: "04386401-6c41-42c3-ad9f-73702962c4a2"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:32:29.298582 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.298447 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04386401-6c41-42c3-ad9f-73702962c4a2-configmap-kubelet-serving-ca-bundle" pod "04386401-6c41-42c3-ad9f-73702962c4a2" (UID: "04386401-6c41-42c3-ad9f-73702962c4a2"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:32:29.298582 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.298530 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-secret-kube-rbac-proxy" pod "04386401-6c41-42c3-ad9f-73702962c4a2" (UID: "04386401-6c41-42c3-ad9f-73702962c4a2"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:32:29.299640 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.299599 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04386401-6c41-42c3-ad9f-73702962c4a2-prometheus-k8s-rulefiles-0" pod "04386401-6c41-42c3-ad9f-73702962c4a2" (UID: "04386401-6c41-42c3-ad9f-73702962c4a2"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:32:29.300043 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.300009 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-secret-prometheus-k8s-thanos-sidecar-tls" pod "04386401-6c41-42c3-ad9f-73702962c4a2" (UID: "04386401-6c41-42c3-ad9f-73702962c4a2"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:32:29.300242 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.300219 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04386401-6c41-42c3-ad9f-73702962c4a2-tls-assets" pod "04386401-6c41-42c3-ad9f-73702962c4a2" (UID: "04386401-6c41-42c3-ad9f-73702962c4a2"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:32:29.300341 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.300270 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-secret-prometheus-k8s-kube-rbac-proxy-web" pod "04386401-6c41-42c3-ad9f-73702962c4a2" (UID: "04386401-6c41-42c3-ad9f-73702962c4a2"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:32:29.300341 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.300282 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-secret-metrics-client-certs" pod "04386401-6c41-42c3-ad9f-73702962c4a2" (UID: "04386401-6c41-42c3-ad9f-73702962c4a2"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:32:29.300341 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.300311 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-secret-prometheus-k8s-tls" pod "04386401-6c41-42c3-ad9f-73702962c4a2" (UID: "04386401-6c41-42c3-ad9f-73702962c4a2"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:32:29.300488 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.300362 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-config" pod "04386401-6c41-42c3-ad9f-73702962c4a2" (UID: "04386401-6c41-42c3-ad9f-73702962c4a2"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:32:29.300653 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.300628 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-secret-grpc-tls" pod "04386401-6c41-42c3-ad9f-73702962c4a2" (UID: "04386401-6c41-42c3-ad9f-73702962c4a2"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:32:29.300750 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.300635 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04386401-6c41-42c3-ad9f-73702962c4a2-config-out" pod "04386401-6c41-42c3-ad9f-73702962c4a2" (UID: "04386401-6c41-42c3-ad9f-73702962c4a2"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:32:29.300750 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.300734 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04386401-6c41-42c3-ad9f-73702962c4a2-kube-api-access-g6h69" pod "04386401-6c41-42c3-ad9f-73702962c4a2" (UID: "04386401-6c41-42c3-ad9f-73702962c4a2"). InnerVolumeSpecName "kube-api-access-g6h69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:32:29.300839 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.300808 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-thanos-prometheus-http-client-file" pod "04386401-6c41-42c3-ad9f-73702962c4a2" (UID: "04386401-6c41-42c3-ad9f-73702962c4a2"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:32:29.311129 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.311100 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-web-config" pod "04386401-6c41-42c3-ad9f-73702962c4a2" (UID: "04386401-6c41-42c3-ad9f-73702962c4a2"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:32:29.397066 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.397038 4484 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/04386401-6c41-42c3-ad9f-73702962c4a2-tls-assets\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:29.397066 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.397066 4484 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:29.397204 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.397076 4484 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:29.397204 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.397089 4484 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04386401-6c41-42c3-ad9f-73702962c4a2-prometheus-trusted-ca-bundle\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:29.397204 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.397099 4484 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-secret-prometheus-k8s-tls\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:29.397204 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.397108 4484 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-config\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:29.397204 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.397117 4484 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04386401-6c41-42c3-ad9f-73702962c4a2-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:29.397204 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.397126 4484 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/04386401-6c41-42c3-ad9f-73702962c4a2-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:29.397204 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.397134 4484 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/04386401-6c41-42c3-ad9f-73702962c4a2-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:29.397204 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.397143 4484 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-secret-grpc-tls\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:29.397204 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.397152 4484 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/04386401-6c41-42c3-ad9f-73702962c4a2-configmap-metrics-client-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:29.397204 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.397160 4484 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:29.397204 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.397192 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6h69\" (UniqueName: \"kubernetes.io/projected/04386401-6c41-42c3-ad9f-73702962c4a2-kube-api-access-g6h69\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:29.397204 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.397201 4484 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-web-config\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:29.397537 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.397210 4484 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/04386401-6c41-42c3-ad9f-73702962c4a2-config-out\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:29.397537 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.397219 4484 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/04386401-6c41-42c3-ad9f-73702962c4a2-prometheus-k8s-db\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:29.397537 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.397227 4484 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-secret-metrics-client-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:29.397537 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.397236 4484 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/04386401-6c41-42c3-ad9f-73702962c4a2-secret-kube-rbac-proxy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:29.859649 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.859611 4484 generic.go:356] "Generic (PLEG): container finished" podID="04386401-6c41-42c3-ad9f-73702962c4a2" containerID="4d400d325b72cea886c224352b3e0ac77faa4d54d0230195bb4a8565956dd020" exitCode=0 Nov 01 20:32:29.859649 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.859644 4484 generic.go:356] "Generic (PLEG): container finished" podID="04386401-6c41-42c3-ad9f-73702962c4a2" containerID="761987ac7dee4c796185aa989dd1dcdf41a13667a650644cb9e7a643601e12c1" exitCode=0 Nov 01 20:32:29.859649 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.859656 4484 generic.go:356] "Generic (PLEG): container finished" podID="04386401-6c41-42c3-ad9f-73702962c4a2" containerID="3b2ec0b592dd5236a454da217a88c714374c7bca27ee4fa2f0274fb1eb70a68a" exitCode=0 Nov 01 20:32:29.859892 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.859669 4484 generic.go:356] "Generic (PLEG): container finished" podID="04386401-6c41-42c3-ad9f-73702962c4a2" containerID="12af7378aec2923ddae356156bc6b219d813668b82743a6de54b104072562f17" exitCode=0 Nov 01 20:32:29.859892 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.859679 4484 generic.go:356] "Generic (PLEG): container finished" podID="04386401-6c41-42c3-ad9f-73702962c4a2" containerID="f21818b0e5208bb61b0859d1d0e7defaf5f31b66f09234809f226959427ace1a" exitCode=0 Nov 01 20:32:29.859892 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.859688 4484 generic.go:356] "Generic (PLEG): container finished" podID="04386401-6c41-42c3-ad9f-73702962c4a2" containerID="8daec41b36d5a569ffc2b6bb641246bda6d21f82e1006a18e55fda5a9133ef21" exitCode=0 Nov 01 20:32:29.859892 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.859692 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"04386401-6c41-42c3-ad9f-73702962c4a2","Type":"ContainerDied","Data":"4d400d325b72cea886c224352b3e0ac77faa4d54d0230195bb4a8565956dd020"} Nov 01 20:32:29.859892 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.859738 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:29.859892 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.859800 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"04386401-6c41-42c3-ad9f-73702962c4a2","Type":"ContainerDied","Data":"761987ac7dee4c796185aa989dd1dcdf41a13667a650644cb9e7a643601e12c1"} Nov 01 20:32:29.859892 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.859830 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"04386401-6c41-42c3-ad9f-73702962c4a2","Type":"ContainerDied","Data":"3b2ec0b592dd5236a454da217a88c714374c7bca27ee4fa2f0274fb1eb70a68a"} Nov 01 20:32:29.859892 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.859854 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"04386401-6c41-42c3-ad9f-73702962c4a2","Type":"ContainerDied","Data":"12af7378aec2923ddae356156bc6b219d813668b82743a6de54b104072562f17"} Nov 01 20:32:29.859892 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.859871 4484 scope.go:122] "RemoveContainer" containerID="4d400d325b72cea886c224352b3e0ac77faa4d54d0230195bb4a8565956dd020" Nov 01 20:32:29.860229 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.859900 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"04386401-6c41-42c3-ad9f-73702962c4a2","Type":"ContainerDied","Data":"f21818b0e5208bb61b0859d1d0e7defaf5f31b66f09234809f226959427ace1a"} Nov 01 20:32:29.860229 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.859924 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"04386401-6c41-42c3-ad9f-73702962c4a2","Type":"ContainerDied","Data":"8daec41b36d5a569ffc2b6bb641246bda6d21f82e1006a18e55fda5a9133ef21"} Nov 01 20:32:29.860229 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.859948 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"04386401-6c41-42c3-ad9f-73702962c4a2","Type":"ContainerDied","Data":"6d8adbf9b13c2a9dfcf25359acaafc32e5b144d4098136c859b6dfd4ee058582"} Nov 01 20:32:29.880114 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.880078 4484 scope.go:122] "RemoveContainer" containerID="761987ac7dee4c796185aa989dd1dcdf41a13667a650644cb9e7a643601e12c1" Nov 01 20:32:29.887333 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.887309 4484 scope.go:122] "RemoveContainer" containerID="3b2ec0b592dd5236a454da217a88c714374c7bca27ee4fa2f0274fb1eb70a68a" Nov 01 20:32:29.895972 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.895949 4484 scope.go:122] "RemoveContainer" containerID="12af7378aec2923ddae356156bc6b219d813668b82743a6de54b104072562f17" Nov 01 20:32:29.902592 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.902575 4484 scope.go:122] "RemoveContainer" containerID="f21818b0e5208bb61b0859d1d0e7defaf5f31b66f09234809f226959427ace1a" Nov 01 20:32:29.909470 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.909454 4484 scope.go:122] "RemoveContainer" containerID="8daec41b36d5a569ffc2b6bb641246bda6d21f82e1006a18e55fda5a9133ef21" Nov 01 20:32:29.916738 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.916723 4484 scope.go:122] "RemoveContainer" containerID="fd9263fae24a75ce31837067678f2dd05979705ba5917ddf8195e2abaa0c3e3e" Nov 01 20:32:29.923386 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.923368 4484 scope.go:122] "RemoveContainer" containerID="4d400d325b72cea886c224352b3e0ac77faa4d54d0230195bb4a8565956dd020" Nov 01 20:32:29.923648 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:32:29.923626 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d400d325b72cea886c224352b3e0ac77faa4d54d0230195bb4a8565956dd020\": container with ID starting with 4d400d325b72cea886c224352b3e0ac77faa4d54d0230195bb4a8565956dd020 not found: ID does not exist" containerID="4d400d325b72cea886c224352b3e0ac77faa4d54d0230195bb4a8565956dd020" Nov 01 20:32:29.923712 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.923658 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d400d325b72cea886c224352b3e0ac77faa4d54d0230195bb4a8565956dd020"} err="failed to get container status \"4d400d325b72cea886c224352b3e0ac77faa4d54d0230195bb4a8565956dd020\": rpc error: code = NotFound desc = could not find container \"4d400d325b72cea886c224352b3e0ac77faa4d54d0230195bb4a8565956dd020\": container with ID starting with 4d400d325b72cea886c224352b3e0ac77faa4d54d0230195bb4a8565956dd020 not found: ID does not exist" Nov 01 20:32:29.923712 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.923692 4484 scope.go:122] "RemoveContainer" containerID="761987ac7dee4c796185aa989dd1dcdf41a13667a650644cb9e7a643601e12c1" Nov 01 20:32:29.923916 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:32:29.923901 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"761987ac7dee4c796185aa989dd1dcdf41a13667a650644cb9e7a643601e12c1\": container with ID starting with 761987ac7dee4c796185aa989dd1dcdf41a13667a650644cb9e7a643601e12c1 not found: ID does not exist" containerID="761987ac7dee4c796185aa989dd1dcdf41a13667a650644cb9e7a643601e12c1" Nov 01 20:32:29.923960 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.923919 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"761987ac7dee4c796185aa989dd1dcdf41a13667a650644cb9e7a643601e12c1"} err="failed to get container status \"761987ac7dee4c796185aa989dd1dcdf41a13667a650644cb9e7a643601e12c1\": rpc error: code = NotFound desc = could not find container \"761987ac7dee4c796185aa989dd1dcdf41a13667a650644cb9e7a643601e12c1\": container with ID starting with 761987ac7dee4c796185aa989dd1dcdf41a13667a650644cb9e7a643601e12c1 not found: ID does not exist" Nov 01 20:32:29.923960 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.923930 4484 scope.go:122] "RemoveContainer" containerID="3b2ec0b592dd5236a454da217a88c714374c7bca27ee4fa2f0274fb1eb70a68a" Nov 01 20:32:29.924169 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:32:29.924152 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b2ec0b592dd5236a454da217a88c714374c7bca27ee4fa2f0274fb1eb70a68a\": container with ID starting with 3b2ec0b592dd5236a454da217a88c714374c7bca27ee4fa2f0274fb1eb70a68a not found: ID does not exist" containerID="3b2ec0b592dd5236a454da217a88c714374c7bca27ee4fa2f0274fb1eb70a68a" Nov 01 20:32:29.924217 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.924173 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b2ec0b592dd5236a454da217a88c714374c7bca27ee4fa2f0274fb1eb70a68a"} err="failed to get container status \"3b2ec0b592dd5236a454da217a88c714374c7bca27ee4fa2f0274fb1eb70a68a\": rpc error: code = NotFound desc = could not find container \"3b2ec0b592dd5236a454da217a88c714374c7bca27ee4fa2f0274fb1eb70a68a\": container with ID starting with 3b2ec0b592dd5236a454da217a88c714374c7bca27ee4fa2f0274fb1eb70a68a not found: ID does not exist" Nov 01 20:32:29.924217 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.924187 4484 scope.go:122] "RemoveContainer" containerID="12af7378aec2923ddae356156bc6b219d813668b82743a6de54b104072562f17" Nov 01 20:32:29.924400 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:32:29.924384 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12af7378aec2923ddae356156bc6b219d813668b82743a6de54b104072562f17\": container with ID starting with 12af7378aec2923ddae356156bc6b219d813668b82743a6de54b104072562f17 not found: ID does not exist" containerID="12af7378aec2923ddae356156bc6b219d813668b82743a6de54b104072562f17" Nov 01 20:32:29.924447 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.924403 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12af7378aec2923ddae356156bc6b219d813668b82743a6de54b104072562f17"} err="failed to get container status \"12af7378aec2923ddae356156bc6b219d813668b82743a6de54b104072562f17\": rpc error: code = NotFound desc = could not find container \"12af7378aec2923ddae356156bc6b219d813668b82743a6de54b104072562f17\": container with ID starting with 12af7378aec2923ddae356156bc6b219d813668b82743a6de54b104072562f17 not found: ID does not exist" Nov 01 20:32:29.924447 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.924415 4484 scope.go:122] "RemoveContainer" containerID="f21818b0e5208bb61b0859d1d0e7defaf5f31b66f09234809f226959427ace1a" Nov 01 20:32:29.924622 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:32:29.924607 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f21818b0e5208bb61b0859d1d0e7defaf5f31b66f09234809f226959427ace1a\": container with ID starting with f21818b0e5208bb61b0859d1d0e7defaf5f31b66f09234809f226959427ace1a not found: ID does not exist" containerID="f21818b0e5208bb61b0859d1d0e7defaf5f31b66f09234809f226959427ace1a" Nov 01 20:32:29.924665 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.924625 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f21818b0e5208bb61b0859d1d0e7defaf5f31b66f09234809f226959427ace1a"} err="failed to get container status \"f21818b0e5208bb61b0859d1d0e7defaf5f31b66f09234809f226959427ace1a\": rpc error: code = NotFound desc = could not find container \"f21818b0e5208bb61b0859d1d0e7defaf5f31b66f09234809f226959427ace1a\": container with ID starting with f21818b0e5208bb61b0859d1d0e7defaf5f31b66f09234809f226959427ace1a not found: ID does not exist" Nov 01 20:32:29.924665 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.924635 4484 scope.go:122] "RemoveContainer" containerID="8daec41b36d5a569ffc2b6bb641246bda6d21f82e1006a18e55fda5a9133ef21" Nov 01 20:32:29.924835 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:32:29.924819 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8daec41b36d5a569ffc2b6bb641246bda6d21f82e1006a18e55fda5a9133ef21\": container with ID starting with 8daec41b36d5a569ffc2b6bb641246bda6d21f82e1006a18e55fda5a9133ef21 not found: ID does not exist" containerID="8daec41b36d5a569ffc2b6bb641246bda6d21f82e1006a18e55fda5a9133ef21" Nov 01 20:32:29.924873 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.924837 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8daec41b36d5a569ffc2b6bb641246bda6d21f82e1006a18e55fda5a9133ef21"} err="failed to get container status \"8daec41b36d5a569ffc2b6bb641246bda6d21f82e1006a18e55fda5a9133ef21\": rpc error: code = NotFound desc = could not find container \"8daec41b36d5a569ffc2b6bb641246bda6d21f82e1006a18e55fda5a9133ef21\": container with ID starting with 8daec41b36d5a569ffc2b6bb641246bda6d21f82e1006a18e55fda5a9133ef21 not found: ID does not exist" Nov 01 20:32:29.924873 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.924849 4484 scope.go:122] "RemoveContainer" containerID="fd9263fae24a75ce31837067678f2dd05979705ba5917ddf8195e2abaa0c3e3e" Nov 01 20:32:29.925064 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:32:29.925046 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd9263fae24a75ce31837067678f2dd05979705ba5917ddf8195e2abaa0c3e3e\": container with ID starting with fd9263fae24a75ce31837067678f2dd05979705ba5917ddf8195e2abaa0c3e3e not found: ID does not exist" containerID="fd9263fae24a75ce31837067678f2dd05979705ba5917ddf8195e2abaa0c3e3e" Nov 01 20:32:29.925129 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.925069 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd9263fae24a75ce31837067678f2dd05979705ba5917ddf8195e2abaa0c3e3e"} err="failed to get container status \"fd9263fae24a75ce31837067678f2dd05979705ba5917ddf8195e2abaa0c3e3e\": rpc error: code = NotFound desc = could not find container \"fd9263fae24a75ce31837067678f2dd05979705ba5917ddf8195e2abaa0c3e3e\": container with ID starting with fd9263fae24a75ce31837067678f2dd05979705ba5917ddf8195e2abaa0c3e3e not found: ID does not exist" Nov 01 20:32:29.925129 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.925079 4484 scope.go:122] "RemoveContainer" containerID="4d400d325b72cea886c224352b3e0ac77faa4d54d0230195bb4a8565956dd020" Nov 01 20:32:29.925296 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.925280 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d400d325b72cea886c224352b3e0ac77faa4d54d0230195bb4a8565956dd020"} err="failed to get container status \"4d400d325b72cea886c224352b3e0ac77faa4d54d0230195bb4a8565956dd020\": rpc error: code = NotFound desc = could not find container \"4d400d325b72cea886c224352b3e0ac77faa4d54d0230195bb4a8565956dd020\": container with ID starting with 4d400d325b72cea886c224352b3e0ac77faa4d54d0230195bb4a8565956dd020 not found: ID does not exist" Nov 01 20:32:29.925296 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.925296 4484 scope.go:122] "RemoveContainer" containerID="761987ac7dee4c796185aa989dd1dcdf41a13667a650644cb9e7a643601e12c1" Nov 01 20:32:29.925503 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.925489 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"761987ac7dee4c796185aa989dd1dcdf41a13667a650644cb9e7a643601e12c1"} err="failed to get container status \"761987ac7dee4c796185aa989dd1dcdf41a13667a650644cb9e7a643601e12c1\": rpc error: code = NotFound desc = could not find container \"761987ac7dee4c796185aa989dd1dcdf41a13667a650644cb9e7a643601e12c1\": container with ID starting with 761987ac7dee4c796185aa989dd1dcdf41a13667a650644cb9e7a643601e12c1 not found: ID does not exist" Nov 01 20:32:29.925542 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.925503 4484 scope.go:122] "RemoveContainer" containerID="3b2ec0b592dd5236a454da217a88c714374c7bca27ee4fa2f0274fb1eb70a68a" Nov 01 20:32:29.925723 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.925703 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b2ec0b592dd5236a454da217a88c714374c7bca27ee4fa2f0274fb1eb70a68a"} err="failed to get container status \"3b2ec0b592dd5236a454da217a88c714374c7bca27ee4fa2f0274fb1eb70a68a\": rpc error: code = NotFound desc = could not find container \"3b2ec0b592dd5236a454da217a88c714374c7bca27ee4fa2f0274fb1eb70a68a\": container with ID starting with 3b2ec0b592dd5236a454da217a88c714374c7bca27ee4fa2f0274fb1eb70a68a not found: ID does not exist" Nov 01 20:32:29.925766 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.925724 4484 scope.go:122] "RemoveContainer" containerID="12af7378aec2923ddae356156bc6b219d813668b82743a6de54b104072562f17" Nov 01 20:32:29.925961 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.925945 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12af7378aec2923ddae356156bc6b219d813668b82743a6de54b104072562f17"} err="failed to get container status \"12af7378aec2923ddae356156bc6b219d813668b82743a6de54b104072562f17\": rpc error: code = NotFound desc = could not find container \"12af7378aec2923ddae356156bc6b219d813668b82743a6de54b104072562f17\": container with ID starting with 12af7378aec2923ddae356156bc6b219d813668b82743a6de54b104072562f17 not found: ID does not exist" Nov 01 20:32:29.926027 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.925961 4484 scope.go:122] "RemoveContainer" containerID="f21818b0e5208bb61b0859d1d0e7defaf5f31b66f09234809f226959427ace1a" Nov 01 20:32:29.926162 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.926147 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f21818b0e5208bb61b0859d1d0e7defaf5f31b66f09234809f226959427ace1a"} err="failed to get container status \"f21818b0e5208bb61b0859d1d0e7defaf5f31b66f09234809f226959427ace1a\": rpc error: code = NotFound desc = could not find container \"f21818b0e5208bb61b0859d1d0e7defaf5f31b66f09234809f226959427ace1a\": container with ID starting with f21818b0e5208bb61b0859d1d0e7defaf5f31b66f09234809f226959427ace1a not found: ID does not exist" Nov 01 20:32:29.926217 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.926162 4484 scope.go:122] "RemoveContainer" containerID="8daec41b36d5a569ffc2b6bb641246bda6d21f82e1006a18e55fda5a9133ef21" Nov 01 20:32:29.926367 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.926346 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8daec41b36d5a569ffc2b6bb641246bda6d21f82e1006a18e55fda5a9133ef21"} err="failed to get container status \"8daec41b36d5a569ffc2b6bb641246bda6d21f82e1006a18e55fda5a9133ef21\": rpc error: code = NotFound desc = could not find container \"8daec41b36d5a569ffc2b6bb641246bda6d21f82e1006a18e55fda5a9133ef21\": container with ID starting with 8daec41b36d5a569ffc2b6bb641246bda6d21f82e1006a18e55fda5a9133ef21 not found: ID does not exist" Nov 01 20:32:29.926367 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.926361 4484 scope.go:122] "RemoveContainer" containerID="fd9263fae24a75ce31837067678f2dd05979705ba5917ddf8195e2abaa0c3e3e" Nov 01 20:32:29.926545 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.926530 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd9263fae24a75ce31837067678f2dd05979705ba5917ddf8195e2abaa0c3e3e"} err="failed to get container status \"fd9263fae24a75ce31837067678f2dd05979705ba5917ddf8195e2abaa0c3e3e\": rpc error: code = NotFound desc = could not find container \"fd9263fae24a75ce31837067678f2dd05979705ba5917ddf8195e2abaa0c3e3e\": container with ID starting with fd9263fae24a75ce31837067678f2dd05979705ba5917ddf8195e2abaa0c3e3e not found: ID does not exist" Nov 01 20:32:29.926580 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.926545 4484 scope.go:122] "RemoveContainer" containerID="4d400d325b72cea886c224352b3e0ac77faa4d54d0230195bb4a8565956dd020" Nov 01 20:32:29.926734 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.926720 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d400d325b72cea886c224352b3e0ac77faa4d54d0230195bb4a8565956dd020"} err="failed to get container status \"4d400d325b72cea886c224352b3e0ac77faa4d54d0230195bb4a8565956dd020\": rpc error: code = NotFound desc = could not find container \"4d400d325b72cea886c224352b3e0ac77faa4d54d0230195bb4a8565956dd020\": container with ID starting with 4d400d325b72cea886c224352b3e0ac77faa4d54d0230195bb4a8565956dd020 not found: ID does not exist" Nov 01 20:32:29.926783 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.926736 4484 scope.go:122] "RemoveContainer" containerID="761987ac7dee4c796185aa989dd1dcdf41a13667a650644cb9e7a643601e12c1" Nov 01 20:32:29.926941 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.926922 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"761987ac7dee4c796185aa989dd1dcdf41a13667a650644cb9e7a643601e12c1"} err="failed to get container status \"761987ac7dee4c796185aa989dd1dcdf41a13667a650644cb9e7a643601e12c1\": rpc error: code = NotFound desc = could not find container \"761987ac7dee4c796185aa989dd1dcdf41a13667a650644cb9e7a643601e12c1\": container with ID starting with 761987ac7dee4c796185aa989dd1dcdf41a13667a650644cb9e7a643601e12c1 not found: ID does not exist" Nov 01 20:32:29.927009 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.926941 4484 scope.go:122] "RemoveContainer" containerID="3b2ec0b592dd5236a454da217a88c714374c7bca27ee4fa2f0274fb1eb70a68a" Nov 01 20:32:29.927140 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.927125 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b2ec0b592dd5236a454da217a88c714374c7bca27ee4fa2f0274fb1eb70a68a"} err="failed to get container status \"3b2ec0b592dd5236a454da217a88c714374c7bca27ee4fa2f0274fb1eb70a68a\": rpc error: code = NotFound desc = could not find container \"3b2ec0b592dd5236a454da217a88c714374c7bca27ee4fa2f0274fb1eb70a68a\": container with ID starting with 3b2ec0b592dd5236a454da217a88c714374c7bca27ee4fa2f0274fb1eb70a68a not found: ID does not exist" Nov 01 20:32:29.927179 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.927140 4484 scope.go:122] "RemoveContainer" containerID="12af7378aec2923ddae356156bc6b219d813668b82743a6de54b104072562f17" Nov 01 20:32:29.927372 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.927357 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12af7378aec2923ddae356156bc6b219d813668b82743a6de54b104072562f17"} err="failed to get container status \"12af7378aec2923ddae356156bc6b219d813668b82743a6de54b104072562f17\": rpc error: code = NotFound desc = could not find container \"12af7378aec2923ddae356156bc6b219d813668b82743a6de54b104072562f17\": container with ID starting with 12af7378aec2923ddae356156bc6b219d813668b82743a6de54b104072562f17 not found: ID does not exist" Nov 01 20:32:29.927427 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.927372 4484 scope.go:122] "RemoveContainer" containerID="f21818b0e5208bb61b0859d1d0e7defaf5f31b66f09234809f226959427ace1a" Nov 01 20:32:29.927566 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.927545 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f21818b0e5208bb61b0859d1d0e7defaf5f31b66f09234809f226959427ace1a"} err="failed to get container status \"f21818b0e5208bb61b0859d1d0e7defaf5f31b66f09234809f226959427ace1a\": rpc error: code = NotFound desc = could not find container \"f21818b0e5208bb61b0859d1d0e7defaf5f31b66f09234809f226959427ace1a\": container with ID starting with f21818b0e5208bb61b0859d1d0e7defaf5f31b66f09234809f226959427ace1a not found: ID does not exist" Nov 01 20:32:29.927624 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.927566 4484 scope.go:122] "RemoveContainer" containerID="8daec41b36d5a569ffc2b6bb641246bda6d21f82e1006a18e55fda5a9133ef21" Nov 01 20:32:29.927826 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.927808 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8daec41b36d5a569ffc2b6bb641246bda6d21f82e1006a18e55fda5a9133ef21"} err="failed to get container status \"8daec41b36d5a569ffc2b6bb641246bda6d21f82e1006a18e55fda5a9133ef21\": rpc error: code = NotFound desc = could not find container \"8daec41b36d5a569ffc2b6bb641246bda6d21f82e1006a18e55fda5a9133ef21\": container with ID starting with 8daec41b36d5a569ffc2b6bb641246bda6d21f82e1006a18e55fda5a9133ef21 not found: ID does not exist" Nov 01 20:32:29.927870 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.927827 4484 scope.go:122] "RemoveContainer" containerID="fd9263fae24a75ce31837067678f2dd05979705ba5917ddf8195e2abaa0c3e3e" Nov 01 20:32:29.928111 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.928091 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd9263fae24a75ce31837067678f2dd05979705ba5917ddf8195e2abaa0c3e3e"} err="failed to get container status \"fd9263fae24a75ce31837067678f2dd05979705ba5917ddf8195e2abaa0c3e3e\": rpc error: code = NotFound desc = could not find container \"fd9263fae24a75ce31837067678f2dd05979705ba5917ddf8195e2abaa0c3e3e\": container with ID starting with fd9263fae24a75ce31837067678f2dd05979705ba5917ddf8195e2abaa0c3e3e not found: ID does not exist" Nov 01 20:32:29.928111 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.928109 4484 scope.go:122] "RemoveContainer" containerID="4d400d325b72cea886c224352b3e0ac77faa4d54d0230195bb4a8565956dd020" Nov 01 20:32:29.928378 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.928356 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d400d325b72cea886c224352b3e0ac77faa4d54d0230195bb4a8565956dd020"} err="failed to get container status \"4d400d325b72cea886c224352b3e0ac77faa4d54d0230195bb4a8565956dd020\": rpc error: code = NotFound desc = could not find container \"4d400d325b72cea886c224352b3e0ac77faa4d54d0230195bb4a8565956dd020\": container with ID starting with 4d400d325b72cea886c224352b3e0ac77faa4d54d0230195bb4a8565956dd020 not found: ID does not exist" Nov 01 20:32:29.928470 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.928377 4484 scope.go:122] "RemoveContainer" containerID="761987ac7dee4c796185aa989dd1dcdf41a13667a650644cb9e7a643601e12c1" Nov 01 20:32:29.928644 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.928628 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"761987ac7dee4c796185aa989dd1dcdf41a13667a650644cb9e7a643601e12c1"} err="failed to get container status \"761987ac7dee4c796185aa989dd1dcdf41a13667a650644cb9e7a643601e12c1\": rpc error: code = NotFound desc = could not find container \"761987ac7dee4c796185aa989dd1dcdf41a13667a650644cb9e7a643601e12c1\": container with ID starting with 761987ac7dee4c796185aa989dd1dcdf41a13667a650644cb9e7a643601e12c1 not found: ID does not exist" Nov 01 20:32:29.928713 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.928646 4484 scope.go:122] "RemoveContainer" containerID="3b2ec0b592dd5236a454da217a88c714374c7bca27ee4fa2f0274fb1eb70a68a" Nov 01 20:32:29.928942 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.928923 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b2ec0b592dd5236a454da217a88c714374c7bca27ee4fa2f0274fb1eb70a68a"} err="failed to get container status \"3b2ec0b592dd5236a454da217a88c714374c7bca27ee4fa2f0274fb1eb70a68a\": rpc error: code = NotFound desc = could not find container \"3b2ec0b592dd5236a454da217a88c714374c7bca27ee4fa2f0274fb1eb70a68a\": container with ID starting with 3b2ec0b592dd5236a454da217a88c714374c7bca27ee4fa2f0274fb1eb70a68a not found: ID does not exist" Nov 01 20:32:29.929017 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.928943 4484 scope.go:122] "RemoveContainer" containerID="12af7378aec2923ddae356156bc6b219d813668b82743a6de54b104072562f17" Nov 01 20:32:29.929300 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.929280 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12af7378aec2923ddae356156bc6b219d813668b82743a6de54b104072562f17"} err="failed to get container status \"12af7378aec2923ddae356156bc6b219d813668b82743a6de54b104072562f17\": rpc error: code = NotFound desc = could not find container \"12af7378aec2923ddae356156bc6b219d813668b82743a6de54b104072562f17\": container with ID starting with 12af7378aec2923ddae356156bc6b219d813668b82743a6de54b104072562f17 not found: ID does not exist" Nov 01 20:32:29.929376 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.929302 4484 scope.go:122] "RemoveContainer" containerID="f21818b0e5208bb61b0859d1d0e7defaf5f31b66f09234809f226959427ace1a" Nov 01 20:32:29.929593 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.929571 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f21818b0e5208bb61b0859d1d0e7defaf5f31b66f09234809f226959427ace1a"} err="failed to get container status \"f21818b0e5208bb61b0859d1d0e7defaf5f31b66f09234809f226959427ace1a\": rpc error: code = NotFound desc = could not find container \"f21818b0e5208bb61b0859d1d0e7defaf5f31b66f09234809f226959427ace1a\": container with ID starting with f21818b0e5208bb61b0859d1d0e7defaf5f31b66f09234809f226959427ace1a not found: ID does not exist" Nov 01 20:32:29.929660 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.929593 4484 scope.go:122] "RemoveContainer" containerID="8daec41b36d5a569ffc2b6bb641246bda6d21f82e1006a18e55fda5a9133ef21" Nov 01 20:32:29.929857 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.929833 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8daec41b36d5a569ffc2b6bb641246bda6d21f82e1006a18e55fda5a9133ef21"} err="failed to get container status \"8daec41b36d5a569ffc2b6bb641246bda6d21f82e1006a18e55fda5a9133ef21\": rpc error: code = NotFound desc = could not find container \"8daec41b36d5a569ffc2b6bb641246bda6d21f82e1006a18e55fda5a9133ef21\": container with ID starting with 8daec41b36d5a569ffc2b6bb641246bda6d21f82e1006a18e55fda5a9133ef21 not found: ID does not exist" Nov 01 20:32:29.929916 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.929858 4484 scope.go:122] "RemoveContainer" containerID="fd9263fae24a75ce31837067678f2dd05979705ba5917ddf8195e2abaa0c3e3e" Nov 01 20:32:29.930154 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.930134 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd9263fae24a75ce31837067678f2dd05979705ba5917ddf8195e2abaa0c3e3e"} err="failed to get container status \"fd9263fae24a75ce31837067678f2dd05979705ba5917ddf8195e2abaa0c3e3e\": rpc error: code = NotFound desc = could not find container \"fd9263fae24a75ce31837067678f2dd05979705ba5917ddf8195e2abaa0c3e3e\": container with ID starting with fd9263fae24a75ce31837067678f2dd05979705ba5917ddf8195e2abaa0c3e3e not found: ID does not exist" Nov 01 20:32:29.930154 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.930154 4484 scope.go:122] "RemoveContainer" containerID="4d400d325b72cea886c224352b3e0ac77faa4d54d0230195bb4a8565956dd020" Nov 01 20:32:29.930411 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.930392 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d400d325b72cea886c224352b3e0ac77faa4d54d0230195bb4a8565956dd020"} err="failed to get container status \"4d400d325b72cea886c224352b3e0ac77faa4d54d0230195bb4a8565956dd020\": rpc error: code = NotFound desc = could not find container \"4d400d325b72cea886c224352b3e0ac77faa4d54d0230195bb4a8565956dd020\": container with ID starting with 4d400d325b72cea886c224352b3e0ac77faa4d54d0230195bb4a8565956dd020 not found: ID does not exist" Nov 01 20:32:29.930411 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.930410 4484 scope.go:122] "RemoveContainer" containerID="761987ac7dee4c796185aa989dd1dcdf41a13667a650644cb9e7a643601e12c1" Nov 01 20:32:29.930654 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.930636 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"761987ac7dee4c796185aa989dd1dcdf41a13667a650644cb9e7a643601e12c1"} err="failed to get container status \"761987ac7dee4c796185aa989dd1dcdf41a13667a650644cb9e7a643601e12c1\": rpc error: code = NotFound desc = could not find container \"761987ac7dee4c796185aa989dd1dcdf41a13667a650644cb9e7a643601e12c1\": container with ID starting with 761987ac7dee4c796185aa989dd1dcdf41a13667a650644cb9e7a643601e12c1 not found: ID does not exist" Nov 01 20:32:29.930654 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.930652 4484 scope.go:122] "RemoveContainer" containerID="3b2ec0b592dd5236a454da217a88c714374c7bca27ee4fa2f0274fb1eb70a68a" Nov 01 20:32:29.930899 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.930878 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b2ec0b592dd5236a454da217a88c714374c7bca27ee4fa2f0274fb1eb70a68a"} err="failed to get container status \"3b2ec0b592dd5236a454da217a88c714374c7bca27ee4fa2f0274fb1eb70a68a\": rpc error: code = NotFound desc = could not find container \"3b2ec0b592dd5236a454da217a88c714374c7bca27ee4fa2f0274fb1eb70a68a\": container with ID starting with 3b2ec0b592dd5236a454da217a88c714374c7bca27ee4fa2f0274fb1eb70a68a not found: ID does not exist" Nov 01 20:32:29.930952 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.930901 4484 scope.go:122] "RemoveContainer" containerID="12af7378aec2923ddae356156bc6b219d813668b82743a6de54b104072562f17" Nov 01 20:32:29.931177 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.931150 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12af7378aec2923ddae356156bc6b219d813668b82743a6de54b104072562f17"} err="failed to get container status \"12af7378aec2923ddae356156bc6b219d813668b82743a6de54b104072562f17\": rpc error: code = NotFound desc = could not find container \"12af7378aec2923ddae356156bc6b219d813668b82743a6de54b104072562f17\": container with ID starting with 12af7378aec2923ddae356156bc6b219d813668b82743a6de54b104072562f17 not found: ID does not exist" Nov 01 20:32:29.931260 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.931179 4484 scope.go:122] "RemoveContainer" containerID="f21818b0e5208bb61b0859d1d0e7defaf5f31b66f09234809f226959427ace1a" Nov 01 20:32:29.931417 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.931401 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f21818b0e5208bb61b0859d1d0e7defaf5f31b66f09234809f226959427ace1a"} err="failed to get container status \"f21818b0e5208bb61b0859d1d0e7defaf5f31b66f09234809f226959427ace1a\": rpc error: code = NotFound desc = could not find container \"f21818b0e5208bb61b0859d1d0e7defaf5f31b66f09234809f226959427ace1a\": container with ID starting with f21818b0e5208bb61b0859d1d0e7defaf5f31b66f09234809f226959427ace1a not found: ID does not exist" Nov 01 20:32:29.931464 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.931417 4484 scope.go:122] "RemoveContainer" containerID="8daec41b36d5a569ffc2b6bb641246bda6d21f82e1006a18e55fda5a9133ef21" Nov 01 20:32:29.931654 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.931633 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8daec41b36d5a569ffc2b6bb641246bda6d21f82e1006a18e55fda5a9133ef21"} err="failed to get container status \"8daec41b36d5a569ffc2b6bb641246bda6d21f82e1006a18e55fda5a9133ef21\": rpc error: code = NotFound desc = could not find container \"8daec41b36d5a569ffc2b6bb641246bda6d21f82e1006a18e55fda5a9133ef21\": container with ID starting with 8daec41b36d5a569ffc2b6bb641246bda6d21f82e1006a18e55fda5a9133ef21 not found: ID does not exist" Nov 01 20:32:29.931654 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.931652 4484 scope.go:122] "RemoveContainer" containerID="fd9263fae24a75ce31837067678f2dd05979705ba5917ddf8195e2abaa0c3e3e" Nov 01 20:32:29.931846 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.931830 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd9263fae24a75ce31837067678f2dd05979705ba5917ddf8195e2abaa0c3e3e"} err="failed to get container status \"fd9263fae24a75ce31837067678f2dd05979705ba5917ddf8195e2abaa0c3e3e\": rpc error: code = NotFound desc = could not find container \"fd9263fae24a75ce31837067678f2dd05979705ba5917ddf8195e2abaa0c3e3e\": container with ID starting with fd9263fae24a75ce31837067678f2dd05979705ba5917ddf8195e2abaa0c3e3e not found: ID does not exist" Nov 01 20:32:29.931846 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.931845 4484 scope.go:122] "RemoveContainer" containerID="4d400d325b72cea886c224352b3e0ac77faa4d54d0230195bb4a8565956dd020" Nov 01 20:32:29.932048 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.932031 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d400d325b72cea886c224352b3e0ac77faa4d54d0230195bb4a8565956dd020"} err="failed to get container status \"4d400d325b72cea886c224352b3e0ac77faa4d54d0230195bb4a8565956dd020\": rpc error: code = NotFound desc = could not find container \"4d400d325b72cea886c224352b3e0ac77faa4d54d0230195bb4a8565956dd020\": container with ID starting with 4d400d325b72cea886c224352b3e0ac77faa4d54d0230195bb4a8565956dd020 not found: ID does not exist" Nov 01 20:32:29.932112 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.932049 4484 scope.go:122] "RemoveContainer" containerID="761987ac7dee4c796185aa989dd1dcdf41a13667a650644cb9e7a643601e12c1" Nov 01 20:32:29.932258 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.932240 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"761987ac7dee4c796185aa989dd1dcdf41a13667a650644cb9e7a643601e12c1"} err="failed to get container status \"761987ac7dee4c796185aa989dd1dcdf41a13667a650644cb9e7a643601e12c1\": rpc error: code = NotFound desc = could not find container \"761987ac7dee4c796185aa989dd1dcdf41a13667a650644cb9e7a643601e12c1\": container with ID starting with 761987ac7dee4c796185aa989dd1dcdf41a13667a650644cb9e7a643601e12c1 not found: ID does not exist" Nov 01 20:32:29.932295 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.932258 4484 scope.go:122] "RemoveContainer" containerID="3b2ec0b592dd5236a454da217a88c714374c7bca27ee4fa2f0274fb1eb70a68a" Nov 01 20:32:29.932464 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.932445 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b2ec0b592dd5236a454da217a88c714374c7bca27ee4fa2f0274fb1eb70a68a"} err="failed to get container status \"3b2ec0b592dd5236a454da217a88c714374c7bca27ee4fa2f0274fb1eb70a68a\": rpc error: code = NotFound desc = could not find container \"3b2ec0b592dd5236a454da217a88c714374c7bca27ee4fa2f0274fb1eb70a68a\": container with ID starting with 3b2ec0b592dd5236a454da217a88c714374c7bca27ee4fa2f0274fb1eb70a68a not found: ID does not exist" Nov 01 20:32:29.932505 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.932464 4484 scope.go:122] "RemoveContainer" containerID="12af7378aec2923ddae356156bc6b219d813668b82743a6de54b104072562f17" Nov 01 20:32:29.932682 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.932666 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12af7378aec2923ddae356156bc6b219d813668b82743a6de54b104072562f17"} err="failed to get container status \"12af7378aec2923ddae356156bc6b219d813668b82743a6de54b104072562f17\": rpc error: code = NotFound desc = could not find container \"12af7378aec2923ddae356156bc6b219d813668b82743a6de54b104072562f17\": container with ID starting with 12af7378aec2923ddae356156bc6b219d813668b82743a6de54b104072562f17 not found: ID does not exist" Nov 01 20:32:29.932718 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.932682 4484 scope.go:122] "RemoveContainer" containerID="f21818b0e5208bb61b0859d1d0e7defaf5f31b66f09234809f226959427ace1a" Nov 01 20:32:29.932874 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.932859 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f21818b0e5208bb61b0859d1d0e7defaf5f31b66f09234809f226959427ace1a"} err="failed to get container status \"f21818b0e5208bb61b0859d1d0e7defaf5f31b66f09234809f226959427ace1a\": rpc error: code = NotFound desc = could not find container \"f21818b0e5208bb61b0859d1d0e7defaf5f31b66f09234809f226959427ace1a\": container with ID starting with f21818b0e5208bb61b0859d1d0e7defaf5f31b66f09234809f226959427ace1a not found: ID does not exist" Nov 01 20:32:29.932910 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.932875 4484 scope.go:122] "RemoveContainer" containerID="8daec41b36d5a569ffc2b6bb641246bda6d21f82e1006a18e55fda5a9133ef21" Nov 01 20:32:29.933084 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.933067 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8daec41b36d5a569ffc2b6bb641246bda6d21f82e1006a18e55fda5a9133ef21"} err="failed to get container status \"8daec41b36d5a569ffc2b6bb641246bda6d21f82e1006a18e55fda5a9133ef21\": rpc error: code = NotFound desc = could not find container \"8daec41b36d5a569ffc2b6bb641246bda6d21f82e1006a18e55fda5a9133ef21\": container with ID starting with 8daec41b36d5a569ffc2b6bb641246bda6d21f82e1006a18e55fda5a9133ef21 not found: ID does not exist" Nov 01 20:32:29.933124 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.933090 4484 scope.go:122] "RemoveContainer" containerID="fd9263fae24a75ce31837067678f2dd05979705ba5917ddf8195e2abaa0c3e3e" Nov 01 20:32:29.933289 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:29.933274 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd9263fae24a75ce31837067678f2dd05979705ba5917ddf8195e2abaa0c3e3e"} err="failed to get container status \"fd9263fae24a75ce31837067678f2dd05979705ba5917ddf8195e2abaa0c3e3e\": rpc error: code = NotFound desc = could not find container \"fd9263fae24a75ce31837067678f2dd05979705ba5917ddf8195e2abaa0c3e3e\": container with ID starting with fd9263fae24a75ce31837067678f2dd05979705ba5917ddf8195e2abaa0c3e3e not found: ID does not exist" Nov 01 20:32:30.029903 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.029866 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 20:32:30.080746 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.080715 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 20:32:30.142816 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.142730 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 20:32:30.143061 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.143049 4484 cpu_manager.go:414] "containerMap: removing container" podUID="04386401-6c41-42c3-ad9f-73702962c4a2" containerName="prometheus" Nov 01 20:32:30.143125 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.143062 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04386401-6c41-42c3-ad9f-73702962c4a2" containerName="prometheus" Nov 01 20:32:30.143125 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.143074 4484 cpu_manager.go:414] "containerMap: removing container" podUID="04386401-6c41-42c3-ad9f-73702962c4a2" containerName="thanos-sidecar" Nov 01 20:32:30.143125 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.143079 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04386401-6c41-42c3-ad9f-73702962c4a2" containerName="thanos-sidecar" Nov 01 20:32:30.143125 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.143095 4484 cpu_manager.go:414] "containerMap: removing container" podUID="04386401-6c41-42c3-ad9f-73702962c4a2" containerName="init-config-reloader" Nov 01 20:32:30.143125 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.143100 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04386401-6c41-42c3-ad9f-73702962c4a2" containerName="init-config-reloader" Nov 01 20:32:30.143125 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.143106 4484 cpu_manager.go:414] "containerMap: removing container" podUID="04386401-6c41-42c3-ad9f-73702962c4a2" containerName="kube-rbac-proxy" Nov 01 20:32:30.143125 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.143111 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04386401-6c41-42c3-ad9f-73702962c4a2" containerName="kube-rbac-proxy" Nov 01 20:32:30.143125 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.143120 4484 cpu_manager.go:414] "containerMap: removing container" podUID="04386401-6c41-42c3-ad9f-73702962c4a2" containerName="kube-rbac-proxy-web" Nov 01 20:32:30.143125 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.143125 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04386401-6c41-42c3-ad9f-73702962c4a2" containerName="kube-rbac-proxy-web" Nov 01 20:32:30.143397 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.143137 4484 cpu_manager.go:414] "containerMap: removing container" podUID="04386401-6c41-42c3-ad9f-73702962c4a2" containerName="config-reloader" Nov 01 20:32:30.143397 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.143142 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04386401-6c41-42c3-ad9f-73702962c4a2" containerName="config-reloader" Nov 01 20:32:30.143397 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.143148 4484 cpu_manager.go:414] "containerMap: removing container" podUID="04386401-6c41-42c3-ad9f-73702962c4a2" containerName="kube-rbac-proxy-thanos" Nov 01 20:32:30.143397 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.143152 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04386401-6c41-42c3-ad9f-73702962c4a2" containerName="kube-rbac-proxy-thanos" Nov 01 20:32:30.143397 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.143200 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="04386401-6c41-42c3-ad9f-73702962c4a2" containerName="kube-rbac-proxy" Nov 01 20:32:30.143397 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.143208 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="04386401-6c41-42c3-ad9f-73702962c4a2" containerName="kube-rbac-proxy-web" Nov 01 20:32:30.143397 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.143214 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="04386401-6c41-42c3-ad9f-73702962c4a2" containerName="config-reloader" Nov 01 20:32:30.143397 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.143219 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="04386401-6c41-42c3-ad9f-73702962c4a2" containerName="kube-rbac-proxy-thanos" Nov 01 20:32:30.143397 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.143226 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="04386401-6c41-42c3-ad9f-73702962c4a2" containerName="prometheus" Nov 01 20:32:30.143397 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.143232 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="04386401-6c41-42c3-ad9f-73702962c4a2" containerName="thanos-sidecar" Nov 01 20:32:30.146611 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.146592 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.201504 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.201466 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 01 20:32:30.201504 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.201494 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-2f24m\"" Nov 01 20:32:30.201504 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.201467 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 01 20:32:30.201813 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.201521 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 01 20:32:30.201813 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.201577 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 01 20:32:30.201813 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.201580 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 01 20:32:30.201813 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.201645 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 01 20:32:30.201813 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.201651 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 01 20:32:30.201813 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.201716 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 01 20:32:30.201813 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.201774 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 01 20:32:30.202179 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.201877 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 01 20:32:30.202232 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.202179 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 01 20:32:30.202289 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.202265 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1m0kfcr09p10u\"" Nov 01 20:32:30.202474 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.202455 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c6a249c8-153f-432a-8f00-aa7f196cb024-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c6a249c8-153f-432a-8f00-aa7f196cb024\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.202511 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.202486 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c6a249c8-153f-432a-8f00-aa7f196cb024-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c6a249c8-153f-432a-8f00-aa7f196cb024\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.202511 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.202504 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c6a249c8-153f-432a-8f00-aa7f196cb024-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c6a249c8-153f-432a-8f00-aa7f196cb024\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.202572 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.202520 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 01 20:32:30.202572 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.202519 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c6a249c8-153f-432a-8f00-aa7f196cb024-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c6a249c8-153f-432a-8f00-aa7f196cb024\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.202666 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.202651 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c6a249c8-153f-432a-8f00-aa7f196cb024-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c6a249c8-153f-432a-8f00-aa7f196cb024\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.202715 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.202702 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c6a249c8-153f-432a-8f00-aa7f196cb024-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c6a249c8-153f-432a-8f00-aa7f196cb024\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.202754 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.202723 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2b96\" (UniqueName: \"kubernetes.io/projected/c6a249c8-153f-432a-8f00-aa7f196cb024-kube-api-access-s2b96\") pod \"prometheus-k8s-0\" (UID: \"c6a249c8-153f-432a-8f00-aa7f196cb024\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.202754 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.202748 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c6a249c8-153f-432a-8f00-aa7f196cb024-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c6a249c8-153f-432a-8f00-aa7f196cb024\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.202815 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.202794 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c6a249c8-153f-432a-8f00-aa7f196cb024-config-out\") pod \"prometheus-k8s-0\" (UID: \"c6a249c8-153f-432a-8f00-aa7f196cb024\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.202845 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.202816 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c6a249c8-153f-432a-8f00-aa7f196cb024-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c6a249c8-153f-432a-8f00-aa7f196cb024\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.202878 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.202846 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c6a249c8-153f-432a-8f00-aa7f196cb024-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c6a249c8-153f-432a-8f00-aa7f196cb024\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.202919 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.202906 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c6a249c8-153f-432a-8f00-aa7f196cb024-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c6a249c8-153f-432a-8f00-aa7f196cb024\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.202981 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.202969 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6a249c8-153f-432a-8f00-aa7f196cb024-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c6a249c8-153f-432a-8f00-aa7f196cb024\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.203031 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.203022 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c6a249c8-153f-432a-8f00-aa7f196cb024-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c6a249c8-153f-432a-8f00-aa7f196cb024\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.203064 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.203045 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6a249c8-153f-432a-8f00-aa7f196cb024-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c6a249c8-153f-432a-8f00-aa7f196cb024\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.203098 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.203065 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c6a249c8-153f-432a-8f00-aa7f196cb024-web-config\") pod \"prometheus-k8s-0\" (UID: \"c6a249c8-153f-432a-8f00-aa7f196cb024\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.203098 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.203088 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c6a249c8-153f-432a-8f00-aa7f196cb024-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c6a249c8-153f-432a-8f00-aa7f196cb024\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.203187 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.203174 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6a249c8-153f-432a-8f00-aa7f196cb024-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c6a249c8-153f-432a-8f00-aa7f196cb024\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.203221 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.203198 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c6a249c8-153f-432a-8f00-aa7f196cb024-config\") pod \"prometheus-k8s-0\" (UID: \"c6a249c8-153f-432a-8f00-aa7f196cb024\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.203221 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.203212 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6a249c8-153f-432a-8f00-aa7f196cb024-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c6a249c8-153f-432a-8f00-aa7f196cb024\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.253397 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.253361 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 01 20:32:30.253539 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.253421 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 01 20:32:30.254672 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.254651 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 20:32:30.303816 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.303779 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c6a249c8-153f-432a-8f00-aa7f196cb024-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c6a249c8-153f-432a-8f00-aa7f196cb024\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.303816 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.303820 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6a249c8-153f-432a-8f00-aa7f196cb024-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c6a249c8-153f-432a-8f00-aa7f196cb024\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.304380 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.303848 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c6a249c8-153f-432a-8f00-aa7f196cb024-web-config\") pod \"prometheus-k8s-0\" (UID: \"c6a249c8-153f-432a-8f00-aa7f196cb024\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.304380 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.303868 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c6a249c8-153f-432a-8f00-aa7f196cb024-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c6a249c8-153f-432a-8f00-aa7f196cb024\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.304380 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.303910 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6a249c8-153f-432a-8f00-aa7f196cb024-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c6a249c8-153f-432a-8f00-aa7f196cb024\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.304380 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.303939 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c6a249c8-153f-432a-8f00-aa7f196cb024-config\") pod \"prometheus-k8s-0\" (UID: \"c6a249c8-153f-432a-8f00-aa7f196cb024\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.304380 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.303962 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6a249c8-153f-432a-8f00-aa7f196cb024-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c6a249c8-153f-432a-8f00-aa7f196cb024\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.304380 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.304040 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c6a249c8-153f-432a-8f00-aa7f196cb024-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c6a249c8-153f-432a-8f00-aa7f196cb024\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.304380 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.304064 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c6a249c8-153f-432a-8f00-aa7f196cb024-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c6a249c8-153f-432a-8f00-aa7f196cb024\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.304380 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.304079 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c6a249c8-153f-432a-8f00-aa7f196cb024-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c6a249c8-153f-432a-8f00-aa7f196cb024\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.304380 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.304097 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c6a249c8-153f-432a-8f00-aa7f196cb024-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c6a249c8-153f-432a-8f00-aa7f196cb024\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.304380 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.304127 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c6a249c8-153f-432a-8f00-aa7f196cb024-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c6a249c8-153f-432a-8f00-aa7f196cb024\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.304380 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.304148 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c6a249c8-153f-432a-8f00-aa7f196cb024-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c6a249c8-153f-432a-8f00-aa7f196cb024\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.304380 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.304168 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2b96\" (UniqueName: \"kubernetes.io/projected/c6a249c8-153f-432a-8f00-aa7f196cb024-kube-api-access-s2b96\") pod \"prometheus-k8s-0\" (UID: \"c6a249c8-153f-432a-8f00-aa7f196cb024\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.304380 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.304187 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c6a249c8-153f-432a-8f00-aa7f196cb024-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c6a249c8-153f-432a-8f00-aa7f196cb024\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.304380 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.304215 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c6a249c8-153f-432a-8f00-aa7f196cb024-config-out\") pod \"prometheus-k8s-0\" (UID: \"c6a249c8-153f-432a-8f00-aa7f196cb024\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.304380 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.304235 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c6a249c8-153f-432a-8f00-aa7f196cb024-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c6a249c8-153f-432a-8f00-aa7f196cb024\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.304380 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.304256 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c6a249c8-153f-432a-8f00-aa7f196cb024-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c6a249c8-153f-432a-8f00-aa7f196cb024\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.304380 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.304286 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c6a249c8-153f-432a-8f00-aa7f196cb024-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c6a249c8-153f-432a-8f00-aa7f196cb024\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.304380 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.304320 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6a249c8-153f-432a-8f00-aa7f196cb024-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c6a249c8-153f-432a-8f00-aa7f196cb024\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.305271 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.304692 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c6a249c8-153f-432a-8f00-aa7f196cb024-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c6a249c8-153f-432a-8f00-aa7f196cb024\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.305271 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.304813 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6a249c8-153f-432a-8f00-aa7f196cb024-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c6a249c8-153f-432a-8f00-aa7f196cb024\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.305271 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.304912 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6a249c8-153f-432a-8f00-aa7f196cb024-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c6a249c8-153f-432a-8f00-aa7f196cb024\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.305271 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.305237 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6a249c8-153f-432a-8f00-aa7f196cb024-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c6a249c8-153f-432a-8f00-aa7f196cb024\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.306122 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.305955 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c6a249c8-153f-432a-8f00-aa7f196cb024-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c6a249c8-153f-432a-8f00-aa7f196cb024\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.306122 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.306060 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c6a249c8-153f-432a-8f00-aa7f196cb024-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c6a249c8-153f-432a-8f00-aa7f196cb024\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.307297 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.307077 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6a249c8-153f-432a-8f00-aa7f196cb024-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c6a249c8-153f-432a-8f00-aa7f196cb024\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.307410 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.307395 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c6a249c8-153f-432a-8f00-aa7f196cb024-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c6a249c8-153f-432a-8f00-aa7f196cb024\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.308243 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.308188 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c6a249c8-153f-432a-8f00-aa7f196cb024-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c6a249c8-153f-432a-8f00-aa7f196cb024\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.309210 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.308936 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c6a249c8-153f-432a-8f00-aa7f196cb024-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c6a249c8-153f-432a-8f00-aa7f196cb024\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.309210 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.308938 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c6a249c8-153f-432a-8f00-aa7f196cb024-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c6a249c8-153f-432a-8f00-aa7f196cb024\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.309210 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.309115 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c6a249c8-153f-432a-8f00-aa7f196cb024-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c6a249c8-153f-432a-8f00-aa7f196cb024\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.309422 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.309281 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c6a249c8-153f-432a-8f00-aa7f196cb024-config-out\") pod \"prometheus-k8s-0\" (UID: \"c6a249c8-153f-432a-8f00-aa7f196cb024\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.309422 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.309318 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c6a249c8-153f-432a-8f00-aa7f196cb024-web-config\") pod \"prometheus-k8s-0\" (UID: \"c6a249c8-153f-432a-8f00-aa7f196cb024\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.309510 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.309415 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c6a249c8-153f-432a-8f00-aa7f196cb024-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c6a249c8-153f-432a-8f00-aa7f196cb024\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.309510 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.309492 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c6a249c8-153f-432a-8f00-aa7f196cb024-config\") pod \"prometheus-k8s-0\" (UID: \"c6a249c8-153f-432a-8f00-aa7f196cb024\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.309930 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.309907 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c6a249c8-153f-432a-8f00-aa7f196cb024-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c6a249c8-153f-432a-8f00-aa7f196cb024\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.310375 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.310357 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c6a249c8-153f-432a-8f00-aa7f196cb024-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c6a249c8-153f-432a-8f00-aa7f196cb024\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.310792 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.310773 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c6a249c8-153f-432a-8f00-aa7f196cb024-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c6a249c8-153f-432a-8f00-aa7f196cb024\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.363605 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.363577 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2b96\" (UniqueName: \"kubernetes.io/projected/c6a249c8-153f-432a-8f00-aa7f196cb024-kube-api-access-s2b96\") pod \"prometheus-k8s-0\" (UID: \"c6a249c8-153f-432a-8f00-aa7f196cb024\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.462882 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.462792 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:30.885230 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:30.885156 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 20:32:30.887369 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:32:30.887341 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6a249c8_153f_432a_8f00_aa7f196cb024.slice/crio-190758dd5e9bfa67a185567d89c74e5eb9dbd3208ae36f28ef779a5042c8672a WatchSource:0}: Error finding container 190758dd5e9bfa67a185567d89c74e5eb9dbd3208ae36f28ef779a5042c8672a: Status 404 returned error can't find the container with id 190758dd5e9bfa67a185567d89c74e5eb9dbd3208ae36f28ef779a5042c8672a Nov 01 20:32:31.368808 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:31.368770 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04386401-6c41-42c3-ad9f-73702962c4a2" path="/var/lib/kubelet/pods/04386401-6c41-42c3-ad9f-73702962c4a2/volumes" Nov 01 20:32:31.869476 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:31.869442 4484 generic.go:356] "Generic (PLEG): container finished" podID="c6a249c8-153f-432a-8f00-aa7f196cb024" containerID="35e265d614906aa173b86814896e0767e1e031cc1a24b376c6173bd1b8e29ef5" exitCode=0 Nov 01 20:32:31.869621 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:31.869528 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c6a249c8-153f-432a-8f00-aa7f196cb024","Type":"ContainerDied","Data":"35e265d614906aa173b86814896e0767e1e031cc1a24b376c6173bd1b8e29ef5"} Nov 01 20:32:31.869621 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:31.869598 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c6a249c8-153f-432a-8f00-aa7f196cb024","Type":"ContainerStarted","Data":"190758dd5e9bfa67a185567d89c74e5eb9dbd3208ae36f28ef779a5042c8672a"} Nov 01 20:32:32.390092 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:32.389947 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:32:32.543125 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:32.543081 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:32:32.877034 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:32.876987 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c6a249c8-153f-432a-8f00-aa7f196cb024","Type":"ContainerStarted","Data":"b3ae255b57552056d24a9aec3cf199e447b6c04e5a4a6418f0ac76fc95ec5611"} Nov 01 20:32:32.877278 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:32.877096 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c6a249c8-153f-432a-8f00-aa7f196cb024","Type":"ContainerStarted","Data":"51f84f13fc241515b4cd7e58a8cc48dcd73fac55802e95800191b1519c455131"} Nov 01 20:32:32.877278 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:32.877126 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c6a249c8-153f-432a-8f00-aa7f196cb024","Type":"ContainerStarted","Data":"afa00a6ba0eb412c4aa38f1061b3af5b9c9d78754590196f90c3a3e76a231a40"} Nov 01 20:32:32.877278 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:32.877153 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c6a249c8-153f-432a-8f00-aa7f196cb024","Type":"ContainerStarted","Data":"5b2473259698358802e0bffcd7387ff516a29b09690e137f0f8c63b159bb7ff1"} Nov 01 20:32:32.877278 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:32.877182 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c6a249c8-153f-432a-8f00-aa7f196cb024","Type":"ContainerStarted","Data":"bd02992a1c2a37ebdd07a2cf86e7f6736f40e947180fcff06476b769bbadf7bf"} Nov 01 20:32:32.877278 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:32.877204 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c6a249c8-153f-432a-8f00-aa7f196cb024","Type":"ContainerStarted","Data":"b2d2d34b502dcc237adcdd70847ffa49d1e82cf43aad202cd150d015e252f06b"} Nov 01 20:32:33.007392 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:33.007334 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.992128269 podStartE2EDuration="3.007317402s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.189133ms" isStatefulPod=false podCreationTimestamp="2026-11-01 20:32:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:32:33.005875406 +0000 UTC m=+103.783515554" watchObservedRunningTime="2026-11-01 20:32:33.007317402 +0000 UTC m=+103.784957537" Nov 01 20:32:33.404520 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:33.404488 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:32:35.462934 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:35.462898 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:32:44.382386 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:32:44.382355 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:32:45.765175 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:32:45.765134 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 20:33:15.764801 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:33:15.764713 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 20:33:30.463173 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:33:30.463135 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:33:30.479074 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:33:30.479050 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:33:31.101053 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:33:31.101015 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:33:33.781838 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:33:33.781793 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:33:45.764173 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:33:45.764131 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 20:34:08.139818 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:34:08.139781 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:34:08.356795 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:34:08.356763 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:34:09.422625 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:34:09.422588 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:34:09.582580 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:34:09.582545 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:34:10.309964 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:34:10.309934 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:34:10.971590 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:34:10.971558 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:34:15.765618 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:34:15.765585 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 20:34:15.819671 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:34:15.819638 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:34:15.980164 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:34:15.980132 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:34:16.895527 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:34:16.895492 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:34:17.139663 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:34:17.139634 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:34:17.294668 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:34:17.294632 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:34:20.789273 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:34:20.789238 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:34:34.082671 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:34:34.082580 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:34:43.389558 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:34:43.389470 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:34:43.543227 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:34:43.543193 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:34:45.690283 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:34:45.690252 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:34:45.764916 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:34:45.764873 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 20:34:59.217898 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:34:59.217858 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:34:59.386834 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:34:59.386795 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:34:59.926878 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:34:59.926842 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:00.154807 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:00.154772 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-zj4qp"] Nov 01 20:35:00.157027 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:00.156984 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-zj4qp" Nov 01 20:35:00.194902 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:00.194797 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:00.210339 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:00.210310 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 01 20:35:00.210532 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:00.210358 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 01 20:35:00.210532 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:00.210375 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 01 20:35:00.210532 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:00.210374 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 01 20:35:00.210532 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:00.210398 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 01 20:35:00.210532 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:00.210468 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-46fcz\"" Nov 01 20:35:00.289116 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:00.289083 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c184b8a4-87fe-4f39-a397-838e22ff40c6-tmp\") pod \"opendatahub-operator-controller-manager-bd887889c-zj4qp\" (UID: \"c184b8a4-87fe-4f39-a397-838e22ff40c6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-zj4qp" Nov 01 20:35:00.289621 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:00.289133 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c184b8a4-87fe-4f39-a397-838e22ff40c6-webhook-cert\") pod \"opendatahub-operator-controller-manager-bd887889c-zj4qp\" (UID: \"c184b8a4-87fe-4f39-a397-838e22ff40c6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-zj4qp" Nov 01 20:35:00.289621 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:00.289160 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c184b8a4-87fe-4f39-a397-838e22ff40c6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-bd887889c-zj4qp\" (UID: \"c184b8a4-87fe-4f39-a397-838e22ff40c6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-zj4qp" Nov 01 20:35:00.289621 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:00.289326 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flnnv\" (UniqueName: \"kubernetes.io/projected/c184b8a4-87fe-4f39-a397-838e22ff40c6-kube-api-access-flnnv\") pod \"opendatahub-operator-controller-manager-bd887889c-zj4qp\" (UID: \"c184b8a4-87fe-4f39-a397-838e22ff40c6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-zj4qp" Nov 01 20:35:00.289621 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:00.289422 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c184b8a4-87fe-4f39-a397-838e22ff40c6-metrics-cert\") pod \"opendatahub-operator-controller-manager-bd887889c-zj4qp\" (UID: \"c184b8a4-87fe-4f39-a397-838e22ff40c6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-zj4qp" Nov 01 20:35:00.289621 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:00.289478 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c184b8a4-87fe-4f39-a397-838e22ff40c6-manifests\") pod \"opendatahub-operator-controller-manager-bd887889c-zj4qp\" (UID: \"c184b8a4-87fe-4f39-a397-838e22ff40c6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-zj4qp" Nov 01 20:35:00.292349 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:00.292324 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-zj4qp"] Nov 01 20:35:00.359721 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:00.359690 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:00.390972 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:00.390930 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flnnv\" (UniqueName: \"kubernetes.io/projected/c184b8a4-87fe-4f39-a397-838e22ff40c6-kube-api-access-flnnv\") pod \"opendatahub-operator-controller-manager-bd887889c-zj4qp\" (UID: \"c184b8a4-87fe-4f39-a397-838e22ff40c6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-zj4qp" Nov 01 20:35:00.391150 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:00.391027 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c184b8a4-87fe-4f39-a397-838e22ff40c6-metrics-cert\") pod \"opendatahub-operator-controller-manager-bd887889c-zj4qp\" (UID: \"c184b8a4-87fe-4f39-a397-838e22ff40c6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-zj4qp" Nov 01 20:35:00.391150 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:00.391073 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c184b8a4-87fe-4f39-a397-838e22ff40c6-manifests\") pod \"opendatahub-operator-controller-manager-bd887889c-zj4qp\" (UID: \"c184b8a4-87fe-4f39-a397-838e22ff40c6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-zj4qp" Nov 01 20:35:00.391150 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:00.391123 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c184b8a4-87fe-4f39-a397-838e22ff40c6-tmp\") pod \"opendatahub-operator-controller-manager-bd887889c-zj4qp\" (UID: \"c184b8a4-87fe-4f39-a397-838e22ff40c6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-zj4qp" Nov 01 20:35:00.391314 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:00.391155 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c184b8a4-87fe-4f39-a397-838e22ff40c6-webhook-cert\") pod \"opendatahub-operator-controller-manager-bd887889c-zj4qp\" (UID: \"c184b8a4-87fe-4f39-a397-838e22ff40c6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-zj4qp" Nov 01 20:35:00.391314 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:00.391189 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c184b8a4-87fe-4f39-a397-838e22ff40c6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-bd887889c-zj4qp\" (UID: \"c184b8a4-87fe-4f39-a397-838e22ff40c6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-zj4qp" Nov 01 20:35:00.391564 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:00.391535 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c184b8a4-87fe-4f39-a397-838e22ff40c6-manifests\") pod \"opendatahub-operator-controller-manager-bd887889c-zj4qp\" (UID: \"c184b8a4-87fe-4f39-a397-838e22ff40c6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-zj4qp" Nov 01 20:35:00.393658 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:00.393633 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c184b8a4-87fe-4f39-a397-838e22ff40c6-tmp\") pod \"opendatahub-operator-controller-manager-bd887889c-zj4qp\" (UID: \"c184b8a4-87fe-4f39-a397-838e22ff40c6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-zj4qp" Nov 01 20:35:00.393769 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:00.393671 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c184b8a4-87fe-4f39-a397-838e22ff40c6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-bd887889c-zj4qp\" (UID: \"c184b8a4-87fe-4f39-a397-838e22ff40c6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-zj4qp" Nov 01 20:35:00.393769 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:00.393636 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c184b8a4-87fe-4f39-a397-838e22ff40c6-webhook-cert\") pod \"opendatahub-operator-controller-manager-bd887889c-zj4qp\" (UID: \"c184b8a4-87fe-4f39-a397-838e22ff40c6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-zj4qp" Nov 01 20:35:00.393769 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:00.393744 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c184b8a4-87fe-4f39-a397-838e22ff40c6-metrics-cert\") pod \"opendatahub-operator-controller-manager-bd887889c-zj4qp\" (UID: \"c184b8a4-87fe-4f39-a397-838e22ff40c6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-zj4qp" Nov 01 20:35:00.454486 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:00.454413 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flnnv\" (UniqueName: \"kubernetes.io/projected/c184b8a4-87fe-4f39-a397-838e22ff40c6-kube-api-access-flnnv\") pod \"opendatahub-operator-controller-manager-bd887889c-zj4qp\" (UID: \"c184b8a4-87fe-4f39-a397-838e22ff40c6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-zj4qp" Nov 01 20:35:00.469682 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:00.469655 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-zj4qp" Nov 01 20:35:00.571194 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:00.571156 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:00.952370 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:00.952348 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-zj4qp"] Nov 01 20:35:00.954725 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:35:00.954696 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc184b8a4_87fe_4f39_a397_838e22ff40c6.slice/crio-5af159e2f991cfcfac81271f74bfd33ee876e8b894818c26800cfbce09daa049 WatchSource:0}: Error finding container 5af159e2f991cfcfac81271f74bfd33ee876e8b894818c26800cfbce09daa049: Status 404 returned error can't find the container with id 5af159e2f991cfcfac81271f74bfd33ee876e8b894818c26800cfbce09daa049 Nov 01 20:35:01.220663 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:01.220584 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:01.395098 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:01.395065 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:01.407808 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:01.407779 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-zj4qp" event={"ID":"c184b8a4-87fe-4f39-a397-838e22ff40c6","Type":"ContainerStarted","Data":"5af159e2f991cfcfac81271f74bfd33ee876e8b894818c26800cfbce09daa049"} Nov 01 20:35:02.651704 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:02.651670 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:02.820271 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:02.820231 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:03.424284 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:03.424154 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:03.609163 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:03.609124 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:06.684029 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:06.683971 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:07.443619 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:07.443589 4484 generic.go:356] "Generic (PLEG): container finished" podID="c184b8a4-87fe-4f39-a397-838e22ff40c6" containerID="749122103e3139a8b61198a7ba9eaa1259fc41273cc8b48225e19dac01a6c206" exitCode=0 Nov 01 20:35:07.443789 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:07.443679 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-zj4qp" event={"ID":"c184b8a4-87fe-4f39-a397-838e22ff40c6","Type":"ContainerDied","Data":"749122103e3139a8b61198a7ba9eaa1259fc41273cc8b48225e19dac01a6c206"} Nov 01 20:35:08.449004 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:08.448970 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-zj4qp" event={"ID":"c184b8a4-87fe-4f39-a397-838e22ff40c6","Type":"ContainerStarted","Data":"96b59dd6a3fff9c9de86eeb8c4b85a4c1999922fb8030e6a68bd9a652a8cd80e"} Nov 01 20:35:08.449400 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:08.449094 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-zj4qp" Nov 01 20:35:08.682686 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:08.682578 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-zj4qp" podStartSLOduration=2.453976042 podStartE2EDuration="9.682561938s" totalImagesPullingTime="6.744218159s" totalInitContainerRuntime="484.367737ms" isStatefulPod=false podCreationTimestamp="2026-11-01 20:34:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:35:08.631525739 +0000 UTC m=+259.409165876" watchObservedRunningTime="2026-11-01 20:35:08.682561938 +0000 UTC m=+259.460202072" Nov 01 20:35:12.965237 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:12.965202 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:14.227277 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:14.227240 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:14.973107 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:14.973070 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:15.429322 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:15.429276 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:15.765708 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:35:15.765660 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 20:35:16.956973 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:16.956929 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:17.174058 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:17.174015 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:18.943371 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:18.943339 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:19.455540 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:19.455512 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd887889c-zj4qp" Nov 01 20:35:19.834686 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:19.834637 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:20.614299 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:20.614266 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:21.487158 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:21.487113 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:21.724840 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:21.724806 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:23.257986 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:23.257953 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-dt84g"] Nov 01 20:35:23.263716 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:23.263682 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-dt84g" Nov 01 20:35:23.373174 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:23.373142 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 01 20:35:23.373174 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:23.373156 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-89h6s\"" Nov 01 20:35:23.373174 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:23.373142 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 01 20:35:23.374383 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:23.374365 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-dt84g"] Nov 01 20:35:23.391403 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:23.391371 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc6sg\" (UniqueName: \"kubernetes.io/projected/3060c0d1-44f0-48ec-843f-d6e8c0833982-kube-api-access-jc6sg\") pod \"cert-manager-webhook-5dbbbf75cc-dt84g\" (UID: \"3060c0d1-44f0-48ec-843f-d6e8c0833982\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-dt84g" Nov 01 20:35:23.391550 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:23.391410 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3060c0d1-44f0-48ec-843f-d6e8c0833982-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-dt84g\" (UID: \"3060c0d1-44f0-48ec-843f-d6e8c0833982\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-dt84g" Nov 01 20:35:23.492765 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:23.492732 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc6sg\" (UniqueName: \"kubernetes.io/projected/3060c0d1-44f0-48ec-843f-d6e8c0833982-kube-api-access-jc6sg\") pod \"cert-manager-webhook-5dbbbf75cc-dt84g\" (UID: \"3060c0d1-44f0-48ec-843f-d6e8c0833982\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-dt84g" Nov 01 20:35:23.492765 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:23.492766 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3060c0d1-44f0-48ec-843f-d6e8c0833982-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-dt84g\" (UID: \"3060c0d1-44f0-48ec-843f-d6e8c0833982\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-dt84g" Nov 01 20:35:23.563818 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:23.563748 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc6sg\" (UniqueName: \"kubernetes.io/projected/3060c0d1-44f0-48ec-843f-d6e8c0833982-kube-api-access-jc6sg\") pod \"cert-manager-webhook-5dbbbf75cc-dt84g\" (UID: \"3060c0d1-44f0-48ec-843f-d6e8c0833982\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-dt84g" Nov 01 20:35:23.569650 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:23.569621 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3060c0d1-44f0-48ec-843f-d6e8c0833982-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-dt84g\" (UID: \"3060c0d1-44f0-48ec-843f-d6e8c0833982\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-dt84g" Nov 01 20:35:23.583218 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:23.583188 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-dt84g" Nov 01 20:35:23.954533 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:23.954503 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-dt84g"] Nov 01 20:35:23.956613 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:35:23.956583 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3060c0d1_44f0_48ec_843f_d6e8c0833982.slice/crio-da8e0d4449a8ccad6ba32781c41d1f212db35c8678d738b5fdb262986a6b8e95 WatchSource:0}: Error finding container da8e0d4449a8ccad6ba32781c41d1f212db35c8678d738b5fdb262986a6b8e95: Status 404 returned error can't find the container with id da8e0d4449a8ccad6ba32781c41d1f212db35c8678d738b5fdb262986a6b8e95 Nov 01 20:35:24.511387 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:24.511346 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-dt84g" event={"ID":"3060c0d1-44f0-48ec-843f-d6e8c0833982","Type":"ContainerStarted","Data":"da8e0d4449a8ccad6ba32781c41d1f212db35c8678d738b5fdb262986a6b8e95"} Nov 01 20:35:25.425308 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:25.425271 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:25.845766 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:25.845731 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:27.861810 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:27.861778 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:28.170627 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:28.170550 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:28.304553 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:28.304514 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-r4f5w"] Nov 01 20:35:28.306850 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:28.306821 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-r4f5w" Nov 01 20:35:28.411206 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:28.411170 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-jbrk4\"" Nov 01 20:35:28.412626 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:28.412602 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-r4f5w"] Nov 01 20:35:28.435924 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:28.435854 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4782e19c-fbc6-407e-bdde-22bb9e4f239a-bound-sa-token\") pod \"cert-manager-597b95cb84-r4f5w\" (UID: \"4782e19c-fbc6-407e-bdde-22bb9e4f239a\") " pod="cert-manager/cert-manager-597b95cb84-r4f5w" Nov 01 20:35:28.436090 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:28.436022 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqgfp\" (UniqueName: \"kubernetes.io/projected/4782e19c-fbc6-407e-bdde-22bb9e4f239a-kube-api-access-pqgfp\") pod \"cert-manager-597b95cb84-r4f5w\" (UID: \"4782e19c-fbc6-407e-bdde-22bb9e4f239a\") " pod="cert-manager/cert-manager-597b95cb84-r4f5w" Nov 01 20:35:28.536606 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:28.536574 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqgfp\" (UniqueName: \"kubernetes.io/projected/4782e19c-fbc6-407e-bdde-22bb9e4f239a-kube-api-access-pqgfp\") pod \"cert-manager-597b95cb84-r4f5w\" (UID: \"4782e19c-fbc6-407e-bdde-22bb9e4f239a\") " pod="cert-manager/cert-manager-597b95cb84-r4f5w" Nov 01 20:35:28.536793 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:28.536617 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4782e19c-fbc6-407e-bdde-22bb9e4f239a-bound-sa-token\") pod \"cert-manager-597b95cb84-r4f5w\" (UID: \"4782e19c-fbc6-407e-bdde-22bb9e4f239a\") " pod="cert-manager/cert-manager-597b95cb84-r4f5w" Nov 01 20:35:28.602839 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:28.602809 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqgfp\" (UniqueName: \"kubernetes.io/projected/4782e19c-fbc6-407e-bdde-22bb9e4f239a-kube-api-access-pqgfp\") pod \"cert-manager-597b95cb84-r4f5w\" (UID: \"4782e19c-fbc6-407e-bdde-22bb9e4f239a\") " pod="cert-manager/cert-manager-597b95cb84-r4f5w" Nov 01 20:35:28.603109 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:28.603090 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4782e19c-fbc6-407e-bdde-22bb9e4f239a-bound-sa-token\") pod \"cert-manager-597b95cb84-r4f5w\" (UID: \"4782e19c-fbc6-407e-bdde-22bb9e4f239a\") " pod="cert-manager/cert-manager-597b95cb84-r4f5w" Nov 01 20:35:28.619516 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:28.619492 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-r4f5w" Nov 01 20:35:28.820689 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:28.820652 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:28.994161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:28.994135 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-r4f5w"] Nov 01 20:35:28.995974 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:35:28.995943 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4782e19c_fbc6_407e_bdde_22bb9e4f239a.slice/crio-e3d4866119bb131db5fd0c256d90a37bb0af3bb6e7227aac2d8a88241b7cdde1 WatchSource:0}: Error finding container e3d4866119bb131db5fd0c256d90a37bb0af3bb6e7227aac2d8a88241b7cdde1: Status 404 returned error can't find the container with id e3d4866119bb131db5fd0c256d90a37bb0af3bb6e7227aac2d8a88241b7cdde1 Nov 01 20:35:29.109792 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:29.109696 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:29.277628 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:29.277594 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:29.534355 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:29.534317 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-dt84g" event={"ID":"3060c0d1-44f0-48ec-843f-d6e8c0833982","Type":"ContainerStarted","Data":"f8624d7352230d3d47005976730d29f729f9f1ae5822b5c0e2fc18e77e85430f"} Nov 01 20:35:29.534553 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:29.534448 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-dt84g" Nov 01 20:35:29.536042 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:29.536012 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-r4f5w" event={"ID":"4782e19c-fbc6-407e-bdde-22bb9e4f239a","Type":"ContainerStarted","Data":"8aaa46a97e75547686eebbc31625f92145357f4901540339d93fcdaecc0a36d7"} Nov 01 20:35:29.536042 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:29.536045 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-r4f5w" event={"ID":"4782e19c-fbc6-407e-bdde-22bb9e4f239a","Type":"ContainerStarted","Data":"e3d4866119bb131db5fd0c256d90a37bb0af3bb6e7227aac2d8a88241b7cdde1"} Nov 01 20:35:29.649399 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:29.649351 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-dt84g" podStartSLOduration=2.060665516 podStartE2EDuration="6.649339085s" totalImagesPullingTime="4.588673569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:35:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:35:29.648546798 +0000 UTC m=+280.426186934" watchObservedRunningTime="2026-11-01 20:35:29.649339085 +0000 UTC m=+280.426979220" Nov 01 20:35:29.776468 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:29.776417 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-r4f5w" podStartSLOduration=1.776401605 podStartE2EDuration="1.776401605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:35:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:35:29.775527405 +0000 UTC m=+280.553167541" watchObservedRunningTime="2026-11-01 20:35:29.776401605 +0000 UTC m=+280.554041734" Nov 01 20:35:29.932375 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:29.932286 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:30.174909 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:30.174870 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:30.353147 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:30.353108 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:30.779748 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:30.779710 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:32.265760 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:32.265693 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:32.306449 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:32.306420 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-85b9bf4598-mg8rp"] Nov 01 20:35:32.309741 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:32.309722 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-85b9bf4598-mg8rp" Nov 01 20:35:32.368370 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:32.368334 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt7wk\" (UniqueName: \"kubernetes.io/projected/b2993469-8d17-47c8-8fd3-e298c468b4a9-kube-api-access-vt7wk\") pod \"kube-auth-proxy-85b9bf4598-mg8rp\" (UID: \"b2993469-8d17-47c8-8fd3-e298c468b4a9\") " pod="openshift-ingress/kube-auth-proxy-85b9bf4598-mg8rp" Nov 01 20:35:32.368553 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:32.368441 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2993469-8d17-47c8-8fd3-e298c468b4a9-tls-certs\") pod \"kube-auth-proxy-85b9bf4598-mg8rp\" (UID: \"b2993469-8d17-47c8-8fd3-e298c468b4a9\") " pod="openshift-ingress/kube-auth-proxy-85b9bf4598-mg8rp" Nov 01 20:35:32.368553 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:32.368477 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2993469-8d17-47c8-8fd3-e298c468b4a9-tmp\") pod \"kube-auth-proxy-85b9bf4598-mg8rp\" (UID: \"b2993469-8d17-47c8-8fd3-e298c468b4a9\") " pod="openshift-ingress/kube-auth-proxy-85b9bf4598-mg8rp" Nov 01 20:35:32.418688 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:32.418638 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 01 20:35:32.418871 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:32.418692 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 01 20:35:32.418871 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:32.418708 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 01 20:35:32.418871 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:32.418746 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-hrkrz\"" Nov 01 20:35:32.419015 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:32.418954 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 01 20:35:32.419966 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:32.419946 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-85b9bf4598-mg8rp"] Nov 01 20:35:32.469134 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:32.469090 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2993469-8d17-47c8-8fd3-e298c468b4a9-tls-certs\") pod \"kube-auth-proxy-85b9bf4598-mg8rp\" (UID: \"b2993469-8d17-47c8-8fd3-e298c468b4a9\") " pod="openshift-ingress/kube-auth-proxy-85b9bf4598-mg8rp" Nov 01 20:35:32.469134 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:32.469142 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2993469-8d17-47c8-8fd3-e298c468b4a9-tmp\") pod \"kube-auth-proxy-85b9bf4598-mg8rp\" (UID: \"b2993469-8d17-47c8-8fd3-e298c468b4a9\") " pod="openshift-ingress/kube-auth-proxy-85b9bf4598-mg8rp" Nov 01 20:35:32.469407 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:32.469194 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt7wk\" (UniqueName: \"kubernetes.io/projected/b2993469-8d17-47c8-8fd3-e298c468b4a9-kube-api-access-vt7wk\") pod \"kube-auth-proxy-85b9bf4598-mg8rp\" (UID: \"b2993469-8d17-47c8-8fd3-e298c468b4a9\") " pod="openshift-ingress/kube-auth-proxy-85b9bf4598-mg8rp" Nov 01 20:35:32.471463 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:32.471438 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2993469-8d17-47c8-8fd3-e298c468b4a9-tmp\") pod \"kube-auth-proxy-85b9bf4598-mg8rp\" (UID: \"b2993469-8d17-47c8-8fd3-e298c468b4a9\") " pod="openshift-ingress/kube-auth-proxy-85b9bf4598-mg8rp" Nov 01 20:35:32.471814 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:32.471794 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2993469-8d17-47c8-8fd3-e298c468b4a9-tls-certs\") pod \"kube-auth-proxy-85b9bf4598-mg8rp\" (UID: \"b2993469-8d17-47c8-8fd3-e298c468b4a9\") " pod="openshift-ingress/kube-auth-proxy-85b9bf4598-mg8rp" Nov 01 20:35:32.479419 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:32.479393 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:32.532587 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:32.532519 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt7wk\" (UniqueName: \"kubernetes.io/projected/b2993469-8d17-47c8-8fd3-e298c468b4a9-kube-api-access-vt7wk\") pod \"kube-auth-proxy-85b9bf4598-mg8rp\" (UID: \"b2993469-8d17-47c8-8fd3-e298c468b4a9\") " pod="openshift-ingress/kube-auth-proxy-85b9bf4598-mg8rp" Nov 01 20:35:32.622291 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:32.622252 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-85b9bf4598-mg8rp" Nov 01 20:35:32.789638 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:32.789554 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:32.995223 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:32.995199 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-85b9bf4598-mg8rp"] Nov 01 20:35:32.998008 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:35:32.997970 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2993469_8d17_47c8_8fd3_e298c468b4a9.slice/crio-12c6159a5324b3e2591cbd8919d46cd38c15e8b739d653e5c3bde7227d07754a WatchSource:0}: Error finding container 12c6159a5324b3e2591cbd8919d46cd38c15e8b739d653e5c3bde7227d07754a: Status 404 returned error can't find the container with id 12c6159a5324b3e2591cbd8919d46cd38c15e8b739d653e5c3bde7227d07754a Nov 01 20:35:33.118638 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:33.118561 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:33.390761 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:33.390673 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:33.552831 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:33.552793 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-85b9bf4598-mg8rp" event={"ID":"b2993469-8d17-47c8-8fd3-e298c468b4a9","Type":"ContainerStarted","Data":"12c6159a5324b3e2591cbd8919d46cd38c15e8b739d653e5c3bde7227d07754a"} Nov 01 20:35:34.345974 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:34.345932 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-kwq9p"] Nov 01 20:35:34.349479 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:34.349454 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-kwq9p" Nov 01 20:35:34.352065 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:34.352037 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:34.407723 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:34.407693 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-kwq9p"] Nov 01 20:35:34.453973 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:34.453945 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 01 20:35:34.453973 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:34.453946 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Nov 01 20:35:34.453973 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:34.453956 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 01 20:35:34.488150 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:34.488063 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/60427b21-7e27-4182-97a3-fce6a4d88207-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-kwq9p\" (UID: \"60427b21-7e27-4182-97a3-fce6a4d88207\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-kwq9p" Nov 01 20:35:34.488150 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:34.488121 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/60427b21-7e27-4182-97a3-fce6a4d88207-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-kwq9p\" (UID: \"60427b21-7e27-4182-97a3-fce6a4d88207\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-kwq9p" Nov 01 20:35:34.488361 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:34.488183 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pq69\" (UniqueName: \"kubernetes.io/projected/60427b21-7e27-4182-97a3-fce6a4d88207-kube-api-access-5pq69\") pod \"kuadrant-console-plugin-66c4fb575d-kwq9p\" (UID: \"60427b21-7e27-4182-97a3-fce6a4d88207\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-kwq9p" Nov 01 20:35:34.505008 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:34.504961 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-2mb6k\"" Nov 01 20:35:34.505008 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:34.504980 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Nov 01 20:35:34.519859 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:34.519812 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:34.589717 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:34.589674 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/60427b21-7e27-4182-97a3-fce6a4d88207-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-kwq9p\" (UID: \"60427b21-7e27-4182-97a3-fce6a4d88207\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-kwq9p" Nov 01 20:35:34.589909 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:34.589736 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/60427b21-7e27-4182-97a3-fce6a4d88207-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-kwq9p\" (UID: \"60427b21-7e27-4182-97a3-fce6a4d88207\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-kwq9p" Nov 01 20:35:34.589909 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:34.589769 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pq69\" (UniqueName: \"kubernetes.io/projected/60427b21-7e27-4182-97a3-fce6a4d88207-kube-api-access-5pq69\") pod \"kuadrant-console-plugin-66c4fb575d-kwq9p\" (UID: \"60427b21-7e27-4182-97a3-fce6a4d88207\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-kwq9p" Nov 01 20:35:34.590533 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:34.590497 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/60427b21-7e27-4182-97a3-fce6a4d88207-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-kwq9p\" (UID: \"60427b21-7e27-4182-97a3-fce6a4d88207\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-kwq9p" Nov 01 20:35:34.593732 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:34.593706 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/60427b21-7e27-4182-97a3-fce6a4d88207-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-kwq9p\" (UID: \"60427b21-7e27-4182-97a3-fce6a4d88207\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-kwq9p" Nov 01 20:35:34.657855 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:34.657787 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pq69\" (UniqueName: \"kubernetes.io/projected/60427b21-7e27-4182-97a3-fce6a4d88207-kube-api-access-5pq69\") pod \"kuadrant-console-plugin-66c4fb575d-kwq9p\" (UID: \"60427b21-7e27-4182-97a3-fce6a4d88207\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-kwq9p" Nov 01 20:35:34.661624 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:34.661599 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-kwq9p" Nov 01 20:35:35.010231 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:35.010188 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:35.035368 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:35.035228 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-kwq9p"] Nov 01 20:35:35.039321 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:35:35.039281 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60427b21_7e27_4182_97a3_fce6a4d88207.slice/crio-30c334722b577506b5ac5468a70d1c658d994503ba5afe2d7687f8a98067751f WatchSource:0}: Error finding container 30c334722b577506b5ac5468a70d1c658d994503ba5afe2d7687f8a98067751f: Status 404 returned error can't find the container with id 30c334722b577506b5ac5468a70d1c658d994503ba5afe2d7687f8a98067751f Nov 01 20:35:35.179049 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:35.178986 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:35.489224 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:35.489174 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:35.542597 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:35.542571 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-dt84g" Nov 01 20:35:35.563694 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:35.563663 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-kwq9p" event={"ID":"60427b21-7e27-4182-97a3-fce6a4d88207","Type":"ContainerStarted","Data":"30c334722b577506b5ac5468a70d1c658d994503ba5afe2d7687f8a98067751f"} Nov 01 20:35:35.639363 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:35.639320 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:35.822223 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:35.822146 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:35.990457 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:35.990430 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:36.569498 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:36.569463 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-85b9bf4598-mg8rp" event={"ID":"b2993469-8d17-47c8-8fd3-e298c468b4a9","Type":"ContainerStarted","Data":"6316842b64c448a9c9049f5df22c1e1cd0954c565545d142d831b5b7a90f0dd0"} Nov 01 20:35:36.685894 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:36.685844 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-85b9bf4598-mg8rp" podStartSLOduration=1.7155743829999999 podStartE2EDuration="4.685829308s" totalImagesPullingTime="2.970254925s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:35:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:35:36.685411675 +0000 UTC m=+287.463051811" watchObservedRunningTime="2026-11-01 20:35:36.685829308 +0000 UTC m=+287.463469480" Nov 01 20:35:39.984457 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:39.984407 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:43.818917 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:43.818883 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:43.993162 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:43.993128 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:44.638956 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:44.638921 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-gx48d"] Nov 01 20:35:44.648088 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:44.648057 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-gx48d" Nov 01 20:35:44.701581 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:44.701547 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 01 20:35:44.752673 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:44.752640 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 01 20:35:44.753024 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:44.752983 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-6dgvs\"" Nov 01 20:35:44.788013 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:44.787962 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/78640135-a1da-48d7-8212-ecb337022dee-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-gx48d\" (UID: \"78640135-a1da-48d7-8212-ecb337022dee\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-gx48d" Nov 01 20:35:44.788196 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:44.788025 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78640135-a1da-48d7-8212-ecb337022dee-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-gx48d\" (UID: \"78640135-a1da-48d7-8212-ecb337022dee\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-gx48d" Nov 01 20:35:44.788196 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:44.788065 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/78640135-a1da-48d7-8212-ecb337022dee-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-gx48d\" (UID: \"78640135-a1da-48d7-8212-ecb337022dee\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-gx48d" Nov 01 20:35:44.788196 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:44.788129 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg722\" (UniqueName: \"kubernetes.io/projected/78640135-a1da-48d7-8212-ecb337022dee-kube-api-access-gg722\") pod \"kserve-module-controller-manager-7566cf5fc8-gx48d\" (UID: \"78640135-a1da-48d7-8212-ecb337022dee\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-gx48d" Nov 01 20:35:44.855774 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:44.855737 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Nov 01 20:35:44.857666 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:44.857634 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-gx48d"] Nov 01 20:35:44.889327 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:44.889247 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/78640135-a1da-48d7-8212-ecb337022dee-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-gx48d\" (UID: \"78640135-a1da-48d7-8212-ecb337022dee\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-gx48d" Nov 01 20:35:44.889327 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:44.889285 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78640135-a1da-48d7-8212-ecb337022dee-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-gx48d\" (UID: \"78640135-a1da-48d7-8212-ecb337022dee\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-gx48d" Nov 01 20:35:44.889327 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:44.889321 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/78640135-a1da-48d7-8212-ecb337022dee-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-gx48d\" (UID: \"78640135-a1da-48d7-8212-ecb337022dee\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-gx48d" Nov 01 20:35:44.889552 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:44.889386 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg722\" (UniqueName: \"kubernetes.io/projected/78640135-a1da-48d7-8212-ecb337022dee-kube-api-access-gg722\") pod \"kserve-module-controller-manager-7566cf5fc8-gx48d\" (UID: \"78640135-a1da-48d7-8212-ecb337022dee\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-gx48d" Nov 01 20:35:44.889651 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:44.889633 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78640135-a1da-48d7-8212-ecb337022dee-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-gx48d\" (UID: \"78640135-a1da-48d7-8212-ecb337022dee\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-gx48d" Nov 01 20:35:44.889729 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:44.889707 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/78640135-a1da-48d7-8212-ecb337022dee-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-gx48d\" (UID: \"78640135-a1da-48d7-8212-ecb337022dee\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-gx48d" Nov 01 20:35:44.890726 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:44.890701 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/78640135-a1da-48d7-8212-ecb337022dee-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-gx48d\" (UID: \"78640135-a1da-48d7-8212-ecb337022dee\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-gx48d" Nov 01 20:35:44.947668 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:44.947634 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg722\" (UniqueName: \"kubernetes.io/projected/78640135-a1da-48d7-8212-ecb337022dee-kube-api-access-gg722\") pod \"kserve-module-controller-manager-7566cf5fc8-gx48d\" (UID: \"78640135-a1da-48d7-8212-ecb337022dee\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-gx48d" Nov 01 20:35:44.963479 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:44.963445 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-gx48d" Nov 01 20:35:45.398205 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:45.398171 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-gx48d"] Nov 01 20:35:45.400220 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:35:45.400196 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78640135_a1da_48d7_8212_ecb337022dee.slice/crio-043735a8b8d0bcd9ccb7a5d1a0f1862796d1221930f2e4c1404b6f9f839afa9b WatchSource:0}: Error finding container 043735a8b8d0bcd9ccb7a5d1a0f1862796d1221930f2e4c1404b6f9f839afa9b: Status 404 returned error can't find the container with id 043735a8b8d0bcd9ccb7a5d1a0f1862796d1221930f2e4c1404b6f9f839afa9b Nov 01 20:35:45.610280 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:45.610193 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-kwq9p" event={"ID":"60427b21-7e27-4182-97a3-fce6a4d88207","Type":"ContainerStarted","Data":"4e1e37a3e2b2fe85acf30971df15aceab78e7dc4cd0c5c78fce428c99b914e91"} Nov 01 20:35:45.611491 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:45.611461 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-gx48d" event={"ID":"78640135-a1da-48d7-8212-ecb337022dee","Type":"ContainerStarted","Data":"043735a8b8d0bcd9ccb7a5d1a0f1862796d1221930f2e4c1404b6f9f839afa9b"} Nov 01 20:35:45.726085 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:45.726026 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-kwq9p" podStartSLOduration=1.4851478519999999 podStartE2EDuration="11.725985801s" totalImagesPullingTime="10.240837949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:35:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:35:45.725627729 +0000 UTC m=+296.503267863" watchObservedRunningTime="2026-11-01 20:35:45.725985801 +0000 UTC m=+296.503625936" Nov 01 20:35:45.769988 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:35:45.769952 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 20:35:46.266269 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:46.266234 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:46.435891 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:46.435855 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:48.632361 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:48.632325 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-gx48d" event={"ID":"78640135-a1da-48d7-8212-ecb337022dee","Type":"ContainerStarted","Data":"03cea7da2cbcb8c669251d370a2921fb6963efb834d18761f5e905ae83bffb7e"} Nov 01 20:35:48.632804 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:48.632481 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-gx48d" Nov 01 20:35:48.749957 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:48.749901 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-gx48d" podStartSLOduration=2.332641292 podStartE2EDuration="4.749888474s" totalImagesPullingTime="2.417247182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:35:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:35:48.749326537 +0000 UTC m=+299.526966672" watchObservedRunningTime="2026-11-01 20:35:48.749888474 +0000 UTC m=+299.527528608" Nov 01 20:35:51.079848 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:51.079811 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:51.268382 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:51.268345 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:53.744642 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:53.744609 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:53.776406 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:53.776363 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-gfq8l"] Nov 01 20:35:53.781745 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:53.781722 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-gfq8l" Nov 01 20:35:53.838091 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:53.838053 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 01 20:35:53.839189 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:53.839156 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-gfq8l"] Nov 01 20:35:53.864191 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:53.864158 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d4d6814b-29e6-43d9-a28b-07826bf3fed9-config-file\") pod \"limitador-limitador-74dfcfb894-gfq8l\" (UID: \"d4d6814b-29e6-43d9-a28b-07826bf3fed9\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-gfq8l" Nov 01 20:35:53.864351 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:53.864198 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkpx2\" (UniqueName: \"kubernetes.io/projected/d4d6814b-29e6-43d9-a28b-07826bf3fed9-kube-api-access-xkpx2\") pod \"limitador-limitador-74dfcfb894-gfq8l\" (UID: \"d4d6814b-29e6-43d9-a28b-07826bf3fed9\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-gfq8l" Nov 01 20:35:53.964778 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:53.964748 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d4d6814b-29e6-43d9-a28b-07826bf3fed9-config-file\") pod \"limitador-limitador-74dfcfb894-gfq8l\" (UID: \"d4d6814b-29e6-43d9-a28b-07826bf3fed9\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-gfq8l" Nov 01 20:35:53.964956 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:53.964789 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkpx2\" (UniqueName: \"kubernetes.io/projected/d4d6814b-29e6-43d9-a28b-07826bf3fed9-kube-api-access-xkpx2\") pod \"limitador-limitador-74dfcfb894-gfq8l\" (UID: \"d4d6814b-29e6-43d9-a28b-07826bf3fed9\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-gfq8l" Nov 01 20:35:53.965498 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:53.965477 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d4d6814b-29e6-43d9-a28b-07826bf3fed9-config-file\") pod \"limitador-limitador-74dfcfb894-gfq8l\" (UID: \"d4d6814b-29e6-43d9-a28b-07826bf3fed9\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-gfq8l" Nov 01 20:35:54.035666 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:54.035584 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkpx2\" (UniqueName: \"kubernetes.io/projected/d4d6814b-29e6-43d9-a28b-07826bf3fed9-kube-api-access-xkpx2\") pod \"limitador-limitador-74dfcfb894-gfq8l\" (UID: \"d4d6814b-29e6-43d9-a28b-07826bf3fed9\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-gfq8l" Nov 01 20:35:54.036155 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:54.036135 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-gfq8l"] Nov 01 20:35:54.094147 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:54.094104 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-gfq8l" Nov 01 20:35:54.470480 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:54.470455 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-gfq8l"] Nov 01 20:35:54.472847 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:35:54.472820 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4d6814b_29e6_43d9_a28b_07826bf3fed9.slice/crio-7fe9b91556d982f14e228f96135ca865903b3171e66512e3528f27677c706001 WatchSource:0}: Error finding container 7fe9b91556d982f14e228f96135ca865903b3171e66512e3528f27677c706001: Status 404 returned error can't find the container with id 7fe9b91556d982f14e228f96135ca865903b3171e66512e3528f27677c706001 Nov 01 20:35:54.475463 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:54.475445 4484 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 20:35:54.657851 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:54.657813 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-gfq8l" event={"ID":"d4d6814b-29e6-43d9-a28b-07826bf3fed9","Type":"ContainerStarted","Data":"7fe9b91556d982f14e228f96135ca865903b3171e66512e3528f27677c706001"} Nov 01 20:35:54.730199 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:54.730119 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:54.897644 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:54.897610 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:56.185936 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:56.185902 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:56.341259 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:56.341221 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:56.629505 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:56.629467 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:56.797352 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:56.797319 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:56.954396 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:56.954308 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:57.622933 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:57.622897 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:57.776866 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:57.776819 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:57.935397 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:57.935326 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:58.678227 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:58.678188 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-gfq8l" event={"ID":"d4d6814b-29e6-43d9-a28b-07826bf3fed9","Type":"ContainerStarted","Data":"713c5411498a41e0b1c2d3e9b87dedf085bebc34d3f6402cc7029656cdbb2204"} Nov 01 20:35:58.678630 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:58.678385 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-gfq8l" Nov 01 20:35:58.795774 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:35:58.795715 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-gfq8l" podStartSLOduration=2.287960924 podStartE2EDuration="5.79569895s" totalImagesPullingTime="3.507738026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:35:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:35:58.794579614 +0000 UTC m=+309.572219750" watchObservedRunningTime="2026-11-01 20:35:58.79569895 +0000 UTC m=+309.573339085" Nov 01 20:36:05.624018 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:05.623922 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:05.794655 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:05.794613 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:08.072836 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:08.072798 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:08.223328 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:08.223291 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:09.637642 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:09.637609 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-gx48d" Nov 01 20:36:09.683753 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:09.683724 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-gfq8l" Nov 01 20:36:12.154281 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:12.154244 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-fmv7k"] Nov 01 20:36:12.159540 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:12.159518 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-fmv7k" Nov 01 20:36:12.227710 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:12.227679 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-tmsnc\"" Nov 01 20:36:12.231147 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:12.231118 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6xfm\" (UniqueName: \"kubernetes.io/projected/90042f69-783f-4cd5-86b3-f95dd2d48fb8-kube-api-access-t6xfm\") pod \"authorino-56c6c9ddd5-fmv7k\" (UID: \"90042f69-783f-4cd5-86b3-f95dd2d48fb8\") " pod="kuadrant-system/authorino-56c6c9ddd5-fmv7k" Nov 01 20:36:12.275588 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:12.275552 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-fmv7k"] Nov 01 20:36:12.331730 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:12.331691 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6xfm\" (UniqueName: \"kubernetes.io/projected/90042f69-783f-4cd5-86b3-f95dd2d48fb8-kube-api-access-t6xfm\") pod \"authorino-56c6c9ddd5-fmv7k\" (UID: \"90042f69-783f-4cd5-86b3-f95dd2d48fb8\") " pod="kuadrant-system/authorino-56c6c9ddd5-fmv7k" Nov 01 20:36:12.389430 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:12.389400 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6xfm\" (UniqueName: \"kubernetes.io/projected/90042f69-783f-4cd5-86b3-f95dd2d48fb8-kube-api-access-t6xfm\") pod \"authorino-56c6c9ddd5-fmv7k\" (UID: \"90042f69-783f-4cd5-86b3-f95dd2d48fb8\") " pod="kuadrant-system/authorino-56c6c9ddd5-fmv7k" Nov 01 20:36:12.471171 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:12.471085 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-fmv7k" Nov 01 20:36:12.835672 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:12.835632 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:12.848027 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:12.847977 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-fmv7k"] Nov 01 20:36:12.850026 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:36:12.849978 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90042f69_783f_4cd5_86b3_f95dd2d48fb8.slice/crio-7fcd87581d7516165fba525a1e9d4f513fce235cd2d8638cdbbaa9d8239988f5 WatchSource:0}: Error finding container 7fcd87581d7516165fba525a1e9d4f513fce235cd2d8638cdbbaa9d8239988f5: Status 404 returned error can't find the container with id 7fcd87581d7516165fba525a1e9d4f513fce235cd2d8638cdbbaa9d8239988f5 Nov 01 20:36:13.001924 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:13.001889 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:13.741363 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:13.741328 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-fmv7k" event={"ID":"90042f69-783f-4cd5-86b3-f95dd2d48fb8","Type":"ContainerStarted","Data":"7fcd87581d7516165fba525a1e9d4f513fce235cd2d8638cdbbaa9d8239988f5"} Nov 01 20:36:15.752030 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:15.751977 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-fmv7k" event={"ID":"90042f69-783f-4cd5-86b3-f95dd2d48fb8","Type":"ContainerStarted","Data":"b5459e63a275fd896b82439eb196f73232cada413394593a29dd9f03b78eb695"} Nov 01 20:36:15.765115 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:36:15.765084 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 20:36:15.866812 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:15.866758 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-fmv7k" podStartSLOduration=1.59244819 podStartE2EDuration="3.866743371s" totalImagesPullingTime="2.274295181s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:36:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:36:15.865897677 +0000 UTC m=+326.643537812" watchObservedRunningTime="2026-11-01 20:36:15.866743371 +0000 UTC m=+326.644383525" Nov 01 20:36:34.814615 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:34.814561 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:35.705422 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:35.705382 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:38.612235 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:38.612201 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-gfq8l"] Nov 01 20:36:38.612715 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:38.612468 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-gfq8l" podUID="d4d6814b-29e6-43d9-a28b-07826bf3fed9" containerName="limitador" containerID="cri-o://713c5411498a41e0b1c2d3e9b87dedf085bebc34d3f6402cc7029656cdbb2204" gracePeriod=30 Nov 01 20:36:39.160245 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:39.160220 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-gfq8l" Nov 01 20:36:39.174319 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:39.174261 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4d6814b-29e6-43d9-a28b-07826bf3fed9-kube-api-access-xkpx2\" (UniqueName: \"kubernetes.io/projected/d4d6814b-29e6-43d9-a28b-07826bf3fed9-kube-api-access-xkpx2\") pod \"d4d6814b-29e6-43d9-a28b-07826bf3fed9\" (UID: \"d4d6814b-29e6-43d9-a28b-07826bf3fed9\") " Nov 01 20:36:39.174427 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:39.174344 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4d6814b-29e6-43d9-a28b-07826bf3fed9-config-file\" (UniqueName: \"kubernetes.io/configmap/d4d6814b-29e6-43d9-a28b-07826bf3fed9-config-file\") pod \"d4d6814b-29e6-43d9-a28b-07826bf3fed9\" (UID: \"d4d6814b-29e6-43d9-a28b-07826bf3fed9\") " Nov 01 20:36:39.174679 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:39.174653 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4d6814b-29e6-43d9-a28b-07826bf3fed9-config-file" pod "d4d6814b-29e6-43d9-a28b-07826bf3fed9" (UID: "d4d6814b-29e6-43d9-a28b-07826bf3fed9"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:36:39.176392 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:39.176364 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4d6814b-29e6-43d9-a28b-07826bf3fed9-kube-api-access-xkpx2" pod "d4d6814b-29e6-43d9-a28b-07826bf3fed9" (UID: "d4d6814b-29e6-43d9-a28b-07826bf3fed9"). InnerVolumeSpecName "kube-api-access-xkpx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:36:39.275042 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:39.274985 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkpx2\" (UniqueName: \"kubernetes.io/projected/d4d6814b-29e6-43d9-a28b-07826bf3fed9-kube-api-access-xkpx2\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:36:39.275042 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:39.275036 4484 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d4d6814b-29e6-43d9-a28b-07826bf3fed9-config-file\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:36:39.857384 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:39.857348 4484 generic.go:356] "Generic (PLEG): container finished" podID="d4d6814b-29e6-43d9-a28b-07826bf3fed9" containerID="713c5411498a41e0b1c2d3e9b87dedf085bebc34d3f6402cc7029656cdbb2204" exitCode=0 Nov 01 20:36:39.857865 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:39.857408 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-gfq8l" Nov 01 20:36:39.857865 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:39.857417 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-gfq8l" event={"ID":"d4d6814b-29e6-43d9-a28b-07826bf3fed9","Type":"ContainerDied","Data":"713c5411498a41e0b1c2d3e9b87dedf085bebc34d3f6402cc7029656cdbb2204"} Nov 01 20:36:39.857865 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:39.857501 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-gfq8l" event={"ID":"d4d6814b-29e6-43d9-a28b-07826bf3fed9","Type":"ContainerDied","Data":"7fe9b91556d982f14e228f96135ca865903b3171e66512e3528f27677c706001"} Nov 01 20:36:39.857865 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:39.857538 4484 scope.go:122] "RemoveContainer" containerID="713c5411498a41e0b1c2d3e9b87dedf085bebc34d3f6402cc7029656cdbb2204" Nov 01 20:36:39.865840 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:39.865815 4484 scope.go:122] "RemoveContainer" containerID="713c5411498a41e0b1c2d3e9b87dedf085bebc34d3f6402cc7029656cdbb2204" Nov 01 20:36:39.866142 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:36:39.866122 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"713c5411498a41e0b1c2d3e9b87dedf085bebc34d3f6402cc7029656cdbb2204\": container with ID starting with 713c5411498a41e0b1c2d3e9b87dedf085bebc34d3f6402cc7029656cdbb2204 not found: ID does not exist" containerID="713c5411498a41e0b1c2d3e9b87dedf085bebc34d3f6402cc7029656cdbb2204" Nov 01 20:36:39.866220 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:39.866157 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"713c5411498a41e0b1c2d3e9b87dedf085bebc34d3f6402cc7029656cdbb2204"} err="failed to get container status \"713c5411498a41e0b1c2d3e9b87dedf085bebc34d3f6402cc7029656cdbb2204\": rpc error: code = NotFound desc = could not find container \"713c5411498a41e0b1c2d3e9b87dedf085bebc34d3f6402cc7029656cdbb2204\": container with ID starting with 713c5411498a41e0b1c2d3e9b87dedf085bebc34d3f6402cc7029656cdbb2204 not found: ID does not exist" Nov 01 20:36:40.024697 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:40.024657 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-gfq8l"] Nov 01 20:36:40.075607 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:40.075578 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-gfq8l"] Nov 01 20:36:41.369484 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:41.369447 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4d6814b-29e6-43d9-a28b-07826bf3fed9" path="/var/lib/kubelet/pods/d4d6814b-29e6-43d9-a28b-07826bf3fed9/volumes" Nov 01 20:36:43.560509 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:43.560479 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4bw98"] Nov 01 20:36:43.560904 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:43.560824 4484 cpu_manager.go:414] "containerMap: removing container" podUID="d4d6814b-29e6-43d9-a28b-07826bf3fed9" containerName="limitador" Nov 01 20:36:43.560904 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:43.560835 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4d6814b-29e6-43d9-a28b-07826bf3fed9" containerName="limitador" Nov 01 20:36:43.560904 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:43.560903 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4d6814b-29e6-43d9-a28b-07826bf3fed9" containerName="limitador" Nov 01 20:36:43.566160 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:43.566143 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4bw98" Nov 01 20:36:43.619950 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:43.619913 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ef701d5e-a280-4ad6-98df-69ee834a7901-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4bw98\" (UID: \"ef701d5e-a280-4ad6-98df-69ee834a7901\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4bw98" Nov 01 20:36:43.619950 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:43.619954 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ef701d5e-a280-4ad6-98df-69ee834a7901-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4bw98\" (UID: \"ef701d5e-a280-4ad6-98df-69ee834a7901\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4bw98" Nov 01 20:36:43.620197 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:43.620028 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ef701d5e-a280-4ad6-98df-69ee834a7901-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4bw98\" (UID: \"ef701d5e-a280-4ad6-98df-69ee834a7901\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4bw98" Nov 01 20:36:43.620197 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:43.620081 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef701d5e-a280-4ad6-98df-69ee834a7901-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4bw98\" (UID: \"ef701d5e-a280-4ad6-98df-69ee834a7901\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4bw98" Nov 01 20:36:43.620197 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:43.620142 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef701d5e-a280-4ad6-98df-69ee834a7901-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4bw98\" (UID: \"ef701d5e-a280-4ad6-98df-69ee834a7901\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4bw98" Nov 01 20:36:43.620323 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:43.620204 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ef701d5e-a280-4ad6-98df-69ee834a7901-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4bw98\" (UID: \"ef701d5e-a280-4ad6-98df-69ee834a7901\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4bw98" Nov 01 20:36:43.620323 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:43.620242 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2s65\" (UniqueName: \"kubernetes.io/projected/ef701d5e-a280-4ad6-98df-69ee834a7901-kube-api-access-j2s65\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4bw98\" (UID: \"ef701d5e-a280-4ad6-98df-69ee834a7901\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4bw98" Nov 01 20:36:43.621146 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:43.621130 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 01 20:36:43.622220 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:43.622188 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 01 20:36:43.622220 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:43.622177 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 01 20:36:43.671555 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:43.671523 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 01 20:36:43.671736 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:43.671629 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-z57nd\"" Nov 01 20:36:43.673216 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:43.673192 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4bw98"] Nov 01 20:36:43.721142 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:43.721113 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef701d5e-a280-4ad6-98df-69ee834a7901-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4bw98\" (UID: \"ef701d5e-a280-4ad6-98df-69ee834a7901\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4bw98" Nov 01 20:36:43.721318 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:43.721148 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef701d5e-a280-4ad6-98df-69ee834a7901-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4bw98\" (UID: \"ef701d5e-a280-4ad6-98df-69ee834a7901\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4bw98" Nov 01 20:36:43.721318 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:43.721176 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ef701d5e-a280-4ad6-98df-69ee834a7901-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4bw98\" (UID: \"ef701d5e-a280-4ad6-98df-69ee834a7901\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4bw98" Nov 01 20:36:43.721318 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:43.721222 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2s65\" (UniqueName: \"kubernetes.io/projected/ef701d5e-a280-4ad6-98df-69ee834a7901-kube-api-access-j2s65\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4bw98\" (UID: \"ef701d5e-a280-4ad6-98df-69ee834a7901\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4bw98" Nov 01 20:36:43.721318 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:43.721272 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ef701d5e-a280-4ad6-98df-69ee834a7901-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4bw98\" (UID: \"ef701d5e-a280-4ad6-98df-69ee834a7901\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4bw98" Nov 01 20:36:43.721318 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:43.721301 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ef701d5e-a280-4ad6-98df-69ee834a7901-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4bw98\" (UID: \"ef701d5e-a280-4ad6-98df-69ee834a7901\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4bw98" Nov 01 20:36:43.721580 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:43.721344 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ef701d5e-a280-4ad6-98df-69ee834a7901-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4bw98\" (UID: \"ef701d5e-a280-4ad6-98df-69ee834a7901\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4bw98" Nov 01 20:36:43.722150 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:43.722124 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ef701d5e-a280-4ad6-98df-69ee834a7901-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4bw98\" (UID: \"ef701d5e-a280-4ad6-98df-69ee834a7901\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4bw98" Nov 01 20:36:43.723518 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:43.723495 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef701d5e-a280-4ad6-98df-69ee834a7901-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4bw98\" (UID: \"ef701d5e-a280-4ad6-98df-69ee834a7901\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4bw98" Nov 01 20:36:43.723679 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:43.723644 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ef701d5e-a280-4ad6-98df-69ee834a7901-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4bw98\" (UID: \"ef701d5e-a280-4ad6-98df-69ee834a7901\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4bw98" Nov 01 20:36:43.723915 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:43.723896 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ef701d5e-a280-4ad6-98df-69ee834a7901-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4bw98\" (UID: \"ef701d5e-a280-4ad6-98df-69ee834a7901\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4bw98" Nov 01 20:36:43.724018 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:43.723977 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ef701d5e-a280-4ad6-98df-69ee834a7901-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4bw98\" (UID: \"ef701d5e-a280-4ad6-98df-69ee834a7901\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4bw98" Nov 01 20:36:43.779473 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:43.779441 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef701d5e-a280-4ad6-98df-69ee834a7901-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4bw98\" (UID: \"ef701d5e-a280-4ad6-98df-69ee834a7901\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4bw98" Nov 01 20:36:43.779615 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:43.779594 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2s65\" (UniqueName: \"kubernetes.io/projected/ef701d5e-a280-4ad6-98df-69ee834a7901-kube-api-access-j2s65\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4bw98\" (UID: \"ef701d5e-a280-4ad6-98df-69ee834a7901\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4bw98" Nov 01 20:36:43.877531 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:43.877455 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4bw98" Nov 01 20:36:43.997796 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:43.997764 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:44.235667 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:44.235581 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:44.371096 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:44.371061 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4bw98"] Nov 01 20:36:44.410148 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:36:44.410116 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef701d5e_a280_4ad6_98df_69ee834a7901.slice/crio-35b18919fb8e06662446fa9e32151b803e0ec633146eb0ed5132fac58c17307f WatchSource:0}: Error finding container 35b18919fb8e06662446fa9e32151b803e0ec633146eb0ed5132fac58c17307f: Status 404 returned error can't find the container with id 35b18919fb8e06662446fa9e32151b803e0ec633146eb0ed5132fac58c17307f Nov 01 20:36:44.705796 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:44.705767 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:44.880007 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:44.879971 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4bw98" event={"ID":"ef701d5e-a280-4ad6-98df-69ee834a7901","Type":"ContainerStarted","Data":"35b18919fb8e06662446fa9e32151b803e0ec633146eb0ed5132fac58c17307f"} Nov 01 20:36:45.765330 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:36:45.765287 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 20:36:48.392482 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:48.392447 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:36:48.392743 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:48.392516 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:36:48.870537 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:48.870490 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:48.900238 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:48.900200 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4bw98" event={"ID":"ef701d5e-a280-4ad6-98df-69ee834a7901","Type":"ContainerStarted","Data":"231824ba4a35758ba8f98251d629efdee39ee5edfc20707a3b4dd9124873d957"} Nov 01 20:36:48.900435 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:48.900321 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4bw98" Nov 01 20:36:49.059249 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:49.059212 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:49.121348 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:49.121233 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4bw98" podStartSLOduration=2.141607173 podStartE2EDuration="6.121219926s" totalImagesPullingTime="3.979612753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:36:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:36:49.120544786 +0000 UTC m=+359.898184923" watchObservedRunningTime="2026-11-01 20:36:49.121219926 +0000 UTC m=+359.898860060" Nov 01 20:36:49.216630 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:49.216594 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:49.373555 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:49.373471 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:49.527966 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:49.527930 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:49.682432 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:49.682322 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:49.844471 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:49.844433 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:49.906447 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:49.906417 4484 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-4bw98 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 20:36:49.907044 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:49.906500 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4bw98" podUID="ef701d5e-a280-4ad6-98df-69ee834a7901" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 20:36:49.998155 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:49.998119 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:51.078611 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:51.078575 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:52.905869 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:52.905836 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4bw98" Nov 01 20:36:56.053542 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:56.053493 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:56.213022 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:56.212952 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:56.405886 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:56.405802 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:56.566198 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:56.566151 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:56.717475 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:56.717388 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:56.862667 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:56.862632 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:57.023825 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:57.023796 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:57.180861 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:57.180824 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:57.336041 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:36:57.335947 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:37:07.210674 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:37:07.210633 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:37:07.383368 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:37:07.383324 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:37:07.538102 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:37:07.538069 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:37:07.701047 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:37:07.700987 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:37:13.450614 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:37:13.450575 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:37:13.615478 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:37:13.615440 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:37:15.765407 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:37:15.765373 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 20:37:16.868845 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:37:16.868816 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:37:17.134730 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:37:17.134632 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:37:35.181282 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:37:35.181190 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:37:42.026898 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:37:42.026852 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:37:42.178099 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:37:42.178062 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:37:45.765559 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:37:45.765519 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 20:37:46.990602 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:37:46.990551 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:37:47.168263 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:37:47.168234 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:37:49.079892 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:37:49.079849 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:37:49.236871 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:37:49.236815 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:37:49.390136 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:37:49.390051 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:37:49.550958 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:37:49.550892 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:38:02.407600 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:38:02.407559 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:38:02.620770 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:38:02.620734 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:38:06.279957 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:38:06.279913 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-fmv7k"] Nov 01 20:38:06.280755 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:38:06.280129 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-fmv7k" podUID="90042f69-783f-4cd5-86b3-f95dd2d48fb8" containerName="authorino" containerID="cri-o://b5459e63a275fd896b82439eb196f73232cada413394593a29dd9f03b78eb695" gracePeriod=30 Nov 01 20:38:06.536473 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:38:06.536409 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-fmv7k" Nov 01 20:38:06.634279 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:38:06.634240 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90042f69-783f-4cd5-86b3-f95dd2d48fb8-kube-api-access-t6xfm\" (UniqueName: \"kubernetes.io/projected/90042f69-783f-4cd5-86b3-f95dd2d48fb8-kube-api-access-t6xfm\") pod \"90042f69-783f-4cd5-86b3-f95dd2d48fb8\" (UID: \"90042f69-783f-4cd5-86b3-f95dd2d48fb8\") " Nov 01 20:38:06.636342 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:38:06.636311 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90042f69-783f-4cd5-86b3-f95dd2d48fb8-kube-api-access-t6xfm" pod "90042f69-783f-4cd5-86b3-f95dd2d48fb8" (UID: "90042f69-783f-4cd5-86b3-f95dd2d48fb8"). InnerVolumeSpecName "kube-api-access-t6xfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:38:06.735493 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:38:06.735462 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6xfm\" (UniqueName: \"kubernetes.io/projected/90042f69-783f-4cd5-86b3-f95dd2d48fb8-kube-api-access-t6xfm\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:07.236531 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:38:07.236490 4484 generic.go:356] "Generic (PLEG): container finished" podID="90042f69-783f-4cd5-86b3-f95dd2d48fb8" containerID="b5459e63a275fd896b82439eb196f73232cada413394593a29dd9f03b78eb695" exitCode=0 Nov 01 20:38:07.236711 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:38:07.236538 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-fmv7k" Nov 01 20:38:07.236711 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:38:07.236576 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-fmv7k" event={"ID":"90042f69-783f-4cd5-86b3-f95dd2d48fb8","Type":"ContainerDied","Data":"b5459e63a275fd896b82439eb196f73232cada413394593a29dd9f03b78eb695"} Nov 01 20:38:07.236711 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:38:07.236622 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-fmv7k" event={"ID":"90042f69-783f-4cd5-86b3-f95dd2d48fb8","Type":"ContainerDied","Data":"7fcd87581d7516165fba525a1e9d4f513fce235cd2d8638cdbbaa9d8239988f5"} Nov 01 20:38:07.236711 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:38:07.236639 4484 scope.go:122] "RemoveContainer" containerID="b5459e63a275fd896b82439eb196f73232cada413394593a29dd9f03b78eb695" Nov 01 20:38:07.244934 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:38:07.244917 4484 scope.go:122] "RemoveContainer" containerID="b5459e63a275fd896b82439eb196f73232cada413394593a29dd9f03b78eb695" Nov 01 20:38:07.245205 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:38:07.245186 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5459e63a275fd896b82439eb196f73232cada413394593a29dd9f03b78eb695\": container with ID starting with b5459e63a275fd896b82439eb196f73232cada413394593a29dd9f03b78eb695 not found: ID does not exist" containerID="b5459e63a275fd896b82439eb196f73232cada413394593a29dd9f03b78eb695" Nov 01 20:38:07.245278 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:38:07.245218 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5459e63a275fd896b82439eb196f73232cada413394593a29dd9f03b78eb695"} err="failed to get container status \"b5459e63a275fd896b82439eb196f73232cada413394593a29dd9f03b78eb695\": rpc error: code = NotFound desc = could not find container \"b5459e63a275fd896b82439eb196f73232cada413394593a29dd9f03b78eb695\": container with ID starting with b5459e63a275fd896b82439eb196f73232cada413394593a29dd9f03b78eb695 not found: ID does not exist" Nov 01 20:38:07.402539 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:38:07.402504 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-fmv7k"] Nov 01 20:38:07.405230 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:38:07.405203 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-fmv7k"] Nov 01 20:38:09.369329 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:38:09.369284 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90042f69-783f-4cd5-86b3-f95dd2d48fb8" path="/var/lib/kubelet/pods/90042f69-783f-4cd5-86b3-f95dd2d48fb8/volumes" Nov 01 20:38:15.764785 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:38:15.764746 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 20:38:35.458055 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:38:35.458021 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:38:41.045020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:38:41.044963 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:38:41.214335 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:38:41.214298 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:38:45.766109 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:38:45.766073 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 20:39:04.274681 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:04.274605 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:39:07.930481 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:07.930437 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:39:11.000098 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:11.000067 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:39:14.865391 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:14.865353 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:39:15.764773 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:39:15.764738 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 20:39:21.680189 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:21.680153 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:39:25.888060 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:25.888016 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fkcnk"] Nov 01 20:39:25.888444 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:25.888376 4484 cpu_manager.go:414] "containerMap: removing container" podUID="90042f69-783f-4cd5-86b3-f95dd2d48fb8" containerName="authorino" Nov 01 20:39:25.888444 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:25.888386 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90042f69-783f-4cd5-86b3-f95dd2d48fb8" containerName="authorino" Nov 01 20:39:25.888515 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:25.888462 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="90042f69-783f-4cd5-86b3-f95dd2d48fb8" containerName="authorino" Nov 01 20:39:25.891672 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:25.891655 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fkcnk" Nov 01 20:39:25.950388 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:25.950357 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a79aee42-1521-4d43-9178-fcf9b1738e7a-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fkcnk\" (UID: \"a79aee42-1521-4d43-9178-fcf9b1738e7a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fkcnk" Nov 01 20:39:25.950566 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:25.950403 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a79aee42-1521-4d43-9178-fcf9b1738e7a-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fkcnk\" (UID: \"a79aee42-1521-4d43-9178-fcf9b1738e7a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fkcnk" Nov 01 20:39:25.950566 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:25.950499 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a79aee42-1521-4d43-9178-fcf9b1738e7a-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fkcnk\" (UID: \"a79aee42-1521-4d43-9178-fcf9b1738e7a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fkcnk" Nov 01 20:39:25.950566 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:25.950534 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a79aee42-1521-4d43-9178-fcf9b1738e7a-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fkcnk\" (UID: \"a79aee42-1521-4d43-9178-fcf9b1738e7a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fkcnk" Nov 01 20:39:25.950699 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:25.950573 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk9gn\" (UniqueName: \"kubernetes.io/projected/a79aee42-1521-4d43-9178-fcf9b1738e7a-kube-api-access-jk9gn\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fkcnk\" (UID: \"a79aee42-1521-4d43-9178-fcf9b1738e7a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fkcnk" Nov 01 20:39:25.950699 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:25.950640 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a79aee42-1521-4d43-9178-fcf9b1738e7a-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fkcnk\" (UID: \"a79aee42-1521-4d43-9178-fcf9b1738e7a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fkcnk" Nov 01 20:39:25.997472 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:25.997440 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Nov 01 20:39:25.997671 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:25.997483 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-p9lqs\"" Nov 01 20:39:25.997671 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:25.997487 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 01 20:39:25.997671 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:25.997562 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 01 20:39:25.998975 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:25.998954 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fkcnk"] Nov 01 20:39:26.011906 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:26.011878 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:39:26.051855 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:26.051823 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a79aee42-1521-4d43-9178-fcf9b1738e7a-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fkcnk\" (UID: \"a79aee42-1521-4d43-9178-fcf9b1738e7a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fkcnk" Nov 01 20:39:26.052043 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:26.051864 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk9gn\" (UniqueName: \"kubernetes.io/projected/a79aee42-1521-4d43-9178-fcf9b1738e7a-kube-api-access-jk9gn\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fkcnk\" (UID: \"a79aee42-1521-4d43-9178-fcf9b1738e7a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fkcnk" Nov 01 20:39:26.052043 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:26.051899 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a79aee42-1521-4d43-9178-fcf9b1738e7a-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fkcnk\" (UID: \"a79aee42-1521-4d43-9178-fcf9b1738e7a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fkcnk" Nov 01 20:39:26.052043 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:26.051962 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a79aee42-1521-4d43-9178-fcf9b1738e7a-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fkcnk\" (UID: \"a79aee42-1521-4d43-9178-fcf9b1738e7a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fkcnk" Nov 01 20:39:26.052043 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:26.052027 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a79aee42-1521-4d43-9178-fcf9b1738e7a-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fkcnk\" (UID: \"a79aee42-1521-4d43-9178-fcf9b1738e7a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fkcnk" Nov 01 20:39:26.052279 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:26.052083 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a79aee42-1521-4d43-9178-fcf9b1738e7a-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fkcnk\" (UID: \"a79aee42-1521-4d43-9178-fcf9b1738e7a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fkcnk" Nov 01 20:39:26.052444 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:26.052406 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a79aee42-1521-4d43-9178-fcf9b1738e7a-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fkcnk\" (UID: \"a79aee42-1521-4d43-9178-fcf9b1738e7a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fkcnk" Nov 01 20:39:26.052444 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:26.052434 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a79aee42-1521-4d43-9178-fcf9b1738e7a-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fkcnk\" (UID: \"a79aee42-1521-4d43-9178-fcf9b1738e7a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fkcnk" Nov 01 20:39:26.052593 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:26.052454 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a79aee42-1521-4d43-9178-fcf9b1738e7a-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fkcnk\" (UID: \"a79aee42-1521-4d43-9178-fcf9b1738e7a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fkcnk" Nov 01 20:39:26.054202 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:26.054176 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a79aee42-1521-4d43-9178-fcf9b1738e7a-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fkcnk\" (UID: \"a79aee42-1521-4d43-9178-fcf9b1738e7a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fkcnk" Nov 01 20:39:26.054436 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:26.054419 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a79aee42-1521-4d43-9178-fcf9b1738e7a-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fkcnk\" (UID: \"a79aee42-1521-4d43-9178-fcf9b1738e7a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fkcnk" Nov 01 20:39:26.109263 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:26.109232 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk9gn\" (UniqueName: \"kubernetes.io/projected/a79aee42-1521-4d43-9178-fcf9b1738e7a-kube-api-access-jk9gn\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fkcnk\" (UID: \"a79aee42-1521-4d43-9178-fcf9b1738e7a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fkcnk" Nov 01 20:39:26.204465 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:26.204378 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fkcnk" Nov 01 20:39:26.573632 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:26.573606 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fkcnk"] Nov 01 20:39:26.575783 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:39:26.575756 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda79aee42_1521_4d43_9178_fcf9b1738e7a.slice/crio-fbd55f691a5ef5fb75b4e9570aa70358e6b7755969d438f0c37bedadf6603f50 WatchSource:0}: Error finding container fbd55f691a5ef5fb75b4e9570aa70358e6b7755969d438f0c37bedadf6603f50: Status 404 returned error can't find the container with id fbd55f691a5ef5fb75b4e9570aa70358e6b7755969d438f0c37bedadf6603f50 Nov 01 20:39:27.581701 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:27.581619 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fkcnk" event={"ID":"a79aee42-1521-4d43-9178-fcf9b1738e7a","Type":"ContainerStarted","Data":"fbd55f691a5ef5fb75b4e9570aa70358e6b7755969d438f0c37bedadf6603f50"} Nov 01 20:39:28.588277 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:28.588240 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fkcnk" event={"ID":"a79aee42-1521-4d43-9178-fcf9b1738e7a","Type":"ContainerStarted","Data":"291a8a2e6d491ecb728d1c2dddb0694be6cfb13785c8d21fd0a65d451fbc384d"} Nov 01 20:39:28.706103 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:28.706049 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fkcnk" podStartSLOduration=2.240627035 podStartE2EDuration="3.706033884s" totalImagesPullingTime="1.465406849s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:39:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:39:28.704861566 +0000 UTC m=+519.482501700" watchObservedRunningTime="2026-11-01 20:39:28.706033884 +0000 UTC m=+519.483674019" Nov 01 20:39:29.112104 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:29.112068 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:39:29.204776 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:29.204738 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fkcnk" Nov 01 20:39:32.555342 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:32.555306 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhk6c"] Nov 01 20:39:32.559294 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:32.559274 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhk6c" Nov 01 20:39:32.613546 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:32.613506 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b12e9430-7403-447a-a280-07a09d0e9ce4-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qhk6c\" (UID: \"b12e9430-7403-447a-a280-07a09d0e9ce4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhk6c" Nov 01 20:39:32.613731 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:32.613576 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b12e9430-7403-447a-a280-07a09d0e9ce4-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qhk6c\" (UID: \"b12e9430-7403-447a-a280-07a09d0e9ce4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhk6c" Nov 01 20:39:32.613731 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:32.613648 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b12e9430-7403-447a-a280-07a09d0e9ce4-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qhk6c\" (UID: \"b12e9430-7403-447a-a280-07a09d0e9ce4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhk6c" Nov 01 20:39:32.613731 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:32.613723 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqbxw\" (UniqueName: \"kubernetes.io/projected/b12e9430-7403-447a-a280-07a09d0e9ce4-kube-api-access-kqbxw\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qhk6c\" (UID: \"b12e9430-7403-447a-a280-07a09d0e9ce4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhk6c" Nov 01 20:39:32.613910 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:32.613764 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b12e9430-7403-447a-a280-07a09d0e9ce4-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qhk6c\" (UID: \"b12e9430-7403-447a-a280-07a09d0e9ce4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhk6c" Nov 01 20:39:32.613910 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:32.613805 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b12e9430-7403-447a-a280-07a09d0e9ce4-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qhk6c\" (UID: \"b12e9430-7403-447a-a280-07a09d0e9ce4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhk6c" Nov 01 20:39:32.664184 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:32.664144 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Nov 01 20:39:32.665675 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:32.665652 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhk6c"] Nov 01 20:39:32.692326 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:32.692293 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:39:32.714758 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:32.714717 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b12e9430-7403-447a-a280-07a09d0e9ce4-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qhk6c\" (UID: \"b12e9430-7403-447a-a280-07a09d0e9ce4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhk6c" Nov 01 20:39:32.714944 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:32.714780 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b12e9430-7403-447a-a280-07a09d0e9ce4-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qhk6c\" (UID: \"b12e9430-7403-447a-a280-07a09d0e9ce4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhk6c" Nov 01 20:39:32.714944 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:32.714813 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b12e9430-7403-447a-a280-07a09d0e9ce4-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qhk6c\" (UID: \"b12e9430-7403-447a-a280-07a09d0e9ce4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhk6c" Nov 01 20:39:32.714944 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:32.714847 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqbxw\" (UniqueName: \"kubernetes.io/projected/b12e9430-7403-447a-a280-07a09d0e9ce4-kube-api-access-kqbxw\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qhk6c\" (UID: \"b12e9430-7403-447a-a280-07a09d0e9ce4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhk6c" Nov 01 20:39:32.714944 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:32.714883 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b12e9430-7403-447a-a280-07a09d0e9ce4-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qhk6c\" (UID: \"b12e9430-7403-447a-a280-07a09d0e9ce4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhk6c" Nov 01 20:39:32.715152 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:32.715057 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b12e9430-7403-447a-a280-07a09d0e9ce4-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qhk6c\" (UID: \"b12e9430-7403-447a-a280-07a09d0e9ce4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhk6c" Nov 01 20:39:32.715231 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:32.715210 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b12e9430-7403-447a-a280-07a09d0e9ce4-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qhk6c\" (UID: \"b12e9430-7403-447a-a280-07a09d0e9ce4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhk6c" Nov 01 20:39:32.715287 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:32.715262 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b12e9430-7403-447a-a280-07a09d0e9ce4-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qhk6c\" (UID: \"b12e9430-7403-447a-a280-07a09d0e9ce4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhk6c" Nov 01 20:39:32.715403 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:32.715386 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b12e9430-7403-447a-a280-07a09d0e9ce4-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qhk6c\" (UID: \"b12e9430-7403-447a-a280-07a09d0e9ce4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhk6c" Nov 01 20:39:32.717192 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:32.717171 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b12e9430-7403-447a-a280-07a09d0e9ce4-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qhk6c\" (UID: \"b12e9430-7403-447a-a280-07a09d0e9ce4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhk6c" Nov 01 20:39:32.717343 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:32.717328 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b12e9430-7403-447a-a280-07a09d0e9ce4-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qhk6c\" (UID: \"b12e9430-7403-447a-a280-07a09d0e9ce4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhk6c" Nov 01 20:39:32.776214 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:32.776176 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqbxw\" (UniqueName: \"kubernetes.io/projected/b12e9430-7403-447a-a280-07a09d0e9ce4-kube-api-access-kqbxw\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qhk6c\" (UID: \"b12e9430-7403-447a-a280-07a09d0e9ce4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhk6c" Nov 01 20:39:32.871558 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:32.871473 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhk6c" Nov 01 20:39:33.239910 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:33.239887 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhk6c"] Nov 01 20:39:33.242221 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:39:33.242187 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb12e9430_7403_447a_a280_07a09d0e9ce4.slice/crio-ef57a680cb89203ff1f96a1a5ada622d4636c62fad6d06014439c9eb7be5f53e WatchSource:0}: Error finding container ef57a680cb89203ff1f96a1a5ada622d4636c62fad6d06014439c9eb7be5f53e: Status 404 returned error can't find the container with id ef57a680cb89203ff1f96a1a5ada622d4636c62fad6d06014439c9eb7be5f53e Nov 01 20:39:33.612518 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:33.612438 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhk6c" event={"ID":"b12e9430-7403-447a-a280-07a09d0e9ce4","Type":"ContainerStarted","Data":"ef57a680cb89203ff1f96a1a5ada622d4636c62fad6d06014439c9eb7be5f53e"} Nov 01 20:39:34.270921 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:34.270884 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zndkb"] Nov 01 20:39:34.274275 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:34.274258 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zndkb" Nov 01 20:39:34.327460 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:34.327432 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-cqvhp\"" Nov 01 20:39:34.330818 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:34.330797 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-zndkb\" (UID: \"81291c2c-8fd4-482b-8ed1-c11e9cbb2aad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zndkb" Nov 01 20:39:34.330886 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:34.330825 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-zndkb\" (UID: \"81291c2c-8fd4-482b-8ed1-c11e9cbb2aad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zndkb" Nov 01 20:39:34.330886 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:34.330853 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-zndkb\" (UID: \"81291c2c-8fd4-482b-8ed1-c11e9cbb2aad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zndkb" Nov 01 20:39:34.330974 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:34.330960 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-zndkb\" (UID: \"81291c2c-8fd4-482b-8ed1-c11e9cbb2aad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zndkb" Nov 01 20:39:34.331020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:34.330981 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-zndkb\" (UID: \"81291c2c-8fd4-482b-8ed1-c11e9cbb2aad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zndkb" Nov 01 20:39:34.331056 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:34.331018 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cncg6\" (UniqueName: \"kubernetes.io/projected/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-kube-api-access-cncg6\") pod \"maas-default-gateway-openshift-default-58c6585447-zndkb\" (UID: \"81291c2c-8fd4-482b-8ed1-c11e9cbb2aad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zndkb" Nov 01 20:39:34.331091 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:34.331081 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-zndkb\" (UID: \"81291c2c-8fd4-482b-8ed1-c11e9cbb2aad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zndkb" Nov 01 20:39:34.331148 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:34.331136 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-zndkb\" (UID: \"81291c2c-8fd4-482b-8ed1-c11e9cbb2aad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zndkb" Nov 01 20:39:34.331183 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:34.331169 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-zndkb\" (UID: \"81291c2c-8fd4-482b-8ed1-c11e9cbb2aad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zndkb" Nov 01 20:39:34.379837 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:34.379804 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zndkb"] Nov 01 20:39:34.432471 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:34.432437 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-zndkb\" (UID: \"81291c2c-8fd4-482b-8ed1-c11e9cbb2aad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zndkb" Nov 01 20:39:34.432681 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:34.432493 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-zndkb\" (UID: \"81291c2c-8fd4-482b-8ed1-c11e9cbb2aad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zndkb" Nov 01 20:39:34.432681 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:34.432521 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-zndkb\" (UID: \"81291c2c-8fd4-482b-8ed1-c11e9cbb2aad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zndkb" Nov 01 20:39:34.432681 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:34.432554 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-zndkb\" (UID: \"81291c2c-8fd4-482b-8ed1-c11e9cbb2aad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zndkb" Nov 01 20:39:34.432681 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:34.432575 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-zndkb\" (UID: \"81291c2c-8fd4-482b-8ed1-c11e9cbb2aad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zndkb" Nov 01 20:39:34.432681 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:34.432610 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-zndkb\" (UID: \"81291c2c-8fd4-482b-8ed1-c11e9cbb2aad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zndkb" Nov 01 20:39:34.432681 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:34.432664 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-zndkb\" (UID: \"81291c2c-8fd4-482b-8ed1-c11e9cbb2aad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zndkb" Nov 01 20:39:34.433015 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:34.432910 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-zndkb\" (UID: \"81291c2c-8fd4-482b-8ed1-c11e9cbb2aad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zndkb" Nov 01 20:39:34.433077 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:34.433017 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-zndkb\" (UID: \"81291c2c-8fd4-482b-8ed1-c11e9cbb2aad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zndkb" Nov 01 20:39:34.433077 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:34.433051 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cncg6\" (UniqueName: \"kubernetes.io/projected/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-kube-api-access-cncg6\") pod \"maas-default-gateway-openshift-default-58c6585447-zndkb\" (UID: \"81291c2c-8fd4-482b-8ed1-c11e9cbb2aad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zndkb" Nov 01 20:39:34.433077 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:34.433052 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-zndkb\" (UID: \"81291c2c-8fd4-482b-8ed1-c11e9cbb2aad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zndkb" Nov 01 20:39:34.433217 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:34.433153 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-zndkb\" (UID: \"81291c2c-8fd4-482b-8ed1-c11e9cbb2aad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zndkb" Nov 01 20:39:34.433217 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:34.433196 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-zndkb\" (UID: \"81291c2c-8fd4-482b-8ed1-c11e9cbb2aad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zndkb" Nov 01 20:39:34.433646 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:34.433622 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-zndkb\" (UID: \"81291c2c-8fd4-482b-8ed1-c11e9cbb2aad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zndkb" Nov 01 20:39:34.434790 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:34.434772 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-zndkb\" (UID: \"81291c2c-8fd4-482b-8ed1-c11e9cbb2aad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zndkb" Nov 01 20:39:34.435053 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:34.435034 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-zndkb\" (UID: \"81291c2c-8fd4-482b-8ed1-c11e9cbb2aad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zndkb" Nov 01 20:39:34.490189 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:34.490155 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-zndkb\" (UID: \"81291c2c-8fd4-482b-8ed1-c11e9cbb2aad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zndkb" Nov 01 20:39:34.490377 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:34.490355 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cncg6\" (UniqueName: \"kubernetes.io/projected/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-kube-api-access-cncg6\") pod \"maas-default-gateway-openshift-default-58c6585447-zndkb\" (UID: \"81291c2c-8fd4-482b-8ed1-c11e9cbb2aad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zndkb" Nov 01 20:39:34.587378 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:34.587296 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zndkb" Nov 01 20:39:34.618896 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:34.618840 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhk6c" event={"ID":"b12e9430-7403-447a-a280-07a09d0e9ce4","Type":"ContainerStarted","Data":"77220518c17bf9e2095dbe3910e6b126e0846e46a54ecd1323854388809f298f"} Nov 01 20:39:34.736765 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:34.736713 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhk6c" podStartSLOduration=2.137458198 podStartE2EDuration="2.736698606s" totalImagesPullingTime="599.240408ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:39:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:39:34.735436639 +0000 UTC m=+525.513076786" watchObservedRunningTime="2026-11-01 20:39:34.736698606 +0000 UTC m=+525.514338741" Nov 01 20:39:34.872287 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:34.872206 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhk6c" Nov 01 20:39:35.009045 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:35.009014 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zndkb"] Nov 01 20:39:35.010766 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:39:35.010739 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81291c2c_8fd4_482b_8ed1_c11e9cbb2aad.slice/crio-2e18a19880dc0b0773ae6ac42ee0184c4e567e0336a498f8490f24f4b40234b6 WatchSource:0}: Error finding container 2e18a19880dc0b0773ae6ac42ee0184c4e567e0336a498f8490f24f4b40234b6: Status 404 returned error can't find the container with id 2e18a19880dc0b0773ae6ac42ee0184c4e567e0336a498f8490f24f4b40234b6 Nov 01 20:39:35.624738 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:35.624703 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zndkb" event={"ID":"81291c2c-8fd4-482b-8ed1-c11e9cbb2aad","Type":"ContainerStarted","Data":"2e18a19880dc0b0773ae6ac42ee0184c4e567e0336a498f8490f24f4b40234b6"} Nov 01 20:39:35.789834 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:35.789791 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:39:36.204859 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:36.204816 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fkcnk" Nov 01 20:39:36.217418 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:36.217385 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fkcnk" Nov 01 20:39:36.640752 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:36.640722 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fkcnk" Nov 01 20:39:42.375832 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:42.375793 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:39:42.376305 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:42.375885 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:39:42.376305 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:42.375945 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:39:42.658168 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:42.658096 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zndkb" event={"ID":"81291c2c-8fd4-482b-8ed1-c11e9cbb2aad","Type":"ContainerStarted","Data":"24d9dfd667fdec153506eb3eb352166ae386e60b9c873ae3e950038b91c3426b"} Nov 01 20:39:42.862808 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:42.862751 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zndkb" podStartSLOduration=1.500972896 podStartE2EDuration="8.862736444s" totalImagesPullingTime="7.361763548s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:39:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:39:42.827523845 +0000 UTC m=+533.605163981" watchObservedRunningTime="2026-11-01 20:39:42.862736444 +0000 UTC m=+533.640376578" Nov 01 20:39:42.871611 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:42.871583 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhk6c" Nov 01 20:39:42.884232 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:42.884199 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhk6c" Nov 01 20:39:43.588395 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:43.588360 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zndkb" Nov 01 20:39:43.589909 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:43.589879 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-zndkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Nov 01 20:39:43.590048 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:43.589943 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zndkb" podUID="81291c2c-8fd4-482b-8ed1-c11e9cbb2aad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Nov 01 20:39:43.673406 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:43.673377 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhk6c" Nov 01 20:39:44.588531 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:44.588491 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-zndkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Nov 01 20:39:44.588925 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:44.588581 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zndkb" podUID="81291c2c-8fd4-482b-8ed1-c11e9cbb2aad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Nov 01 20:39:45.587925 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:45.587890 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-zndkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Nov 01 20:39:45.588136 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:45.587957 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zndkb" podUID="81291c2c-8fd4-482b-8ed1-c11e9cbb2aad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Nov 01 20:39:45.764880 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:39:45.764842 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 20:39:46.588703 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:46.588670 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-zndkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Nov 01 20:39:46.588884 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:46.588736 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zndkb" podUID="81291c2c-8fd4-482b-8ed1-c11e9cbb2aad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Nov 01 20:39:47.588140 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:47.588103 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-zndkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Nov 01 20:39:47.588607 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:47.588157 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zndkb" podUID="81291c2c-8fd4-482b-8ed1-c11e9cbb2aad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Nov 01 20:39:48.588076 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:48.588040 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-zndkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Nov 01 20:39:48.588263 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:48.588095 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zndkb" podUID="81291c2c-8fd4-482b-8ed1-c11e9cbb2aad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Nov 01 20:39:49.475736 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:49.475702 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wwpvf"] Nov 01 20:39:49.482008 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:49.481970 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wwpvf" Nov 01 20:39:49.587832 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:49.587803 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zndkb" Nov 01 20:39:49.588032 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:49.587862 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wwpvf"] Nov 01 20:39:49.588127 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:49.588105 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-zndkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Nov 01 20:39:49.588192 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:49.588171 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zndkb" podUID="81291c2c-8fd4-482b-8ed1-c11e9cbb2aad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Nov 01 20:39:49.588490 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:49.588467 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-wwpvf\" (UID: \"f411ea1f-0db8-4748-87e9-4ae94a7a12c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wwpvf" Nov 01 20:39:49.588800 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:49.588504 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-wwpvf\" (UID: \"f411ea1f-0db8-4748-87e9-4ae94a7a12c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wwpvf" Nov 01 20:39:49.588800 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:49.588616 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-wwpvf\" (UID: \"f411ea1f-0db8-4748-87e9-4ae94a7a12c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wwpvf" Nov 01 20:39:49.588800 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:49.588648 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-wwpvf\" (UID: \"f411ea1f-0db8-4748-87e9-4ae94a7a12c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wwpvf" Nov 01 20:39:49.588800 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:49.588691 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-wwpvf\" (UID: \"f411ea1f-0db8-4748-87e9-4ae94a7a12c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wwpvf" Nov 01 20:39:49.588800 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:49.588740 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2klq\" (UniqueName: \"kubernetes.io/projected/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-kube-api-access-p2klq\") pod \"maas-default-gateway-openshift-default-58c6585447-wwpvf\" (UID: \"f411ea1f-0db8-4748-87e9-4ae94a7a12c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wwpvf" Nov 01 20:39:49.588800 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:49.588765 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-wwpvf\" (UID: \"f411ea1f-0db8-4748-87e9-4ae94a7a12c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wwpvf" Nov 01 20:39:49.588800 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:49.588784 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-wwpvf\" (UID: \"f411ea1f-0db8-4748-87e9-4ae94a7a12c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wwpvf" Nov 01 20:39:49.588800 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:49.588800 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-wwpvf\" (UID: \"f411ea1f-0db8-4748-87e9-4ae94a7a12c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wwpvf" Nov 01 20:39:49.689961 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:49.689915 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-wwpvf\" (UID: \"f411ea1f-0db8-4748-87e9-4ae94a7a12c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wwpvf" Nov 01 20:39:49.690155 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:49.690037 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-wwpvf\" (UID: \"f411ea1f-0db8-4748-87e9-4ae94a7a12c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wwpvf" Nov 01 20:39:49.690155 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:49.690106 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-wwpvf\" (UID: \"f411ea1f-0db8-4748-87e9-4ae94a7a12c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wwpvf" Nov 01 20:39:49.690155 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:49.690148 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-wwpvf\" (UID: \"f411ea1f-0db8-4748-87e9-4ae94a7a12c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wwpvf" Nov 01 20:39:49.690331 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:49.690206 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2klq\" (UniqueName: \"kubernetes.io/projected/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-kube-api-access-p2klq\") pod \"maas-default-gateway-openshift-default-58c6585447-wwpvf\" (UID: \"f411ea1f-0db8-4748-87e9-4ae94a7a12c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wwpvf" Nov 01 20:39:49.690331 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:49.690260 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-wwpvf\" (UID: \"f411ea1f-0db8-4748-87e9-4ae94a7a12c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wwpvf" Nov 01 20:39:49.690331 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:49.690292 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-wwpvf\" (UID: \"f411ea1f-0db8-4748-87e9-4ae94a7a12c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wwpvf" Nov 01 20:39:49.690331 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:49.690320 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-wwpvf\" (UID: \"f411ea1f-0db8-4748-87e9-4ae94a7a12c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wwpvf" Nov 01 20:39:49.690541 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:49.690373 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-wwpvf\" (UID: \"f411ea1f-0db8-4748-87e9-4ae94a7a12c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wwpvf" Nov 01 20:39:49.690541 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:49.690434 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-wwpvf\" (UID: \"f411ea1f-0db8-4748-87e9-4ae94a7a12c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wwpvf" Nov 01 20:39:49.690714 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:49.690692 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-wwpvf\" (UID: \"f411ea1f-0db8-4748-87e9-4ae94a7a12c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wwpvf" Nov 01 20:39:49.690819 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:49.690792 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-wwpvf\" (UID: \"f411ea1f-0db8-4748-87e9-4ae94a7a12c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wwpvf" Nov 01 20:39:49.691005 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:49.690967 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-wwpvf\" (UID: \"f411ea1f-0db8-4748-87e9-4ae94a7a12c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wwpvf" Nov 01 20:39:49.691123 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:49.691100 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-wwpvf\" (UID: \"f411ea1f-0db8-4748-87e9-4ae94a7a12c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wwpvf" Nov 01 20:39:49.692589 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:49.692571 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-wwpvf\" (UID: \"f411ea1f-0db8-4748-87e9-4ae94a7a12c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wwpvf" Nov 01 20:39:49.692876 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:49.692858 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-wwpvf\" (UID: \"f411ea1f-0db8-4748-87e9-4ae94a7a12c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wwpvf" Nov 01 20:39:49.748010 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:49.747964 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-wwpvf\" (UID: \"f411ea1f-0db8-4748-87e9-4ae94a7a12c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wwpvf" Nov 01 20:39:49.748229 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:49.748205 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2klq\" (UniqueName: \"kubernetes.io/projected/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-kube-api-access-p2klq\") pod \"maas-default-gateway-openshift-default-58c6585447-wwpvf\" (UID: \"f411ea1f-0db8-4748-87e9-4ae94a7a12c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wwpvf" Nov 01 20:39:49.795033 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:49.794971 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wwpvf" Nov 01 20:39:50.217893 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:50.217813 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wwpvf"] Nov 01 20:39:50.259336 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:39:50.259294 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf411ea1f_0db8_4748_87e9_4ae94a7a12c4.slice/crio-a0eaa2f0b7e72c67a1192bd7b78916ed7aaeb1c188736a9573d87d28cc92185a WatchSource:0}: Error finding container a0eaa2f0b7e72c67a1192bd7b78916ed7aaeb1c188736a9573d87d28cc92185a: Status 404 returned error can't find the container with id a0eaa2f0b7e72c67a1192bd7b78916ed7aaeb1c188736a9573d87d28cc92185a Nov 01 20:39:50.261829 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:50.261788 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:39:50.261907 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:50.261886 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:39:50.261948 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:50.261931 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:39:50.699824 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:50.699794 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zndkb" Nov 01 20:39:50.701407 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:50.701384 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zndkb" Nov 01 20:39:50.701826 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:50.701802 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wwpvf" event={"ID":"f411ea1f-0db8-4748-87e9-4ae94a7a12c4","Type":"ContainerStarted","Data":"af34fd9250775f247bfdcd4f9169ccc1e6bd82e6ca2cc00fcb321eb322581d1e"} Nov 01 20:39:50.701903 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:50.701854 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wwpvf" event={"ID":"f411ea1f-0db8-4748-87e9-4ae94a7a12c4","Type":"ContainerStarted","Data":"a0eaa2f0b7e72c67a1192bd7b78916ed7aaeb1c188736a9573d87d28cc92185a"} Nov 01 20:39:50.795272 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:50.795223 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wwpvf" Nov 01 20:39:51.062890 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:51.062835 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wwpvf" podStartSLOduration=2.062821042 podStartE2EDuration="2.062821042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:39:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:39:51.061465186 +0000 UTC m=+541.839105363" watchObservedRunningTime="2026-11-01 20:39:51.062821042 +0000 UTC m=+541.840461177" Nov 01 20:39:51.797212 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:51.797178 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-wwpvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Nov 01 20:39:51.797615 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:51.797231 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wwpvf" podUID="f411ea1f-0db8-4748-87e9-4ae94a7a12c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Nov 01 20:39:52.795929 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:52.795889 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-wwpvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Nov 01 20:39:52.796138 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:52.795946 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wwpvf" podUID="f411ea1f-0db8-4748-87e9-4ae94a7a12c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Nov 01 20:39:53.795462 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:53.795425 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-wwpvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Nov 01 20:39:53.795864 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:53.795482 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wwpvf" podUID="f411ea1f-0db8-4748-87e9-4ae94a7a12c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Nov 01 20:39:54.796282 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:54.796243 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-wwpvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Nov 01 20:39:54.796693 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:54.796305 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wwpvf" podUID="f411ea1f-0db8-4748-87e9-4ae94a7a12c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Nov 01 20:39:55.641674 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:55.641634 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:39:55.796079 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:55.796043 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-wwpvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Nov 01 20:39:55.796258 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:55.796099 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wwpvf" podUID="f411ea1f-0db8-4748-87e9-4ae94a7a12c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Nov 01 20:39:56.796201 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:56.796159 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-wwpvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Nov 01 20:39:56.796584 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:56.796226 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wwpvf" podUID="f411ea1f-0db8-4748-87e9-4ae94a7a12c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Nov 01 20:39:58.572112 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:58.572084 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wwpvf" Nov 01 20:39:58.572541 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:58.572379 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wwpvf" Nov 01 20:39:58.573485 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:58.573467 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wwpvf" Nov 01 20:39:59.388536 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:59.388504 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7fbbcb5dbd-t6wcx"] Nov 01 20:39:59.392297 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:59.392277 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7fbbcb5dbd-t6wcx" Nov 01 20:39:59.446184 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:59.446144 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-tmsnc\"" Nov 01 20:39:59.446184 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:59.446164 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 01 20:39:59.447375 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:59.447351 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7fbbcb5dbd-t6wcx"] Nov 01 20:39:59.505058 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:59.505025 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d16d3b45-f23d-486a-bb21-e888963aec8b-tls-cert\") pod \"authorino-7fbbcb5dbd-t6wcx\" (UID: \"d16d3b45-f23d-486a-bb21-e888963aec8b\") " pod="kuadrant-system/authorino-7fbbcb5dbd-t6wcx" Nov 01 20:39:59.505406 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:59.505385 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgkds\" (UniqueName: \"kubernetes.io/projected/d16d3b45-f23d-486a-bb21-e888963aec8b-kube-api-access-fgkds\") pod \"authorino-7fbbcb5dbd-t6wcx\" (UID: \"d16d3b45-f23d-486a-bb21-e888963aec8b\") " pod="kuadrant-system/authorino-7fbbcb5dbd-t6wcx" Nov 01 20:39:59.573829 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:59.573793 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:39:59.606907 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:59.606877 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgkds\" (UniqueName: \"kubernetes.io/projected/d16d3b45-f23d-486a-bb21-e888963aec8b-kube-api-access-fgkds\") pod \"authorino-7fbbcb5dbd-t6wcx\" (UID: \"d16d3b45-f23d-486a-bb21-e888963aec8b\") " pod="kuadrant-system/authorino-7fbbcb5dbd-t6wcx" Nov 01 20:39:59.607094 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:59.606935 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d16d3b45-f23d-486a-bb21-e888963aec8b-tls-cert\") pod \"authorino-7fbbcb5dbd-t6wcx\" (UID: \"d16d3b45-f23d-486a-bb21-e888963aec8b\") " pod="kuadrant-system/authorino-7fbbcb5dbd-t6wcx" Nov 01 20:39:59.609515 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:59.609484 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d16d3b45-f23d-486a-bb21-e888963aec8b-tls-cert\") pod \"authorino-7fbbcb5dbd-t6wcx\" (UID: \"d16d3b45-f23d-486a-bb21-e888963aec8b\") " pod="kuadrant-system/authorino-7fbbcb5dbd-t6wcx" Nov 01 20:39:59.665126 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:59.665051 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgkds\" (UniqueName: \"kubernetes.io/projected/d16d3b45-f23d-486a-bb21-e888963aec8b-kube-api-access-fgkds\") pod \"authorino-7fbbcb5dbd-t6wcx\" (UID: \"d16d3b45-f23d-486a-bb21-e888963aec8b\") " pod="kuadrant-system/authorino-7fbbcb5dbd-t6wcx" Nov 01 20:39:59.704500 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:59.704460 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7fbbcb5dbd-t6wcx" Nov 01 20:39:59.795486 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:39:59.795451 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:00.072674 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:00.072648 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7fbbcb5dbd-t6wcx"] Nov 01 20:40:00.074700 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:40:00.074663 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd16d3b45_f23d_486a_bb21_e888963aec8b.slice/crio-f8ccf2c85187ab1a1a08ee9b18b4f2f317dda96100a653de789c7d21fbb39e47 WatchSource:0}: Error finding container f8ccf2c85187ab1a1a08ee9b18b4f2f317dda96100a653de789c7d21fbb39e47: Status 404 returned error can't find the container with id f8ccf2c85187ab1a1a08ee9b18b4f2f317dda96100a653de789c7d21fbb39e47 Nov 01 20:40:00.779151 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:00.779112 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7fbbcb5dbd-t6wcx" event={"ID":"d16d3b45-f23d-486a-bb21-e888963aec8b","Type":"ContainerStarted","Data":"f8ccf2c85187ab1a1a08ee9b18b4f2f317dda96100a653de789c7d21fbb39e47"} Nov 01 20:40:01.784798 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:01.784758 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7fbbcb5dbd-t6wcx" event={"ID":"d16d3b45-f23d-486a-bb21-e888963aec8b","Type":"ContainerStarted","Data":"73dd0b3c3878ce685033a4061d750a659d058645350fea881a5be714c9648562"} Nov 01 20:40:01.899723 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:01.899669 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-7fbbcb5dbd-t6wcx" podStartSLOduration=1.270997521 podStartE2EDuration="2.899653721s" totalImagesPullingTime="1.6286562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:39:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:40:01.899389547 +0000 UTC m=+552.677029681" watchObservedRunningTime="2026-11-01 20:40:01.899653721 +0000 UTC m=+552.677293856" Nov 01 20:40:04.531783 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:04.531748 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vz6nd"] Nov 01 20:40:04.535265 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:04.535246 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vz6nd" Nov 01 20:40:04.555195 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:04.555158 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59bfe493-3735-4e0f-bfca-03a345fcbeea-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-vz6nd\" (UID: \"59bfe493-3735-4e0f-bfca-03a345fcbeea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vz6nd" Nov 01 20:40:04.555195 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:04.555196 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59bfe493-3735-4e0f-bfca-03a345fcbeea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-vz6nd\" (UID: \"59bfe493-3735-4e0f-bfca-03a345fcbeea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vz6nd" Nov 01 20:40:04.555396 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:04.555295 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59bfe493-3735-4e0f-bfca-03a345fcbeea-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-vz6nd\" (UID: \"59bfe493-3735-4e0f-bfca-03a345fcbeea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vz6nd" Nov 01 20:40:04.555396 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:04.555343 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59bfe493-3735-4e0f-bfca-03a345fcbeea-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-vz6nd\" (UID: \"59bfe493-3735-4e0f-bfca-03a345fcbeea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vz6nd" Nov 01 20:40:04.555396 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:04.555368 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59bfe493-3735-4e0f-bfca-03a345fcbeea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-vz6nd\" (UID: \"59bfe493-3735-4e0f-bfca-03a345fcbeea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vz6nd" Nov 01 20:40:04.555543 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:04.555507 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hbql\" (UniqueName: \"kubernetes.io/projected/59bfe493-3735-4e0f-bfca-03a345fcbeea-kube-api-access-7hbql\") pod \"maas-default-gateway-openshift-default-58c6585447-vz6nd\" (UID: \"59bfe493-3735-4e0f-bfca-03a345fcbeea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vz6nd" Nov 01 20:40:04.555587 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:04.555571 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59bfe493-3735-4e0f-bfca-03a345fcbeea-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-vz6nd\" (UID: \"59bfe493-3735-4e0f-bfca-03a345fcbeea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vz6nd" Nov 01 20:40:04.555623 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:04.555592 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59bfe493-3735-4e0f-bfca-03a345fcbeea-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-vz6nd\" (UID: \"59bfe493-3735-4e0f-bfca-03a345fcbeea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vz6nd" Nov 01 20:40:04.555623 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:04.555608 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59bfe493-3735-4e0f-bfca-03a345fcbeea-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-vz6nd\" (UID: \"59bfe493-3735-4e0f-bfca-03a345fcbeea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vz6nd" Nov 01 20:40:04.656792 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:04.656751 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59bfe493-3735-4e0f-bfca-03a345fcbeea-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-vz6nd\" (UID: \"59bfe493-3735-4e0f-bfca-03a345fcbeea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vz6nd" Nov 01 20:40:04.656792 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:04.656794 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59bfe493-3735-4e0f-bfca-03a345fcbeea-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-vz6nd\" (UID: \"59bfe493-3735-4e0f-bfca-03a345fcbeea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vz6nd" Nov 01 20:40:04.657085 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:04.656914 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59bfe493-3735-4e0f-bfca-03a345fcbeea-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-vz6nd\" (UID: \"59bfe493-3735-4e0f-bfca-03a345fcbeea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vz6nd" Nov 01 20:40:04.657085 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:04.656947 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59bfe493-3735-4e0f-bfca-03a345fcbeea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-vz6nd\" (UID: \"59bfe493-3735-4e0f-bfca-03a345fcbeea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vz6nd" Nov 01 20:40:04.657085 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:04.657018 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59bfe493-3735-4e0f-bfca-03a345fcbeea-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-vz6nd\" (UID: \"59bfe493-3735-4e0f-bfca-03a345fcbeea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vz6nd" Nov 01 20:40:04.657085 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:04.657048 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59bfe493-3735-4e0f-bfca-03a345fcbeea-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-vz6nd\" (UID: \"59bfe493-3735-4e0f-bfca-03a345fcbeea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vz6nd" Nov 01 20:40:04.657309 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:04.657205 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59bfe493-3735-4e0f-bfca-03a345fcbeea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-vz6nd\" (UID: \"59bfe493-3735-4e0f-bfca-03a345fcbeea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vz6nd" Nov 01 20:40:04.657309 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:04.657294 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hbql\" (UniqueName: \"kubernetes.io/projected/59bfe493-3735-4e0f-bfca-03a345fcbeea-kube-api-access-7hbql\") pod \"maas-default-gateway-openshift-default-58c6585447-vz6nd\" (UID: \"59bfe493-3735-4e0f-bfca-03a345fcbeea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vz6nd" Nov 01 20:40:04.657417 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:04.657307 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59bfe493-3735-4e0f-bfca-03a345fcbeea-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-vz6nd\" (UID: \"59bfe493-3735-4e0f-bfca-03a345fcbeea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vz6nd" Nov 01 20:40:04.657417 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:04.657341 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59bfe493-3735-4e0f-bfca-03a345fcbeea-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-vz6nd\" (UID: \"59bfe493-3735-4e0f-bfca-03a345fcbeea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vz6nd" Nov 01 20:40:04.657605 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:04.657577 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59bfe493-3735-4e0f-bfca-03a345fcbeea-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-vz6nd\" (UID: \"59bfe493-3735-4e0f-bfca-03a345fcbeea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vz6nd" Nov 01 20:40:04.657863 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:04.657840 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59bfe493-3735-4e0f-bfca-03a345fcbeea-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-vz6nd\" (UID: \"59bfe493-3735-4e0f-bfca-03a345fcbeea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vz6nd" Nov 01 20:40:04.657863 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:04.657856 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59bfe493-3735-4e0f-bfca-03a345fcbeea-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-vz6nd\" (UID: \"59bfe493-3735-4e0f-bfca-03a345fcbeea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vz6nd" Nov 01 20:40:04.658162 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:04.658131 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59bfe493-3735-4e0f-bfca-03a345fcbeea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-vz6nd\" (UID: \"59bfe493-3735-4e0f-bfca-03a345fcbeea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vz6nd" Nov 01 20:40:04.659544 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:04.659513 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59bfe493-3735-4e0f-bfca-03a345fcbeea-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-vz6nd\" (UID: \"59bfe493-3735-4e0f-bfca-03a345fcbeea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vz6nd" Nov 01 20:40:04.659768 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:04.659751 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59bfe493-3735-4e0f-bfca-03a345fcbeea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-vz6nd\" (UID: \"59bfe493-3735-4e0f-bfca-03a345fcbeea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vz6nd" Nov 01 20:40:04.725016 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:04.724512 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59bfe493-3735-4e0f-bfca-03a345fcbeea-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-vz6nd\" (UID: \"59bfe493-3735-4e0f-bfca-03a345fcbeea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vz6nd" Nov 01 20:40:04.725016 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:04.724717 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hbql\" (UniqueName: \"kubernetes.io/projected/59bfe493-3735-4e0f-bfca-03a345fcbeea-kube-api-access-7hbql\") pod \"maas-default-gateway-openshift-default-58c6585447-vz6nd\" (UID: \"59bfe493-3735-4e0f-bfca-03a345fcbeea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vz6nd" Nov 01 20:40:04.741887 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:04.741834 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vz6nd"] Nov 01 20:40:04.848494 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:04.848405 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vz6nd" Nov 01 20:40:05.281164 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:05.281129 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vz6nd"] Nov 01 20:40:05.284057 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:40:05.284027 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59bfe493_3735_4e0f_bfca_03a345fcbeea.slice/crio-6686861e437c5ca59e699cf18c43239fcb159c0d656da1833b2229527efcd5d0 WatchSource:0}: Error finding container 6686861e437c5ca59e699cf18c43239fcb159c0d656da1833b2229527efcd5d0: Status 404 returned error can't find the container with id 6686861e437c5ca59e699cf18c43239fcb159c0d656da1833b2229527efcd5d0 Nov 01 20:40:05.286322 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:05.286286 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:40:05.286407 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:05.286360 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:40:05.286449 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:05.286419 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:40:05.828756 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:05.828714 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vz6nd" event={"ID":"59bfe493-3735-4e0f-bfca-03a345fcbeea","Type":"ContainerStarted","Data":"1fb90a461fa142b7e8c2bd7efc618ee56daa0fb7b2b7e069f2faeefb62e0c11c"} Nov 01 20:40:05.829238 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:05.828782 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vz6nd" event={"ID":"59bfe493-3735-4e0f-bfca-03a345fcbeea","Type":"ContainerStarted","Data":"6686861e437c5ca59e699cf18c43239fcb159c0d656da1833b2229527efcd5d0"} Nov 01 20:40:05.849169 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:05.849131 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vz6nd" Nov 01 20:40:06.048545 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:06.048469 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vz6nd" podStartSLOduration=2.048446552 podStartE2EDuration="2.048446552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:40:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:40:05.99644915 +0000 UTC m=+556.774089297" watchObservedRunningTime="2026-11-01 20:40:06.048446552 +0000 UTC m=+556.826086687" Nov 01 20:40:06.850687 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:06.850646 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-vz6nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 01 20:40:06.851118 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:06.850714 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vz6nd" podUID="59bfe493-3735-4e0f-bfca-03a345fcbeea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 01 20:40:07.849195 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:07.849159 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-vz6nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 01 20:40:07.849378 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:07.849215 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vz6nd" podUID="59bfe493-3735-4e0f-bfca-03a345fcbeea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 01 20:40:08.849466 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:08.849431 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-vz6nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 01 20:40:08.849859 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:08.849482 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vz6nd" podUID="59bfe493-3735-4e0f-bfca-03a345fcbeea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 01 20:40:09.341503 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:09.341470 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:09.849132 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:09.849099 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-vz6nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 01 20:40:09.849280 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:09.849152 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vz6nd" podUID="59bfe493-3735-4e0f-bfca-03a345fcbeea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 01 20:40:10.848897 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:10.848861 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-vz6nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 01 20:40:10.849608 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:10.849570 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vz6nd" podUID="59bfe493-3735-4e0f-bfca-03a345fcbeea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 01 20:40:11.346966 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:11.346918 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:11.850067 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:11.849611 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-vz6nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 01 20:40:11.850067 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:11.849674 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vz6nd" podUID="59bfe493-3735-4e0f-bfca-03a345fcbeea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 01 20:40:12.849171 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:12.849137 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-vz6nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 01 20:40:12.849375 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:12.849188 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vz6nd" podUID="59bfe493-3735-4e0f-bfca-03a345fcbeea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 01 20:40:14.007240 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:14.007207 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vz6nd" Nov 01 20:40:14.013686 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:14.007775 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vz6nd" Nov 01 20:40:14.013686 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:14.008411 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vz6nd" Nov 01 20:40:15.768581 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:40:15.768533 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 20:40:33.245289 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:33.245210 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:33.775927 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:33.775892 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:35.257733 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:35.257693 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:35.430181 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:35.430148 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:36.135477 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:36.135445 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:36.875492 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:36.875458 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:37.052145 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:37.052110 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:45.764247 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:40:45.764208 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 20:40:48.162713 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:40:48.162681 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:03.913695 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:03.913655 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:04.083604 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.083570 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:04.382470 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.382438 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9"] Nov 01 20:41:04.386287 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.386268 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9" Nov 01 20:41:04.462512 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.462482 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-944263-openshift-default-dockercfg-s5hxs\"" Nov 01 20:41:04.493264 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.493230 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9"] Nov 01 20:41:04.508679 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.508643 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:04.514939 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.514903 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d20fc4c-5175-4154-8669-2494376150ce-istio-envoy\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9\" (UID: \"3d20fc4c-5175-4154-8669-2494376150ce\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9" Nov 01 20:41:04.515208 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.515181 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgwlc\" (UniqueName: \"kubernetes.io/projected/3d20fc4c-5175-4154-8669-2494376150ce-kube-api-access-jgwlc\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9\" (UID: \"3d20fc4c-5175-4154-8669-2494376150ce\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9" Nov 01 20:41:04.515270 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.515234 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d20fc4c-5175-4154-8669-2494376150ce-istiod-ca-cert\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9\" (UID: \"3d20fc4c-5175-4154-8669-2494376150ce\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9" Nov 01 20:41:04.515308 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.515273 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d20fc4c-5175-4154-8669-2494376150ce-istio-token\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9\" (UID: \"3d20fc4c-5175-4154-8669-2494376150ce\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9" Nov 01 20:41:04.515353 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.515311 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d20fc4c-5175-4154-8669-2494376150ce-workload-socket\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9\" (UID: \"3d20fc4c-5175-4154-8669-2494376150ce\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9" Nov 01 20:41:04.515353 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.515342 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d20fc4c-5175-4154-8669-2494376150ce-workload-certs\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9\" (UID: \"3d20fc4c-5175-4154-8669-2494376150ce\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9" Nov 01 20:41:04.515424 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.515375 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d20fc4c-5175-4154-8669-2494376150ce-istio-data\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9\" (UID: \"3d20fc4c-5175-4154-8669-2494376150ce\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9" Nov 01 20:41:04.515424 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.515412 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d20fc4c-5175-4154-8669-2494376150ce-credential-socket\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9\" (UID: \"3d20fc4c-5175-4154-8669-2494376150ce\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9" Nov 01 20:41:04.515500 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.515446 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d20fc4c-5175-4154-8669-2494376150ce-istio-podinfo\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9\" (UID: \"3d20fc4c-5175-4154-8669-2494376150ce\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9" Nov 01 20:41:04.616697 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.616663 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d20fc4c-5175-4154-8669-2494376150ce-istio-envoy\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9\" (UID: \"3d20fc4c-5175-4154-8669-2494376150ce\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9" Nov 01 20:41:04.616880 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.616764 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgwlc\" (UniqueName: \"kubernetes.io/projected/3d20fc4c-5175-4154-8669-2494376150ce-kube-api-access-jgwlc\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9\" (UID: \"3d20fc4c-5175-4154-8669-2494376150ce\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9" Nov 01 20:41:04.616880 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.616788 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d20fc4c-5175-4154-8669-2494376150ce-istiod-ca-cert\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9\" (UID: \"3d20fc4c-5175-4154-8669-2494376150ce\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9" Nov 01 20:41:04.616880 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.616810 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d20fc4c-5175-4154-8669-2494376150ce-istio-token\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9\" (UID: \"3d20fc4c-5175-4154-8669-2494376150ce\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9" Nov 01 20:41:04.616880 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.616844 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d20fc4c-5175-4154-8669-2494376150ce-workload-socket\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9\" (UID: \"3d20fc4c-5175-4154-8669-2494376150ce\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9" Nov 01 20:41:04.616880 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.616873 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d20fc4c-5175-4154-8669-2494376150ce-workload-certs\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9\" (UID: \"3d20fc4c-5175-4154-8669-2494376150ce\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9" Nov 01 20:41:04.617226 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.616896 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d20fc4c-5175-4154-8669-2494376150ce-istio-data\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9\" (UID: \"3d20fc4c-5175-4154-8669-2494376150ce\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9" Nov 01 20:41:04.617226 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.616933 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d20fc4c-5175-4154-8669-2494376150ce-credential-socket\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9\" (UID: \"3d20fc4c-5175-4154-8669-2494376150ce\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9" Nov 01 20:41:04.617226 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.616961 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d20fc4c-5175-4154-8669-2494376150ce-istio-podinfo\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9\" (UID: \"3d20fc4c-5175-4154-8669-2494376150ce\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9" Nov 01 20:41:04.617399 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.617368 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d20fc4c-5175-4154-8669-2494376150ce-workload-socket\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9\" (UID: \"3d20fc4c-5175-4154-8669-2494376150ce\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9" Nov 01 20:41:04.617463 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.617433 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d20fc4c-5175-4154-8669-2494376150ce-workload-certs\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9\" (UID: \"3d20fc4c-5175-4154-8669-2494376150ce\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9" Nov 01 20:41:04.617557 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.617521 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d20fc4c-5175-4154-8669-2494376150ce-istio-data\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9\" (UID: \"3d20fc4c-5175-4154-8669-2494376150ce\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9" Nov 01 20:41:04.617641 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.617619 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d20fc4c-5175-4154-8669-2494376150ce-istiod-ca-cert\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9\" (UID: \"3d20fc4c-5175-4154-8669-2494376150ce\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9" Nov 01 20:41:04.617693 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.617654 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d20fc4c-5175-4154-8669-2494376150ce-credential-socket\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9\" (UID: \"3d20fc4c-5175-4154-8669-2494376150ce\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9" Nov 01 20:41:04.619109 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.619085 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d20fc4c-5175-4154-8669-2494376150ce-istio-envoy\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9\" (UID: \"3d20fc4c-5175-4154-8669-2494376150ce\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9" Nov 01 20:41:04.619461 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.619439 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d20fc4c-5175-4154-8669-2494376150ce-istio-podinfo\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9\" (UID: \"3d20fc4c-5175-4154-8669-2494376150ce\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9" Nov 01 20:41:04.674789 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.674715 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgwlc\" (UniqueName: \"kubernetes.io/projected/3d20fc4c-5175-4154-8669-2494376150ce-kube-api-access-jgwlc\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9\" (UID: \"3d20fc4c-5175-4154-8669-2494376150ce\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9" Nov 01 20:41:04.674937 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.674811 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d20fc4c-5175-4154-8669-2494376150ce-istio-token\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9\" (UID: \"3d20fc4c-5175-4154-8669-2494376150ce\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9" Nov 01 20:41:04.699478 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.699442 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9" Nov 01 20:41:04.708237 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.708211 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv"] Nov 01 20:41:04.712605 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.712581 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv" Nov 01 20:41:04.769133 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.769104 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-cd248f-openshift-default-dockercfg-6v4rd\"" Nov 01 20:41:04.775133 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.771219 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv"] Nov 01 20:41:04.828490 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.828450 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ae0b2c8-270b-4a3f-bb16-523874681ade-istiod-ca-cert\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv\" (UID: \"3ae0b2c8-270b-4a3f-bb16-523874681ade\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv" Nov 01 20:41:04.828692 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.828513 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ae0b2c8-270b-4a3f-bb16-523874681ade-credential-socket\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv\" (UID: \"3ae0b2c8-270b-4a3f-bb16-523874681ade\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv" Nov 01 20:41:04.828692 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.828641 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ae0b2c8-270b-4a3f-bb16-523874681ade-istio-token\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv\" (UID: \"3ae0b2c8-270b-4a3f-bb16-523874681ade\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv" Nov 01 20:41:04.828804 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.828744 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ae0b2c8-270b-4a3f-bb16-523874681ade-workload-socket\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv\" (UID: \"3ae0b2c8-270b-4a3f-bb16-523874681ade\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv" Nov 01 20:41:04.828855 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.828801 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ae0b2c8-270b-4a3f-bb16-523874681ade-istio-data\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv\" (UID: \"3ae0b2c8-270b-4a3f-bb16-523874681ade\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv" Nov 01 20:41:04.828855 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.828820 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ae0b2c8-270b-4a3f-bb16-523874681ade-workload-certs\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv\" (UID: \"3ae0b2c8-270b-4a3f-bb16-523874681ade\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv" Nov 01 20:41:04.828855 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.828837 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ae0b2c8-270b-4a3f-bb16-523874681ade-istio-envoy\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv\" (UID: \"3ae0b2c8-270b-4a3f-bb16-523874681ade\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv" Nov 01 20:41:04.829027 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.828939 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5dj8\" (UniqueName: \"kubernetes.io/projected/3ae0b2c8-270b-4a3f-bb16-523874681ade-kube-api-access-c5dj8\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv\" (UID: \"3ae0b2c8-270b-4a3f-bb16-523874681ade\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv" Nov 01 20:41:04.829027 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.828961 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ae0b2c8-270b-4a3f-bb16-523874681ade-istio-podinfo\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv\" (UID: \"3ae0b2c8-270b-4a3f-bb16-523874681ade\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv" Nov 01 20:41:04.929862 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.929762 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5dj8\" (UniqueName: \"kubernetes.io/projected/3ae0b2c8-270b-4a3f-bb16-523874681ade-kube-api-access-c5dj8\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv\" (UID: \"3ae0b2c8-270b-4a3f-bb16-523874681ade\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv" Nov 01 20:41:04.929862 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.929804 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ae0b2c8-270b-4a3f-bb16-523874681ade-istio-podinfo\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv\" (UID: \"3ae0b2c8-270b-4a3f-bb16-523874681ade\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv" Nov 01 20:41:04.929862 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.929834 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ae0b2c8-270b-4a3f-bb16-523874681ade-istiod-ca-cert\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv\" (UID: \"3ae0b2c8-270b-4a3f-bb16-523874681ade\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv" Nov 01 20:41:04.930513 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.929951 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ae0b2c8-270b-4a3f-bb16-523874681ade-credential-socket\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv\" (UID: \"3ae0b2c8-270b-4a3f-bb16-523874681ade\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv" Nov 01 20:41:04.930513 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.930043 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ae0b2c8-270b-4a3f-bb16-523874681ade-istio-token\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv\" (UID: \"3ae0b2c8-270b-4a3f-bb16-523874681ade\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv" Nov 01 20:41:04.930513 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.930112 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ae0b2c8-270b-4a3f-bb16-523874681ade-workload-socket\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv\" (UID: \"3ae0b2c8-270b-4a3f-bb16-523874681ade\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv" Nov 01 20:41:04.930513 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.930164 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ae0b2c8-270b-4a3f-bb16-523874681ade-istio-data\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv\" (UID: \"3ae0b2c8-270b-4a3f-bb16-523874681ade\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv" Nov 01 20:41:04.930513 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.930192 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ae0b2c8-270b-4a3f-bb16-523874681ade-workload-certs\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv\" (UID: \"3ae0b2c8-270b-4a3f-bb16-523874681ade\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv" Nov 01 20:41:04.930513 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.930222 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ae0b2c8-270b-4a3f-bb16-523874681ade-istio-envoy\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv\" (UID: \"3ae0b2c8-270b-4a3f-bb16-523874681ade\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv" Nov 01 20:41:04.930513 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.930408 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ae0b2c8-270b-4a3f-bb16-523874681ade-credential-socket\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv\" (UID: \"3ae0b2c8-270b-4a3f-bb16-523874681ade\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv" Nov 01 20:41:04.930795 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.930522 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ae0b2c8-270b-4a3f-bb16-523874681ade-workload-socket\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv\" (UID: \"3ae0b2c8-270b-4a3f-bb16-523874681ade\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv" Nov 01 20:41:04.930795 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.930567 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ae0b2c8-270b-4a3f-bb16-523874681ade-istiod-ca-cert\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv\" (UID: \"3ae0b2c8-270b-4a3f-bb16-523874681ade\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv" Nov 01 20:41:04.930795 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.930659 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ae0b2c8-270b-4a3f-bb16-523874681ade-istio-data\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv\" (UID: \"3ae0b2c8-270b-4a3f-bb16-523874681ade\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv" Nov 01 20:41:04.930795 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.930738 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ae0b2c8-270b-4a3f-bb16-523874681ade-workload-certs\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv\" (UID: \"3ae0b2c8-270b-4a3f-bb16-523874681ade\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv" Nov 01 20:41:04.932614 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.932583 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ae0b2c8-270b-4a3f-bb16-523874681ade-istio-envoy\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv\" (UID: \"3ae0b2c8-270b-4a3f-bb16-523874681ade\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv" Nov 01 20:41:04.932820 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.932804 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ae0b2c8-270b-4a3f-bb16-523874681ade-istio-podinfo\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv\" (UID: \"3ae0b2c8-270b-4a3f-bb16-523874681ade\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv" Nov 01 20:41:04.990753 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.990717 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5dj8\" (UniqueName: \"kubernetes.io/projected/3ae0b2c8-270b-4a3f-bb16-523874681ade-kube-api-access-c5dj8\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv\" (UID: \"3ae0b2c8-270b-4a3f-bb16-523874681ade\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv" Nov 01 20:41:04.990930 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:04.990909 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ae0b2c8-270b-4a3f-bb16-523874681ade-istio-token\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv\" (UID: \"3ae0b2c8-270b-4a3f-bb16-523874681ade\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv" Nov 01 20:41:05.051045 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.049638 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv" Nov 01 20:41:05.204081 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.186453 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9"] Nov 01 20:41:05.204081 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.197457 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:41:05.204081 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.197552 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:41:05.204081 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.197595 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:41:05.222764 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.222722 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:05.346833 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.346806 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr"] Nov 01 20:41:05.350564 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.350541 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr" Nov 01 20:41:05.384284 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.384246 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:05.409200 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.409150 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-db0157-openshift-default-dockercfg-gx5m8\"" Nov 01 20:41:05.411886 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.411861 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr"] Nov 01 20:41:05.435823 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.435784 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35f67043-6238-4386-a37a-3c2dbed1fae9-workload-certs\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr\" (UID: \"35f67043-6238-4386-a37a-3c2dbed1fae9\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr" Nov 01 20:41:05.436057 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.435853 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35f67043-6238-4386-a37a-3c2dbed1fae9-istio-podinfo\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr\" (UID: \"35f67043-6238-4386-a37a-3c2dbed1fae9\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr" Nov 01 20:41:05.436057 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.435891 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35f67043-6238-4386-a37a-3c2dbed1fae9-istio-envoy\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr\" (UID: \"35f67043-6238-4386-a37a-3c2dbed1fae9\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr" Nov 01 20:41:05.436057 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.435943 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35f67043-6238-4386-a37a-3c2dbed1fae9-istio-data\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr\" (UID: \"35f67043-6238-4386-a37a-3c2dbed1fae9\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr" Nov 01 20:41:05.436057 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.436029 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f67043-6238-4386-a37a-3c2dbed1fae9-credential-socket\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr\" (UID: \"35f67043-6238-4386-a37a-3c2dbed1fae9\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr" Nov 01 20:41:05.436304 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.436062 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f67043-6238-4386-a37a-3c2dbed1fae9-workload-socket\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr\" (UID: \"35f67043-6238-4386-a37a-3c2dbed1fae9\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr" Nov 01 20:41:05.436304 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.436092 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35f67043-6238-4386-a37a-3c2dbed1fae9-istiod-ca-cert\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr\" (UID: \"35f67043-6238-4386-a37a-3c2dbed1fae9\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr" Nov 01 20:41:05.436304 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.436176 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35f67043-6238-4386-a37a-3c2dbed1fae9-istio-token\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr\" (UID: \"35f67043-6238-4386-a37a-3c2dbed1fae9\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr" Nov 01 20:41:05.436304 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.436218 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktmxx\" (UniqueName: \"kubernetes.io/projected/35f67043-6238-4386-a37a-3c2dbed1fae9-kube-api-access-ktmxx\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr\" (UID: \"35f67043-6238-4386-a37a-3c2dbed1fae9\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr" Nov 01 20:41:05.473848 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.473768 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv"] Nov 01 20:41:05.486192 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:41:05.486131 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ae0b2c8_270b_4a3f_bb16_523874681ade.slice/crio-a38be239ccd76dd09887dc7e0567072b542401cbcfdbc0c73539e5b2d8dc76b3 WatchSource:0}: Error finding container a38be239ccd76dd09887dc7e0567072b542401cbcfdbc0c73539e5b2d8dc76b3: Status 404 returned error can't find the container with id a38be239ccd76dd09887dc7e0567072b542401cbcfdbc0c73539e5b2d8dc76b3 Nov 01 20:41:05.488130 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.488095 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:41:05.488207 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.488180 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:41:05.488250 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.488219 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:41:05.542457 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.538686 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35f67043-6238-4386-a37a-3c2dbed1fae9-istio-envoy\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr\" (UID: \"35f67043-6238-4386-a37a-3c2dbed1fae9\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr" Nov 01 20:41:05.542457 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.538772 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35f67043-6238-4386-a37a-3c2dbed1fae9-istio-data\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr\" (UID: \"35f67043-6238-4386-a37a-3c2dbed1fae9\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr" Nov 01 20:41:05.542457 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.538866 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f67043-6238-4386-a37a-3c2dbed1fae9-credential-socket\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr\" (UID: \"35f67043-6238-4386-a37a-3c2dbed1fae9\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr" Nov 01 20:41:05.542457 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.538915 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f67043-6238-4386-a37a-3c2dbed1fae9-workload-socket\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr\" (UID: \"35f67043-6238-4386-a37a-3c2dbed1fae9\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr" Nov 01 20:41:05.542457 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.538942 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35f67043-6238-4386-a37a-3c2dbed1fae9-istiod-ca-cert\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr\" (UID: \"35f67043-6238-4386-a37a-3c2dbed1fae9\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr" Nov 01 20:41:05.542457 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.539041 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35f67043-6238-4386-a37a-3c2dbed1fae9-istio-token\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr\" (UID: \"35f67043-6238-4386-a37a-3c2dbed1fae9\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr" Nov 01 20:41:05.542457 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.539094 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktmxx\" (UniqueName: \"kubernetes.io/projected/35f67043-6238-4386-a37a-3c2dbed1fae9-kube-api-access-ktmxx\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr\" (UID: \"35f67043-6238-4386-a37a-3c2dbed1fae9\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr" Nov 01 20:41:05.542457 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.539126 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35f67043-6238-4386-a37a-3c2dbed1fae9-workload-certs\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr\" (UID: \"35f67043-6238-4386-a37a-3c2dbed1fae9\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr" Nov 01 20:41:05.542457 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.539183 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35f67043-6238-4386-a37a-3c2dbed1fae9-istio-podinfo\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr\" (UID: \"35f67043-6238-4386-a37a-3c2dbed1fae9\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr" Nov 01 20:41:05.543104 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.542579 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35f67043-6238-4386-a37a-3c2dbed1fae9-istio-data\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr\" (UID: \"35f67043-6238-4386-a37a-3c2dbed1fae9\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr" Nov 01 20:41:05.543104 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.542849 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f67043-6238-4386-a37a-3c2dbed1fae9-credential-socket\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr\" (UID: \"35f67043-6238-4386-a37a-3c2dbed1fae9\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr" Nov 01 20:41:05.543252 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.543133 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f67043-6238-4386-a37a-3c2dbed1fae9-workload-socket\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr\" (UID: \"35f67043-6238-4386-a37a-3c2dbed1fae9\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr" Nov 01 20:41:05.543870 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.543844 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35f67043-6238-4386-a37a-3c2dbed1fae9-istiod-ca-cert\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr\" (UID: \"35f67043-6238-4386-a37a-3c2dbed1fae9\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr" Nov 01 20:41:05.544460 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.544418 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35f67043-6238-4386-a37a-3c2dbed1fae9-workload-certs\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr\" (UID: \"35f67043-6238-4386-a37a-3c2dbed1fae9\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr" Nov 01 20:41:05.544567 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.544544 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35f67043-6238-4386-a37a-3c2dbed1fae9-istio-envoy\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr\" (UID: \"35f67043-6238-4386-a37a-3c2dbed1fae9\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr" Nov 01 20:41:05.545167 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.545125 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35f67043-6238-4386-a37a-3c2dbed1fae9-istio-podinfo\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr\" (UID: \"35f67043-6238-4386-a37a-3c2dbed1fae9\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr" Nov 01 20:41:05.554968 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.554924 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:05.603704 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.602066 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35f67043-6238-4386-a37a-3c2dbed1fae9-istio-token\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr\" (UID: \"35f67043-6238-4386-a37a-3c2dbed1fae9\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr" Nov 01 20:41:05.603704 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.602478 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktmxx\" (UniqueName: \"kubernetes.io/projected/35f67043-6238-4386-a37a-3c2dbed1fae9-kube-api-access-ktmxx\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr\" (UID: \"35f67043-6238-4386-a37a-3c2dbed1fae9\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr" Nov 01 20:41:05.653006 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.652959 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw"] Nov 01 20:41:05.656716 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.656696 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw" Nov 01 20:41:05.664633 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.664609 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr" Nov 01 20:41:05.718021 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.717342 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw"] Nov 01 20:41:05.742020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.741691 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9c302ea-168a-4fa6-885f-2a3263835c79-workload-certs\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw\" (UID: \"e9c302ea-168a-4fa6-885f-2a3263835c79\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw" Nov 01 20:41:05.742020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.741774 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9c302ea-168a-4fa6-885f-2a3263835c79-istio-data\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw\" (UID: \"e9c302ea-168a-4fa6-885f-2a3263835c79\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw" Nov 01 20:41:05.742020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.741827 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9c302ea-168a-4fa6-885f-2a3263835c79-istio-token\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw\" (UID: \"e9c302ea-168a-4fa6-885f-2a3263835c79\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw" Nov 01 20:41:05.742020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.741869 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9c302ea-168a-4fa6-885f-2a3263835c79-workload-socket\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw\" (UID: \"e9c302ea-168a-4fa6-885f-2a3263835c79\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw" Nov 01 20:41:05.742020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.741896 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9c302ea-168a-4fa6-885f-2a3263835c79-credential-socket\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw\" (UID: \"e9c302ea-168a-4fa6-885f-2a3263835c79\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw" Nov 01 20:41:05.742020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.741934 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9c302ea-168a-4fa6-885f-2a3263835c79-istio-podinfo\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw\" (UID: \"e9c302ea-168a-4fa6-885f-2a3263835c79\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw" Nov 01 20:41:05.742020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.741960 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9c302ea-168a-4fa6-885f-2a3263835c79-istio-envoy\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw\" (UID: \"e9c302ea-168a-4fa6-885f-2a3263835c79\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw" Nov 01 20:41:05.743894 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.742519 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9c302ea-168a-4fa6-885f-2a3263835c79-istiod-ca-cert\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw\" (UID: \"e9c302ea-168a-4fa6-885f-2a3263835c79\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw" Nov 01 20:41:05.743894 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.742673 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nkkg\" (UniqueName: \"kubernetes.io/projected/e9c302ea-168a-4fa6-885f-2a3263835c79-kube-api-access-6nkkg\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw\" (UID: \"e9c302ea-168a-4fa6-885f-2a3263835c79\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw" Nov 01 20:41:05.768650 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.768615 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-208f2b-openshift-default-dockercfg-x8lc4\"" Nov 01 20:41:05.777113 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.777076 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:05.843821 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.843774 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9c302ea-168a-4fa6-885f-2a3263835c79-istio-podinfo\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw\" (UID: \"e9c302ea-168a-4fa6-885f-2a3263835c79\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw" Nov 01 20:41:05.844029 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.843838 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9c302ea-168a-4fa6-885f-2a3263835c79-istio-envoy\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw\" (UID: \"e9c302ea-168a-4fa6-885f-2a3263835c79\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw" Nov 01 20:41:05.844029 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.843887 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9c302ea-168a-4fa6-885f-2a3263835c79-istiod-ca-cert\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw\" (UID: \"e9c302ea-168a-4fa6-885f-2a3263835c79\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw" Nov 01 20:41:05.844029 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.843944 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nkkg\" (UniqueName: \"kubernetes.io/projected/e9c302ea-168a-4fa6-885f-2a3263835c79-kube-api-access-6nkkg\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw\" (UID: \"e9c302ea-168a-4fa6-885f-2a3263835c79\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw" Nov 01 20:41:05.844238 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.844038 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9c302ea-168a-4fa6-885f-2a3263835c79-workload-certs\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw\" (UID: \"e9c302ea-168a-4fa6-885f-2a3263835c79\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw" Nov 01 20:41:05.844238 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.844110 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9c302ea-168a-4fa6-885f-2a3263835c79-istio-data\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw\" (UID: \"e9c302ea-168a-4fa6-885f-2a3263835c79\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw" Nov 01 20:41:05.844238 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.844154 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9c302ea-168a-4fa6-885f-2a3263835c79-istio-token\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw\" (UID: \"e9c302ea-168a-4fa6-885f-2a3263835c79\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw" Nov 01 20:41:05.844238 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.844204 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9c302ea-168a-4fa6-885f-2a3263835c79-workload-socket\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw\" (UID: \"e9c302ea-168a-4fa6-885f-2a3263835c79\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw" Nov 01 20:41:05.844238 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.844234 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9c302ea-168a-4fa6-885f-2a3263835c79-credential-socket\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw\" (UID: \"e9c302ea-168a-4fa6-885f-2a3263835c79\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw" Nov 01 20:41:05.845507 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.844697 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9c302ea-168a-4fa6-885f-2a3263835c79-credential-socket\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw\" (UID: \"e9c302ea-168a-4fa6-885f-2a3263835c79\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw" Nov 01 20:41:05.846534 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.845988 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9c302ea-168a-4fa6-885f-2a3263835c79-istiod-ca-cert\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw\" (UID: \"e9c302ea-168a-4fa6-885f-2a3263835c79\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw" Nov 01 20:41:05.846643 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.846610 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9c302ea-168a-4fa6-885f-2a3263835c79-istio-data\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw\" (UID: \"e9c302ea-168a-4fa6-885f-2a3263835c79\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw" Nov 01 20:41:05.846713 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.846680 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9c302ea-168a-4fa6-885f-2a3263835c79-workload-socket\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw\" (UID: \"e9c302ea-168a-4fa6-885f-2a3263835c79\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw" Nov 01 20:41:05.847128 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.847105 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9c302ea-168a-4fa6-885f-2a3263835c79-workload-certs\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw\" (UID: \"e9c302ea-168a-4fa6-885f-2a3263835c79\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw" Nov 01 20:41:05.858986 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.858829 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9c302ea-168a-4fa6-885f-2a3263835c79-istio-envoy\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw\" (UID: \"e9c302ea-168a-4fa6-885f-2a3263835c79\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw" Nov 01 20:41:05.860109 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.860078 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9c302ea-168a-4fa6-885f-2a3263835c79-istio-podinfo\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw\" (UID: \"e9c302ea-168a-4fa6-885f-2a3263835c79\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw" Nov 01 20:41:05.904617 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.904582 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9c302ea-168a-4fa6-885f-2a3263835c79-istio-token\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw\" (UID: \"e9c302ea-168a-4fa6-885f-2a3263835c79\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw" Nov 01 20:41:05.904769 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.904583 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nkkg\" (UniqueName: \"kubernetes.io/projected/e9c302ea-168a-4fa6-885f-2a3263835c79-kube-api-access-6nkkg\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw\" (UID: \"e9c302ea-168a-4fa6-885f-2a3263835c79\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw" Nov 01 20:41:05.945593 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.945557 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:05.973570 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:05.973536 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw" Nov 01 20:41:06.070424 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:06.070394 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr"] Nov 01 20:41:06.085024 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:41:06.080132 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35f67043_6238_4386_a37a_3c2dbed1fae9.slice/crio-2cb0d83910ce159451bf940ebc68d0f121925c28c04b269f78788a7a2d96f276 WatchSource:0}: Error finding container 2cb0d83910ce159451bf940ebc68d0f121925c28c04b269f78788a7a2d96f276: Status 404 returned error can't find the container with id 2cb0d83910ce159451bf940ebc68d0f121925c28c04b269f78788a7a2d96f276 Nov 01 20:41:06.085024 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:06.081968 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:41:06.085024 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:06.082063 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:41:06.085024 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:06.082111 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:41:06.117405 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:06.117306 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:06.118423 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:06.118396 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr" event={"ID":"35f67043-6238-4386-a37a-3c2dbed1fae9","Type":"ContainerStarted","Data":"2cb0d83910ce159451bf940ebc68d0f121925c28c04b269f78788a7a2d96f276"} Nov 01 20:41:06.124177 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:06.121715 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv" event={"ID":"3ae0b2c8-270b-4a3f-bb16-523874681ade","Type":"ContainerStarted","Data":"5f3068fefdcfccb633c8492083fc1e3420cd25dc941ed3797c287eeaadc6303d"} Nov 01 20:41:06.124177 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:06.121765 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv" event={"ID":"3ae0b2c8-270b-4a3f-bb16-523874681ade","Type":"ContainerStarted","Data":"a38be239ccd76dd09887dc7e0567072b542401cbcfdbc0c73539e5b2d8dc76b3"} Nov 01 20:41:06.127046 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:06.126977 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9" event={"ID":"3d20fc4c-5175-4154-8669-2494376150ce","Type":"ContainerStarted","Data":"e6568ce5159b31428ae7351e04d78dadeaf28e26e6db11e199f4feb8af86d852"} Nov 01 20:41:06.127179 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:06.127080 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9" event={"ID":"3d20fc4c-5175-4154-8669-2494376150ce","Type":"ContainerStarted","Data":"2af5f98367dc2e08af7317fc7c30131c3c9e2c4ec75846299cf4ea4b2663198b"} Nov 01 20:41:06.245318 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:06.245256 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv" podStartSLOduration=2.245236053 podStartE2EDuration="2.245236053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:41:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:41:06.243670338 +0000 UTC m=+617.021310476" watchObservedRunningTime="2026-11-01 20:41:06.245236053 +0000 UTC m=+617.022876193" Nov 01 20:41:06.281471 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:06.281385 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:06.361404 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:06.361380 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw"] Nov 01 20:41:06.363399 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:41:06.363361 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9c302ea_168a_4fa6_885f_2a3263835c79.slice/crio-b4ed7865476bbf19cbad5b1be56863705b025aa812967d7a304be1c1e92714cb WatchSource:0}: Error finding container b4ed7865476bbf19cbad5b1be56863705b025aa812967d7a304be1c1e92714cb: Status 404 returned error can't find the container with id b4ed7865476bbf19cbad5b1be56863705b025aa812967d7a304be1c1e92714cb Nov 01 20:41:06.370921 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:06.370858 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:41:06.388451 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:06.386144 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:41:06.388451 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:06.386220 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:41:06.388451 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:06.371388 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9" podStartSLOduration=2.371370585 podStartE2EDuration="2.371370585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:41:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:41:06.369151268 +0000 UTC m=+617.146791422" watchObservedRunningTime="2026-11-01 20:41:06.371370585 +0000 UTC m=+617.149010722" Nov 01 20:41:06.465004 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:06.464967 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:06.617392 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:06.617308 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:06.700448 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:06.700404 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9" Nov 01 20:41:06.706124 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:06.706093 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9" Nov 01 20:41:06.774135 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:06.774105 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:07.051034 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:07.050985 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv" Nov 01 20:41:07.056355 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:07.056322 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv" Nov 01 20:41:07.133526 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:07.133489 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr" event={"ID":"35f67043-6238-4386-a37a-3c2dbed1fae9","Type":"ContainerStarted","Data":"61b1906e39baee0d36b4f4ec168741c9a02e049517234025034b92d856b82356"} Nov 01 20:41:07.135153 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:07.135122 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw" event={"ID":"e9c302ea-168a-4fa6-885f-2a3263835c79","Type":"ContainerStarted","Data":"297e2449b94efbf6299679b7996c8245b7492d28246c60a767b6a54e6716a790"} Nov 01 20:41:07.135294 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:07.135173 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw" event={"ID":"e9c302ea-168a-4fa6-885f-2a3263835c79","Type":"ContainerStarted","Data":"b4ed7865476bbf19cbad5b1be56863705b025aa812967d7a304be1c1e92714cb"} Nov 01 20:41:07.135677 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:07.135562 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9" Nov 01 20:41:07.135677 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:07.135611 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv" Nov 01 20:41:07.136484 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:07.136461 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9" Nov 01 20:41:07.136583 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:07.136515 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv" Nov 01 20:41:07.297612 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:07.297544 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr" podStartSLOduration=2.297525663 podStartE2EDuration="2.297525663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:41:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:41:07.294980071 +0000 UTC m=+618.072620207" watchObservedRunningTime="2026-11-01 20:41:07.297525663 +0000 UTC m=+618.075165799" Nov 01 20:41:07.424531 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:07.421160 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw" podStartSLOduration=2.421137733 podStartE2EDuration="2.421137733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:41:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:41:07.418353445 +0000 UTC m=+618.195993583" watchObservedRunningTime="2026-11-01 20:41:07.421137733 +0000 UTC m=+618.198777884" Nov 01 20:41:07.665753 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:07.665714 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr" Nov 01 20:41:07.676637 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:07.676549 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr" Nov 01 20:41:07.974394 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:07.974310 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw" Nov 01 20:41:08.013038 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:08.012364 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw" Nov 01 20:41:08.141313 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:08.141281 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr" Nov 01 20:41:08.141730 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:08.141342 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw" Nov 01 20:41:08.142029 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:08.141988 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw" Nov 01 20:41:08.142362 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:08.142347 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr" Nov 01 20:41:09.279200 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:09.279141 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:09.440716 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:09.440674 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:09.610403 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:09.610318 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:09.930504 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:09.930426 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:10.112455 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:10.112420 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:10.290944 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:10.290914 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:11.015636 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:11.015594 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:11.179834 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:11.179795 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:11.563503 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:11.563471 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:11.748162 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:11.748130 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:11.910153 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:11.910080 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:12.171591 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:12.171503 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:12.338805 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:12.338773 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:12.540705 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:12.540659 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:14.417035 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:14.416980 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:14.588744 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:14.588710 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:14.956351 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:14.956315 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-cd248f-6c9889fc9c-wbd52"] Nov 01 20:41:14.959989 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:14.959971 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-cd248f-6c9889fc9c-wbd52" Nov 01 20:41:14.972269 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:14.972243 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69c9a660-98c7-436f-8bee-b7781c596d0c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-cd248f-6c9889fc9c-wbd52\" (UID: \"69c9a660-98c7-436f-8bee-b7781c596d0c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-cd248f-6c9889fc9c-wbd52" Nov 01 20:41:14.972397 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:14.972282 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvqxb\" (UniqueName: \"kubernetes.io/projected/69c9a660-98c7-436f-8bee-b7781c596d0c-kube-api-access-jvqxb\") pod \"payload-processing-e2e-worker-w2-cd248f-6c9889fc9c-wbd52\" (UID: \"69c9a660-98c7-436f-8bee-b7781c596d0c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-cd248f-6c9889fc9c-wbd52" Nov 01 20:41:14.988620 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:14.988592 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:15.013478 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:15.013446 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-cd248f\"" Nov 01 20:41:15.064679 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:15.064646 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-cd248f-dockercfg-xvs4n\"" Nov 01 20:41:15.066624 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:15.066601 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-cd248f-6c9889fc9c-wbd52"] Nov 01 20:41:15.073161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:15.073135 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69c9a660-98c7-436f-8bee-b7781c596d0c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-cd248f-6c9889fc9c-wbd52\" (UID: \"69c9a660-98c7-436f-8bee-b7781c596d0c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-cd248f-6c9889fc9c-wbd52" Nov 01 20:41:15.073267 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:15.073172 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvqxb\" (UniqueName: \"kubernetes.io/projected/69c9a660-98c7-436f-8bee-b7781c596d0c-kube-api-access-jvqxb\") pod \"payload-processing-e2e-worker-w2-cd248f-6c9889fc9c-wbd52\" (UID: \"69c9a660-98c7-436f-8bee-b7781c596d0c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-cd248f-6c9889fc9c-wbd52" Nov 01 20:41:15.073833 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:15.073812 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69c9a660-98c7-436f-8bee-b7781c596d0c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-cd248f-6c9889fc9c-wbd52\" (UID: \"69c9a660-98c7-436f-8bee-b7781c596d0c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-cd248f-6c9889fc9c-wbd52" Nov 01 20:41:15.142269 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:15.142237 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvqxb\" (UniqueName: \"kubernetes.io/projected/69c9a660-98c7-436f-8bee-b7781c596d0c-kube-api-access-jvqxb\") pod \"payload-processing-e2e-worker-w2-cd248f-6c9889fc9c-wbd52\" (UID: \"69c9a660-98c7-436f-8bee-b7781c596d0c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-cd248f-6c9889fc9c-wbd52" Nov 01 20:41:15.207679 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:15.207587 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:15.274269 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:15.274229 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-cd248f-6c9889fc9c-wbd52" Nov 01 20:41:15.375148 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:15.375121 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:15.644297 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:15.644268 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-cd248f-6c9889fc9c-wbd52"] Nov 01 20:41:15.647888 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:41:15.647849 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69c9a660_98c7_436f_8bee_b7781c596d0c.slice/crio-5503f03feb4db749e9e6eec80a9fb5028e6bc0563107ddec22210b7b555c8c72 WatchSource:0}: Error finding container 5503f03feb4db749e9e6eec80a9fb5028e6bc0563107ddec22210b7b555c8c72: Status 404 returned error can't find the container with id 5503f03feb4db749e9e6eec80a9fb5028e6bc0563107ddec22210b7b555c8c72 Nov 01 20:41:15.649951 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:15.649933 4484 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 20:41:15.675226 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:15.675195 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:15.766134 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:41:15.766100 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 20:41:15.841410 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:15.841371 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:16.037494 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:16.037458 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:16.178843 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:16.178812 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-cd248f-6c9889fc9c-wbd52" event={"ID":"69c9a660-98c7-436f-8bee-b7781c596d0c","Type":"ContainerStarted","Data":"5503f03feb4db749e9e6eec80a9fb5028e6bc0563107ddec22210b7b555c8c72"} Nov 01 20:41:16.959160 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:16.959126 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:17.124131 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:17.124095 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:19.202644 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:19.202606 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-cd248f-6c9889fc9c-wbd52" event={"ID":"69c9a660-98c7-436f-8bee-b7781c596d0c","Type":"ContainerStarted","Data":"79c6672bdfe0e3dca2fa9c24bfa462019ee8f8926e053b6e218e88358fb61e76"} Nov 01 20:41:19.203151 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:19.202751 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-cd248f-6c9889fc9c-wbd52" Nov 01 20:41:19.259729 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:19.259697 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:19.319218 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:19.319159 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-cd248f-6c9889fc9c-wbd52" podStartSLOduration=2.704873265 podStartE2EDuration="5.319146229s" totalImagesPullingTime="2.614272964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:41:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:41:19.317359697 +0000 UTC m=+630.094999860" watchObservedRunningTime="2026-11-01 20:41:19.319146229 +0000 UTC m=+630.096786363" Nov 01 20:41:19.439117 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:19.439086 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:19.651577 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:19.651540 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-944263-78999d46dc-m8ml8"] Nov 01 20:41:19.655222 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:19.655197 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-944263-78999d46dc-m8ml8" Nov 01 20:41:19.718737 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:19.718703 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-944263\"" Nov 01 20:41:19.727461 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:19.722723 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86181c50-8350-4bb1-9d9d-c3e46768606f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-944263-78999d46dc-m8ml8\" (UID: \"86181c50-8350-4bb1-9d9d-c3e46768606f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-944263-78999d46dc-m8ml8" Nov 01 20:41:19.728117 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:19.727973 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q8p2\" (UniqueName: \"kubernetes.io/projected/86181c50-8350-4bb1-9d9d-c3e46768606f-kube-api-access-4q8p2\") pod \"payload-pre-processing-e2e-worker-w3-944263-78999d46dc-m8ml8\" (UID: \"86181c50-8350-4bb1-9d9d-c3e46768606f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-944263-78999d46dc-m8ml8" Nov 01 20:41:19.735013 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:19.734364 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-944263-78999d46dc-m8ml8"] Nov 01 20:41:19.762709 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:19.762668 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-944263-dockercfg-kxqcj\"" Nov 01 20:41:19.802828 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:19.802790 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-944263-5cf85467cc-sl4cb"] Nov 01 20:41:19.806551 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:19.806529 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-944263-5cf85467cc-sl4cb" Nov 01 20:41:19.830301 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:19.830269 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86181c50-8350-4bb1-9d9d-c3e46768606f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-944263-78999d46dc-m8ml8\" (UID: \"86181c50-8350-4bb1-9d9d-c3e46768606f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-944263-78999d46dc-m8ml8" Nov 01 20:41:19.830454 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:19.830307 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crdnn\" (UniqueName: \"kubernetes.io/projected/aca41804-736c-425b-9449-f4a331b56769-kube-api-access-crdnn\") pod \"payload-processing-e2e-worker-w3-944263-5cf85467cc-sl4cb\" (UID: \"aca41804-736c-425b-9449-f4a331b56769\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-944263-5cf85467cc-sl4cb" Nov 01 20:41:19.830454 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:19.830389 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aca41804-736c-425b-9449-f4a331b56769-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-944263-5cf85467cc-sl4cb\" (UID: \"aca41804-736c-425b-9449-f4a331b56769\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-944263-5cf85467cc-sl4cb" Nov 01 20:41:19.830454 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:19.830418 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q8p2\" (UniqueName: \"kubernetes.io/projected/86181c50-8350-4bb1-9d9d-c3e46768606f-kube-api-access-4q8p2\") pod \"payload-pre-processing-e2e-worker-w3-944263-78999d46dc-m8ml8\" (UID: \"86181c50-8350-4bb1-9d9d-c3e46768606f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-944263-78999d46dc-m8ml8" Nov 01 20:41:19.830985 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:19.830967 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86181c50-8350-4bb1-9d9d-c3e46768606f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-944263-78999d46dc-m8ml8\" (UID: \"86181c50-8350-4bb1-9d9d-c3e46768606f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-944263-78999d46dc-m8ml8" Nov 01 20:41:19.872681 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:19.872651 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-944263-5cf85467cc-sl4cb"] Nov 01 20:41:19.897960 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:19.897922 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q8p2\" (UniqueName: \"kubernetes.io/projected/86181c50-8350-4bb1-9d9d-c3e46768606f-kube-api-access-4q8p2\") pod \"payload-pre-processing-e2e-worker-w3-944263-78999d46dc-m8ml8\" (UID: \"86181c50-8350-4bb1-9d9d-c3e46768606f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-944263-78999d46dc-m8ml8" Nov 01 20:41:19.932117 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:19.932028 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crdnn\" (UniqueName: \"kubernetes.io/projected/aca41804-736c-425b-9449-f4a331b56769-kube-api-access-crdnn\") pod \"payload-processing-e2e-worker-w3-944263-5cf85467cc-sl4cb\" (UID: \"aca41804-736c-425b-9449-f4a331b56769\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-944263-5cf85467cc-sl4cb" Nov 01 20:41:19.932291 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:19.932132 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aca41804-736c-425b-9449-f4a331b56769-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-944263-5cf85467cc-sl4cb\" (UID: \"aca41804-736c-425b-9449-f4a331b56769\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-944263-5cf85467cc-sl4cb" Nov 01 20:41:19.932768 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:19.932749 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aca41804-736c-425b-9449-f4a331b56769-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-944263-5cf85467cc-sl4cb\" (UID: \"aca41804-736c-425b-9449-f4a331b56769\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-944263-5cf85467cc-sl4cb" Nov 01 20:41:19.968144 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:19.968112 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-944263-78999d46dc-m8ml8" Nov 01 20:41:19.991213 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:19.991191 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crdnn\" (UniqueName: \"kubernetes.io/projected/aca41804-736c-425b-9449-f4a331b56769-kube-api-access-crdnn\") pod \"payload-processing-e2e-worker-w3-944263-5cf85467cc-sl4cb\" (UID: \"aca41804-736c-425b-9449-f4a331b56769\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-944263-5cf85467cc-sl4cb" Nov 01 20:41:20.119565 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:20.119534 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-944263-5cf85467cc-sl4cb" Nov 01 20:41:20.344446 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:20.344423 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-944263-78999d46dc-m8ml8"] Nov 01 20:41:20.346335 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:41:20.346306 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86181c50_8350_4bb1_9d9d_c3e46768606f.slice/crio-0b31fe530dfeed00ce796185db8a8e82b414081063fda3c96be81c409fc95ebd WatchSource:0}: Error finding container 0b31fe530dfeed00ce796185db8a8e82b414081063fda3c96be81c409fc95ebd: Status 404 returned error can't find the container with id 0b31fe530dfeed00ce796185db8a8e82b414081063fda3c96be81c409fc95ebd Nov 01 20:41:20.496219 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:20.496191 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-944263-5cf85467cc-sl4cb"] Nov 01 20:41:20.497950 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:41:20.497921 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaca41804_736c_425b_9449_f4a331b56769.slice/crio-51623c5e1229419d55d29b80893bec22d377e2bd93c4806d1bb324961039dd54 WatchSource:0}: Error finding container 51623c5e1229419d55d29b80893bec22d377e2bd93c4806d1bb324961039dd54: Status 404 returned error can't find the container with id 51623c5e1229419d55d29b80893bec22d377e2bd93c4806d1bb324961039dd54 Nov 01 20:41:20.873410 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:20.873372 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:21.031756 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:21.031724 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:21.189768 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:21.189697 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:21.213674 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:21.213634 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-944263-5cf85467cc-sl4cb" event={"ID":"aca41804-736c-425b-9449-f4a331b56769","Type":"ContainerStarted","Data":"aea2e69ddf4ba383c068edce30acde5f210f74ba9000de88bcee44033712f556"} Nov 01 20:41:21.213848 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:21.213694 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-944263-5cf85467cc-sl4cb" event={"ID":"aca41804-736c-425b-9449-f4a331b56769","Type":"ContainerStarted","Data":"51623c5e1229419d55d29b80893bec22d377e2bd93c4806d1bb324961039dd54"} Nov 01 20:41:21.213848 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:21.213804 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-944263-5cf85467cc-sl4cb" Nov 01 20:41:21.215183 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:21.215157 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-944263-78999d46dc-m8ml8" event={"ID":"86181c50-8350-4bb1-9d9d-c3e46768606f","Type":"ContainerStarted","Data":"0db602d024ea9cd86437fb8be96b50f6028443b8e1f3564a11e36e28d9c13e1e"} Nov 01 20:41:21.215311 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:21.215201 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-944263-78999d46dc-m8ml8" event={"ID":"86181c50-8350-4bb1-9d9d-c3e46768606f","Type":"ContainerStarted","Data":"0b31fe530dfeed00ce796185db8a8e82b414081063fda3c96be81c409fc95ebd"} Nov 01 20:41:21.215311 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:21.215299 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-944263-78999d46dc-m8ml8" Nov 01 20:41:21.330927 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:21.330876 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-944263-5cf85467cc-sl4cb" podStartSLOduration=2.330863028 podStartE2EDuration="2.330863028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:41:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:41:21.330020389 +0000 UTC m=+632.107660524" watchObservedRunningTime="2026-11-01 20:41:21.330863028 +0000 UTC m=+632.108503163" Nov 01 20:41:21.459227 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:21.459100 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-944263-78999d46dc-m8ml8" podStartSLOduration=2.459077636 podStartE2EDuration="2.459077636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:41:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:41:21.452303297 +0000 UTC m=+632.229943433" watchObservedRunningTime="2026-11-01 20:41:21.459077636 +0000 UTC m=+632.236717801" Nov 01 20:41:21.797482 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:21.797451 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:21.965552 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:21.965517 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:22.188192 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:22.188116 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-dd038b-c5cc45b55-nd59r"] Nov 01 20:41:22.191926 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:22.191900 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-dd038b-c5cc45b55-nd59r" Nov 01 20:41:22.261938 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:22.261904 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-dd038b\"" Nov 01 20:41:22.307640 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:22.280271 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkmgk\" (UniqueName: \"kubernetes.io/projected/bbbfd5b3-4926-4ec2-beff-1b224fa49c5c-kube-api-access-rkmgk\") pod \"payload-pre-processing-e2e-worker-w6-dd038b-c5cc45b55-nd59r\" (UID: \"bbbfd5b3-4926-4ec2-beff-1b224fa49c5c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-dd038b-c5cc45b55-nd59r" Nov 01 20:41:22.307640 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:22.280568 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbbfd5b3-4926-4ec2-beff-1b224fa49c5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-dd038b-c5cc45b55-nd59r\" (UID: \"bbbfd5b3-4926-4ec2-beff-1b224fa49c5c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-dd038b-c5cc45b55-nd59r" Nov 01 20:41:22.312201 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:22.312168 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-dd038b-dockercfg-m2nrt\"" Nov 01 20:41:22.315640 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:22.315610 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-dd038b-c5cc45b55-nd59r"] Nov 01 20:41:22.334873 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:22.334839 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-dd038b-58bf969f7c-5tbbt"] Nov 01 20:41:22.338802 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:22.338783 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-dd038b-58bf969f7c-5tbbt" Nov 01 20:41:22.381525 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:22.381486 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkmgk\" (UniqueName: \"kubernetes.io/projected/bbbfd5b3-4926-4ec2-beff-1b224fa49c5c-kube-api-access-rkmgk\") pod \"payload-pre-processing-e2e-worker-w6-dd038b-c5cc45b55-nd59r\" (UID: \"bbbfd5b3-4926-4ec2-beff-1b224fa49c5c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-dd038b-c5cc45b55-nd59r" Nov 01 20:41:22.381760 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:22.381595 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwnk9\" (UniqueName: \"kubernetes.io/projected/9406d322-46c5-4c90-b984-46496f105414-kube-api-access-wwnk9\") pod \"payload-processing-e2e-worker-w6-dd038b-58bf969f7c-5tbbt\" (UID: \"9406d322-46c5-4c90-b984-46496f105414\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-dd038b-58bf969f7c-5tbbt" Nov 01 20:41:22.381760 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:22.381653 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9406d322-46c5-4c90-b984-46496f105414-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-dd038b-58bf969f7c-5tbbt\" (UID: \"9406d322-46c5-4c90-b984-46496f105414\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-dd038b-58bf969f7c-5tbbt" Nov 01 20:41:22.381867 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:22.381784 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbbfd5b3-4926-4ec2-beff-1b224fa49c5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-dd038b-c5cc45b55-nd59r\" (UID: \"bbbfd5b3-4926-4ec2-beff-1b224fa49c5c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-dd038b-c5cc45b55-nd59r" Nov 01 20:41:22.382495 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:22.382474 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbbfd5b3-4926-4ec2-beff-1b224fa49c5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-dd038b-c5cc45b55-nd59r\" (UID: \"bbbfd5b3-4926-4ec2-beff-1b224fa49c5c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-dd038b-c5cc45b55-nd59r" Nov 01 20:41:22.394612 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:22.394587 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-dd038b-58bf969f7c-5tbbt"] Nov 01 20:41:22.438977 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:22.438905 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkmgk\" (UniqueName: \"kubernetes.io/projected/bbbfd5b3-4926-4ec2-beff-1b224fa49c5c-kube-api-access-rkmgk\") pod \"payload-pre-processing-e2e-worker-w6-dd038b-c5cc45b55-nd59r\" (UID: \"bbbfd5b3-4926-4ec2-beff-1b224fa49c5c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-dd038b-c5cc45b55-nd59r" Nov 01 20:41:22.483323 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:22.483288 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwnk9\" (UniqueName: \"kubernetes.io/projected/9406d322-46c5-4c90-b984-46496f105414-kube-api-access-wwnk9\") pod \"payload-processing-e2e-worker-w6-dd038b-58bf969f7c-5tbbt\" (UID: \"9406d322-46c5-4c90-b984-46496f105414\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-dd038b-58bf969f7c-5tbbt" Nov 01 20:41:22.483796 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:22.483336 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9406d322-46c5-4c90-b984-46496f105414-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-dd038b-58bf969f7c-5tbbt\" (UID: \"9406d322-46c5-4c90-b984-46496f105414\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-dd038b-58bf969f7c-5tbbt" Nov 01 20:41:22.484233 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:22.484206 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9406d322-46c5-4c90-b984-46496f105414-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-dd038b-58bf969f7c-5tbbt\" (UID: \"9406d322-46c5-4c90-b984-46496f105414\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-dd038b-58bf969f7c-5tbbt" Nov 01 20:41:22.507149 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:22.507115 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-dd038b-c5cc45b55-nd59r" Nov 01 20:41:22.541697 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:22.541661 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwnk9\" (UniqueName: \"kubernetes.io/projected/9406d322-46c5-4c90-b984-46496f105414-kube-api-access-wwnk9\") pod \"payload-processing-e2e-worker-w6-dd038b-58bf969f7c-5tbbt\" (UID: \"9406d322-46c5-4c90-b984-46496f105414\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-dd038b-58bf969f7c-5tbbt" Nov 01 20:41:22.652280 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:22.652249 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-dd038b-58bf969f7c-5tbbt" Nov 01 20:41:22.876742 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:22.876713 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-dd038b-c5cc45b55-nd59r"] Nov 01 20:41:22.879884 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:41:22.879857 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbbfd5b3_4926_4ec2_beff_1b224fa49c5c.slice/crio-9f119e8f28a6e1233d7c417a75f45c0543d511f218ad30c22fb15370ed044769 WatchSource:0}: Error finding container 9f119e8f28a6e1233d7c417a75f45c0543d511f218ad30c22fb15370ed044769: Status 404 returned error can't find the container with id 9f119e8f28a6e1233d7c417a75f45c0543d511f218ad30c22fb15370ed044769 Nov 01 20:41:23.024495 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:23.024467 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-dd038b-58bf969f7c-5tbbt"] Nov 01 20:41:23.025525 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:41:23.025499 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9406d322_46c5_4c90_b984_46496f105414.slice/crio-e234dffd0333c2468855c5f9839bcaba180946b1caeaeb3743a7d071e98de909 WatchSource:0}: Error finding container e234dffd0333c2468855c5f9839bcaba180946b1caeaeb3743a7d071e98de909: Status 404 returned error can't find the container with id e234dffd0333c2468855c5f9839bcaba180946b1caeaeb3743a7d071e98de909 Nov 01 20:41:23.226514 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:23.226473 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-dd038b-c5cc45b55-nd59r" event={"ID":"bbbfd5b3-4926-4ec2-beff-1b224fa49c5c","Type":"ContainerStarted","Data":"e943c9c1d9eed282da49ed07f1e09205e45dae2b070cc1a0a5a0d03c44e842b8"} Nov 01 20:41:23.226700 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:23.226532 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-dd038b-c5cc45b55-nd59r" event={"ID":"bbbfd5b3-4926-4ec2-beff-1b224fa49c5c","Type":"ContainerStarted","Data":"9f119e8f28a6e1233d7c417a75f45c0543d511f218ad30c22fb15370ed044769"} Nov 01 20:41:23.226700 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:23.226558 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-dd038b-c5cc45b55-nd59r" Nov 01 20:41:23.228038 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:23.228013 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-dd038b-58bf969f7c-5tbbt" event={"ID":"9406d322-46c5-4c90-b984-46496f105414","Type":"ContainerStarted","Data":"82f165a7468d8bb857f5da25cd812adb42a71208286cb149f2110085eb783ff7"} Nov 01 20:41:23.228154 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:23.228058 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-dd038b-58bf969f7c-5tbbt" event={"ID":"9406d322-46c5-4c90-b984-46496f105414","Type":"ContainerStarted","Data":"e234dffd0333c2468855c5f9839bcaba180946b1caeaeb3743a7d071e98de909"} Nov 01 20:41:23.228154 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:23.228088 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-dd038b-58bf969f7c-5tbbt" Nov 01 20:41:23.341948 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:23.341851 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-dd038b-c5cc45b55-nd59r" podStartSLOduration=1.3418366879999999 podStartE2EDuration="1.341836688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:41:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:41:23.341470826 +0000 UTC m=+634.119110963" watchObservedRunningTime="2026-11-01 20:41:23.341836688 +0000 UTC m=+634.119476819" Nov 01 20:41:23.460433 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:23.458807 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-dd038b-58bf969f7c-5tbbt" podStartSLOduration=1.458787579 podStartE2EDuration="1.458787579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:41:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:41:23.458520754 +0000 UTC m=+634.236160890" watchObservedRunningTime="2026-11-01 20:41:23.458787579 +0000 UTC m=+634.236427716" Nov 01 20:41:24.293607 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:24.283848 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:24.441363 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:24.441320 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:24.655557 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:24.655470 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-db0157-6689cd59bc-nd7kc"] Nov 01 20:41:24.660295 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:24.660271 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-db0157-6689cd59bc-nd7kc" Nov 01 20:41:24.675367 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:24.675332 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e365260-257f-45ef-9306-6e899ae672e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-db0157-6689cd59bc-nd7kc\" (UID: \"8e365260-257f-45ef-9306-6e899ae672e9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-db0157-6689cd59bc-nd7kc" Nov 01 20:41:24.675507 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:24.675446 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9csqq\" (UniqueName: \"kubernetes.io/projected/8e365260-257f-45ef-9306-6e899ae672e9-kube-api-access-9csqq\") pod \"payload-pre-processing-e2e-worker-w1-db0157-6689cd59bc-nd7kc\" (UID: \"8e365260-257f-45ef-9306-6e899ae672e9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-db0157-6689cd59bc-nd7kc" Nov 01 20:41:24.715826 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:24.715781 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-db0157-6689cd59bc-nd7kc"] Nov 01 20:41:24.716407 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:24.716387 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-db0157\"" Nov 01 20:41:24.716407 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:24.716387 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-db0157-dockercfg-jddfr\"" Nov 01 20:41:24.776284 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:24.776243 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e365260-257f-45ef-9306-6e899ae672e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-db0157-6689cd59bc-nd7kc\" (UID: \"8e365260-257f-45ef-9306-6e899ae672e9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-db0157-6689cd59bc-nd7kc" Nov 01 20:41:24.776480 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:24.776333 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9csqq\" (UniqueName: \"kubernetes.io/projected/8e365260-257f-45ef-9306-6e899ae672e9-kube-api-access-9csqq\") pod \"payload-pre-processing-e2e-worker-w1-db0157-6689cd59bc-nd7kc\" (UID: \"8e365260-257f-45ef-9306-6e899ae672e9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-db0157-6689cd59bc-nd7kc" Nov 01 20:41:24.785792 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:24.785757 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e365260-257f-45ef-9306-6e899ae672e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-db0157-6689cd59bc-nd7kc\" (UID: \"8e365260-257f-45ef-9306-6e899ae672e9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-db0157-6689cd59bc-nd7kc" Nov 01 20:41:24.793501 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:24.793107 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-db0157-654558c4b7-vsbx9"] Nov 01 20:41:24.798474 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:24.798442 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-db0157-654558c4b7-vsbx9" Nov 01 20:41:24.833688 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:24.833652 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9csqq\" (UniqueName: \"kubernetes.io/projected/8e365260-257f-45ef-9306-6e899ae672e9-kube-api-access-9csqq\") pod \"payload-pre-processing-e2e-worker-w1-db0157-6689cd59bc-nd7kc\" (UID: \"8e365260-257f-45ef-9306-6e899ae672e9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-db0157-6689cd59bc-nd7kc" Nov 01 20:41:24.853686 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:24.853656 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-db0157-654558c4b7-vsbx9"] Nov 01 20:41:24.975353 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:24.975252 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-db0157-6689cd59bc-nd7kc" Nov 01 20:41:24.978954 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:24.978700 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b416214-5015-4e36-89e4-e9dab20238fd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-db0157-654558c4b7-vsbx9\" (UID: \"0b416214-5015-4e36-89e4-e9dab20238fd\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-db0157-654558c4b7-vsbx9" Nov 01 20:41:24.978954 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:24.978909 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27k6n\" (UniqueName: \"kubernetes.io/projected/0b416214-5015-4e36-89e4-e9dab20238fd-kube-api-access-27k6n\") pod \"payload-processing-e2e-worker-w1-db0157-654558c4b7-vsbx9\" (UID: \"0b416214-5015-4e36-89e4-e9dab20238fd\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-db0157-654558c4b7-vsbx9" Nov 01 20:41:25.079964 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:25.079910 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27k6n\" (UniqueName: \"kubernetes.io/projected/0b416214-5015-4e36-89e4-e9dab20238fd-kube-api-access-27k6n\") pod \"payload-processing-e2e-worker-w1-db0157-654558c4b7-vsbx9\" (UID: \"0b416214-5015-4e36-89e4-e9dab20238fd\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-db0157-654558c4b7-vsbx9" Nov 01 20:41:25.080112 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:25.080077 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b416214-5015-4e36-89e4-e9dab20238fd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-db0157-654558c4b7-vsbx9\" (UID: \"0b416214-5015-4e36-89e4-e9dab20238fd\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-db0157-654558c4b7-vsbx9" Nov 01 20:41:25.080758 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:25.080734 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b416214-5015-4e36-89e4-e9dab20238fd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-db0157-654558c4b7-vsbx9\" (UID: \"0b416214-5015-4e36-89e4-e9dab20238fd\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-db0157-654558c4b7-vsbx9" Nov 01 20:41:25.138488 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:25.138458 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27k6n\" (UniqueName: \"kubernetes.io/projected/0b416214-5015-4e36-89e4-e9dab20238fd-kube-api-access-27k6n\") pod \"payload-processing-e2e-worker-w1-db0157-654558c4b7-vsbx9\" (UID: \"0b416214-5015-4e36-89e4-e9dab20238fd\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-db0157-654558c4b7-vsbx9" Nov 01 20:41:25.371936 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:41:25.371902 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e365260_257f_45ef_9306_6e899ae672e9.slice/crio-22b7b0ba36c3448ccb4beb2d69663149cf03925bc5df0131a3da526db0d09cc7 WatchSource:0}: Error finding container 22b7b0ba36c3448ccb4beb2d69663149cf03925bc5df0131a3da526db0d09cc7: Status 404 returned error can't find the container with id 22b7b0ba36c3448ccb4beb2d69663149cf03925bc5df0131a3da526db0d09cc7 Nov 01 20:41:25.372364 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:25.371973 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-db0157-6689cd59bc-nd7kc"] Nov 01 20:41:25.415635 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:25.415364 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-db0157-654558c4b7-vsbx9" Nov 01 20:41:25.780469 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:25.780440 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:25.801822 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:25.801781 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-db0157-654558c4b7-vsbx9"] Nov 01 20:41:25.802633 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:41:25.802603 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b416214_5015_4e36_89e4_e9dab20238fd.slice/crio-75551f2d1f425a0a1d5fad7cce6aad651922ee69cd1075d21ae25bda3f7e7018 WatchSource:0}: Error finding container 75551f2d1f425a0a1d5fad7cce6aad651922ee69cd1075d21ae25bda3f7e7018: Status 404 returned error can't find the container with id 75551f2d1f425a0a1d5fad7cce6aad651922ee69cd1075d21ae25bda3f7e7018 Nov 01 20:41:25.934495 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:25.934440 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:26.089166 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:26.089083 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:26.340049 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:26.339942 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-db0157-654558c4b7-vsbx9" event={"ID":"0b416214-5015-4e36-89e4-e9dab20238fd","Type":"ContainerStarted","Data":"8e5e0b63de1ddb3e6a9856b1de46170f993606739250ef98c40b1cb9dd7416ed"} Nov 01 20:41:26.340049 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:26.340013 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-db0157-654558c4b7-vsbx9" event={"ID":"0b416214-5015-4e36-89e4-e9dab20238fd","Type":"ContainerStarted","Data":"75551f2d1f425a0a1d5fad7cce6aad651922ee69cd1075d21ae25bda3f7e7018"} Nov 01 20:41:26.340049 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:26.340046 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-db0157-654558c4b7-vsbx9" Nov 01 20:41:26.341542 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:26.341516 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-db0157-6689cd59bc-nd7kc" event={"ID":"8e365260-257f-45ef-9306-6e899ae672e9","Type":"ContainerStarted","Data":"5dd647bb2931451cc37d3a893c1c95c6e3ba16c0629cacda915aee8f8ac785b7"} Nov 01 20:41:26.341635 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:26.341560 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-db0157-6689cd59bc-nd7kc" event={"ID":"8e365260-257f-45ef-9306-6e899ae672e9","Type":"ContainerStarted","Data":"22b7b0ba36c3448ccb4beb2d69663149cf03925bc5df0131a3da526db0d09cc7"} Nov 01 20:41:26.341674 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:26.341634 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-db0157-6689cd59bc-nd7kc" Nov 01 20:41:26.456988 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:26.456932 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-db0157-654558c4b7-vsbx9" podStartSLOduration=2.456917533 podStartE2EDuration="2.456917533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:41:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:41:26.455090759 +0000 UTC m=+637.232730895" watchObservedRunningTime="2026-11-01 20:41:26.456917533 +0000 UTC m=+637.234557667" Nov 01 20:41:29.070511 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:29.070466 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:29.256975 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:29.256939 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:29.467587 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:29.467493 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-db0157-6689cd59bc-nd7kc" podStartSLOduration=5.467480307 podStartE2EDuration="5.467480307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:41:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:41:26.583561292 +0000 UTC m=+637.361201427" watchObservedRunningTime="2026-11-01 20:41:29.467480307 +0000 UTC m=+640.245120442" Nov 01 20:41:29.468137 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:29.468119 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-208f2b-76b7b8c9f-fxd8f"] Nov 01 20:41:29.471886 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:29.471872 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-208f2b-76b7b8c9f-fxd8f" Nov 01 20:41:29.532289 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:29.532257 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-208f2b-76b7b8c9f-fxd8f"] Nov 01 20:41:29.581438 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:29.581250 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-208f2b\"" Nov 01 20:41:29.581438 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:29.581384 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-208f2b-dockercfg-n46vw\"" Nov 01 20:41:29.594284 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:29.591903 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2801b6dc-8797-4099-81fa-5fed21c2de2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-208f2b-76b7b8c9f-fxd8f\" (UID: \"2801b6dc-8797-4099-81fa-5fed21c2de2e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-208f2b-76b7b8c9f-fxd8f" Nov 01 20:41:29.594284 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:29.592055 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt64r\" (UniqueName: \"kubernetes.io/projected/2801b6dc-8797-4099-81fa-5fed21c2de2e-kube-api-access-zt64r\") pod \"payload-pre-processing-e2e-worker-w0-208f2b-76b7b8c9f-fxd8f\" (UID: \"2801b6dc-8797-4099-81fa-5fed21c2de2e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-208f2b-76b7b8c9f-fxd8f" Nov 01 20:41:29.637128 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:29.637097 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-208f2b-7fb7755ddc-lx76l"] Nov 01 20:41:29.641927 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:29.641901 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-208f2b-7fb7755ddc-lx76l" Nov 01 20:41:29.685726 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:29.685696 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-208f2b-7fb7755ddc-lx76l"] Nov 01 20:41:29.693687 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:29.693655 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63adc0cc-05ea-4b6a-9f41-cde67a045b02-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-208f2b-7fb7755ddc-lx76l\" (UID: \"63adc0cc-05ea-4b6a-9f41-cde67a045b02\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-208f2b-7fb7755ddc-lx76l" Nov 01 20:41:29.693858 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:29.693723 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2801b6dc-8797-4099-81fa-5fed21c2de2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-208f2b-76b7b8c9f-fxd8f\" (UID: \"2801b6dc-8797-4099-81fa-5fed21c2de2e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-208f2b-76b7b8c9f-fxd8f" Nov 01 20:41:29.693858 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:29.693798 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf8dq\" (UniqueName: \"kubernetes.io/projected/63adc0cc-05ea-4b6a-9f41-cde67a045b02-kube-api-access-mf8dq\") pod \"payload-processing-e2e-worker-w0-208f2b-7fb7755ddc-lx76l\" (UID: \"63adc0cc-05ea-4b6a-9f41-cde67a045b02\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-208f2b-7fb7755ddc-lx76l" Nov 01 20:41:29.693968 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:29.693865 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt64r\" (UniqueName: \"kubernetes.io/projected/2801b6dc-8797-4099-81fa-5fed21c2de2e-kube-api-access-zt64r\") pod \"payload-pre-processing-e2e-worker-w0-208f2b-76b7b8c9f-fxd8f\" (UID: \"2801b6dc-8797-4099-81fa-5fed21c2de2e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-208f2b-76b7b8c9f-fxd8f" Nov 01 20:41:29.699522 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:29.699487 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2801b6dc-8797-4099-81fa-5fed21c2de2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-208f2b-76b7b8c9f-fxd8f\" (UID: \"2801b6dc-8797-4099-81fa-5fed21c2de2e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-208f2b-76b7b8c9f-fxd8f" Nov 01 20:41:29.760941 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:29.760911 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt64r\" (UniqueName: \"kubernetes.io/projected/2801b6dc-8797-4099-81fa-5fed21c2de2e-kube-api-access-zt64r\") pod \"payload-pre-processing-e2e-worker-w0-208f2b-76b7b8c9f-fxd8f\" (UID: \"2801b6dc-8797-4099-81fa-5fed21c2de2e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-208f2b-76b7b8c9f-fxd8f" Nov 01 20:41:29.784688 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:29.784659 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-208f2b-76b7b8c9f-fxd8f" Nov 01 20:41:29.795362 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:29.795336 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63adc0cc-05ea-4b6a-9f41-cde67a045b02-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-208f2b-7fb7755ddc-lx76l\" (UID: \"63adc0cc-05ea-4b6a-9f41-cde67a045b02\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-208f2b-7fb7755ddc-lx76l" Nov 01 20:41:29.795495 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:29.795435 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf8dq\" (UniqueName: \"kubernetes.io/projected/63adc0cc-05ea-4b6a-9f41-cde67a045b02-kube-api-access-mf8dq\") pod \"payload-processing-e2e-worker-w0-208f2b-7fb7755ddc-lx76l\" (UID: \"63adc0cc-05ea-4b6a-9f41-cde67a045b02\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-208f2b-7fb7755ddc-lx76l" Nov 01 20:41:29.796144 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:29.796122 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63adc0cc-05ea-4b6a-9f41-cde67a045b02-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-208f2b-7fb7755ddc-lx76l\" (UID: \"63adc0cc-05ea-4b6a-9f41-cde67a045b02\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-208f2b-7fb7755ddc-lx76l" Nov 01 20:41:29.853503 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:29.853468 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf8dq\" (UniqueName: \"kubernetes.io/projected/63adc0cc-05ea-4b6a-9f41-cde67a045b02-kube-api-access-mf8dq\") pod \"payload-processing-e2e-worker-w0-208f2b-7fb7755ddc-lx76l\" (UID: \"63adc0cc-05ea-4b6a-9f41-cde67a045b02\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-208f2b-7fb7755ddc-lx76l" Nov 01 20:41:29.957659 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:29.957571 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-208f2b-7fb7755ddc-lx76l" Nov 01 20:41:30.193311 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:30.193282 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-208f2b-76b7b8c9f-fxd8f"] Nov 01 20:41:30.194292 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:41:30.194264 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2801b6dc_8797_4099_81fa_5fed21c2de2e.slice/crio-0792c2a279cf4edcb1cbee17984c1f47fe62f625552f315e41f4924bfd7a3920 WatchSource:0}: Error finding container 0792c2a279cf4edcb1cbee17984c1f47fe62f625552f315e41f4924bfd7a3920: Status 404 returned error can't find the container with id 0792c2a279cf4edcb1cbee17984c1f47fe62f625552f315e41f4924bfd7a3920 Nov 01 20:41:30.231151 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:30.231117 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-cd248f-6c9889fc9c-wbd52" Nov 01 20:41:30.362117 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:30.362087 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-208f2b-7fb7755ddc-lx76l"] Nov 01 20:41:30.365242 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:41:30.365216 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63adc0cc_05ea_4b6a_9f41_cde67a045b02.slice/crio-f31a3842b344296b2a48c23b617d5b94aa4171179e3fb73db05d01a59e4015a0 WatchSource:0}: Error finding container f31a3842b344296b2a48c23b617d5b94aa4171179e3fb73db05d01a59e4015a0: Status 404 returned error can't find the container with id f31a3842b344296b2a48c23b617d5b94aa4171179e3fb73db05d01a59e4015a0 Nov 01 20:41:30.384783 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:30.384751 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-208f2b-7fb7755ddc-lx76l" event={"ID":"63adc0cc-05ea-4b6a-9f41-cde67a045b02","Type":"ContainerStarted","Data":"f31a3842b344296b2a48c23b617d5b94aa4171179e3fb73db05d01a59e4015a0"} Nov 01 20:41:30.386559 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:30.386511 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-208f2b-76b7b8c9f-fxd8f" event={"ID":"2801b6dc-8797-4099-81fa-5fed21c2de2e","Type":"ContainerStarted","Data":"61f951c2913180db98752f89fbbb4d4af848cce81f6e65e421fe55d201799d2b"} Nov 01 20:41:30.386684 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:30.386581 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-208f2b-76b7b8c9f-fxd8f" event={"ID":"2801b6dc-8797-4099-81fa-5fed21c2de2e","Type":"ContainerStarted","Data":"0792c2a279cf4edcb1cbee17984c1f47fe62f625552f315e41f4924bfd7a3920"} Nov 01 20:41:30.386894 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:30.386814 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-208f2b-76b7b8c9f-fxd8f" Nov 01 20:41:30.503799 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:30.503747 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-208f2b-76b7b8c9f-fxd8f" podStartSLOduration=1.503732979 podStartE2EDuration="1.503732979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:41:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:41:30.502258988 +0000 UTC m=+641.279899123" watchObservedRunningTime="2026-11-01 20:41:30.503732979 +0000 UTC m=+641.281373166" Nov 01 20:41:30.729116 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:30.729018 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:31.394533 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:31.394494 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-208f2b-7fb7755ddc-lx76l" event={"ID":"63adc0cc-05ea-4b6a-9f41-cde67a045b02","Type":"ContainerStarted","Data":"854406f946db48c116ed03210fd1fedcda582d58c9168c829649feb819a89755"} Nov 01 20:41:31.395032 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:31.394586 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-208f2b-7fb7755ddc-lx76l" Nov 01 20:41:31.510898 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:31.510836 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-208f2b-7fb7755ddc-lx76l" podStartSLOduration=2.510818411 podStartE2EDuration="2.510818411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:41:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:41:31.510409737 +0000 UTC m=+642.288049874" watchObservedRunningTime="2026-11-01 20:41:31.510818411 +0000 UTC m=+642.288458547" Nov 01 20:41:32.222841 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:32.222813 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-944263-5cf85467cc-sl4cb" Nov 01 20:41:32.223032 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:32.222878 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-944263-78999d46dc-m8ml8" Nov 01 20:41:33.933597 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:33.933551 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:34.134202 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:34.129773 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:34.328780 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:34.328748 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-dd038b-58bf969f7c-5tbbt" Nov 01 20:41:34.329839 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:34.328834 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-dd038b-c5cc45b55-nd59r" Nov 01 20:41:34.347500 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:34.347443 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-1a1e9e-8fc6dcb7-bs6sm"] Nov 01 20:41:34.352113 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:34.352087 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1a1e9e-8fc6dcb7-bs6sm" Nov 01 20:41:34.450032 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:34.435915 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-1a1e9e-dockercfg-9jwxc\"" Nov 01 20:41:34.450032 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:34.436014 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-1a1e9e\"" Nov 01 20:41:34.450032 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:34.441442 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-1a1e9e-8fc6dcb7-bs6sm"] Nov 01 20:41:34.466022 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:34.463663 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsq2l\" (UniqueName: \"kubernetes.io/projected/903377a5-cfac-4d8a-adf0-aa729a3cfdde-kube-api-access-lsq2l\") pod \"payload-pre-processing-e2e-worker-w4-1a1e9e-8fc6dcb7-bs6sm\" (UID: \"903377a5-cfac-4d8a-adf0-aa729a3cfdde\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1a1e9e-8fc6dcb7-bs6sm" Nov 01 20:41:34.466022 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:34.463908 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/903377a5-cfac-4d8a-adf0-aa729a3cfdde-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-1a1e9e-8fc6dcb7-bs6sm\" (UID: \"903377a5-cfac-4d8a-adf0-aa729a3cfdde\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1a1e9e-8fc6dcb7-bs6sm" Nov 01 20:41:34.496087 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:34.494425 4484 generic.go:356] "Generic (PLEG): container finished" podID="3d20fc4c-5175-4154-8669-2494376150ce" containerID="e6568ce5159b31428ae7351e04d78dadeaf28e26e6db11e199f4feb8af86d852" exitCode=0 Nov 01 20:41:34.496087 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:34.494483 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9" event={"ID":"3d20fc4c-5175-4154-8669-2494376150ce","Type":"ContainerDied","Data":"e6568ce5159b31428ae7351e04d78dadeaf28e26e6db11e199f4feb8af86d852"} Nov 01 20:41:34.496087 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:34.495103 4484 scope.go:122] "RemoveContainer" containerID="e6568ce5159b31428ae7351e04d78dadeaf28e26e6db11e199f4feb8af86d852" Nov 01 20:41:34.496940 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:34.496911 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:41:34.497062 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:34.497050 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:41:34.497198 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:34.497181 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:41:34.501216 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:34.499964 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-1a1e9e-66d4f8c5f-cr9fp"] Nov 01 20:41:34.535553 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:34.534364 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-1a1e9e-66d4f8c5f-cr9fp" Nov 01 20:41:34.565686 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:34.565651 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/903377a5-cfac-4d8a-adf0-aa729a3cfdde-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-1a1e9e-8fc6dcb7-bs6sm\" (UID: \"903377a5-cfac-4d8a-adf0-aa729a3cfdde\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1a1e9e-8fc6dcb7-bs6sm" Nov 01 20:41:34.565850 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:34.565788 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsq2l\" (UniqueName: \"kubernetes.io/projected/903377a5-cfac-4d8a-adf0-aa729a3cfdde-kube-api-access-lsq2l\") pod \"payload-pre-processing-e2e-worker-w4-1a1e9e-8fc6dcb7-bs6sm\" (UID: \"903377a5-cfac-4d8a-adf0-aa729a3cfdde\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1a1e9e-8fc6dcb7-bs6sm" Nov 01 20:41:34.566321 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:34.566297 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-1a1e9e-66d4f8c5f-cr9fp"] Nov 01 20:41:34.569045 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:34.567319 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/903377a5-cfac-4d8a-adf0-aa729a3cfdde-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-1a1e9e-8fc6dcb7-bs6sm\" (UID: \"903377a5-cfac-4d8a-adf0-aa729a3cfdde\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1a1e9e-8fc6dcb7-bs6sm" Nov 01 20:41:34.635075 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:34.633963 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsq2l\" (UniqueName: \"kubernetes.io/projected/903377a5-cfac-4d8a-adf0-aa729a3cfdde-kube-api-access-lsq2l\") pod \"payload-pre-processing-e2e-worker-w4-1a1e9e-8fc6dcb7-bs6sm\" (UID: \"903377a5-cfac-4d8a-adf0-aa729a3cfdde\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1a1e9e-8fc6dcb7-bs6sm" Nov 01 20:41:34.666643 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:34.666609 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a9e967b-5887-4361-95ad-9a4122698833-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-1a1e9e-66d4f8c5f-cr9fp\" (UID: \"8a9e967b-5887-4361-95ad-9a4122698833\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-1a1e9e-66d4f8c5f-cr9fp" Nov 01 20:41:34.666812 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:34.666691 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn4s2\" (UniqueName: \"kubernetes.io/projected/8a9e967b-5887-4361-95ad-9a4122698833-kube-api-access-rn4s2\") pod \"payload-processing-e2e-worker-w4-1a1e9e-66d4f8c5f-cr9fp\" (UID: \"8a9e967b-5887-4361-95ad-9a4122698833\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-1a1e9e-66d4f8c5f-cr9fp" Nov 01 20:41:34.666906 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:34.666891 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1a1e9e-8fc6dcb7-bs6sm" Nov 01 20:41:34.767671 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:34.767611 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a9e967b-5887-4361-95ad-9a4122698833-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-1a1e9e-66d4f8c5f-cr9fp\" (UID: \"8a9e967b-5887-4361-95ad-9a4122698833\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-1a1e9e-66d4f8c5f-cr9fp" Nov 01 20:41:34.768568 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:34.768543 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a9e967b-5887-4361-95ad-9a4122698833-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-1a1e9e-66d4f8c5f-cr9fp\" (UID: \"8a9e967b-5887-4361-95ad-9a4122698833\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-1a1e9e-66d4f8c5f-cr9fp" Nov 01 20:41:34.768965 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:34.768945 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn4s2\" (UniqueName: \"kubernetes.io/projected/8a9e967b-5887-4361-95ad-9a4122698833-kube-api-access-rn4s2\") pod \"payload-processing-e2e-worker-w4-1a1e9e-66d4f8c5f-cr9fp\" (UID: \"8a9e967b-5887-4361-95ad-9a4122698833\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-1a1e9e-66d4f8c5f-cr9fp" Nov 01 20:41:34.844353 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:34.844291 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn4s2\" (UniqueName: \"kubernetes.io/projected/8a9e967b-5887-4361-95ad-9a4122698833-kube-api-access-rn4s2\") pod \"payload-processing-e2e-worker-w4-1a1e9e-66d4f8c5f-cr9fp\" (UID: \"8a9e967b-5887-4361-95ad-9a4122698833\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-1a1e9e-66d4f8c5f-cr9fp" Nov 01 20:41:34.884668 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:34.884476 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-1a1e9e-66d4f8c5f-cr9fp" Nov 01 20:41:35.135264 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:35.134310 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-1a1e9e-8fc6dcb7-bs6sm"] Nov 01 20:41:35.292545 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:35.292512 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-1a1e9e-66d4f8c5f-cr9fp"] Nov 01 20:41:35.293727 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:41:35.293697 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a9e967b_5887_4361_95ad_9a4122698833.slice/crio-ce683eca5367ccd60c9a8e9325bdbb3118685b5031ab0b2c4107d455fd3eb7c1 WatchSource:0}: Error finding container ce683eca5367ccd60c9a8e9325bdbb3118685b5031ab0b2c4107d455fd3eb7c1: Status 404 returned error can't find the container with id ce683eca5367ccd60c9a8e9325bdbb3118685b5031ab0b2c4107d455fd3eb7c1 Nov 01 20:41:35.500498 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:35.500461 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:35.531420 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:35.531103 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9" event={"ID":"3d20fc4c-5175-4154-8669-2494376150ce","Type":"ContainerStarted","Data":"c3cc07d5d8d25d52eabbaed8d6102a7c6ad4962435a83d0a0a2bed1d88d14207"} Nov 01 20:41:35.534966 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:35.534360 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1a1e9e-8fc6dcb7-bs6sm" event={"ID":"903377a5-cfac-4d8a-adf0-aa729a3cfdde","Type":"ContainerStarted","Data":"c18048df1d8d86a50759e6376198bc7db34272c3c122c1e3070a9671b7b67572"} Nov 01 20:41:35.534966 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:35.534435 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1a1e9e-8fc6dcb7-bs6sm" event={"ID":"903377a5-cfac-4d8a-adf0-aa729a3cfdde","Type":"ContainerStarted","Data":"477a68da7bfef68ff662b749645af0bc0bb32c37b47175944ab698cba8952e8a"} Nov 01 20:41:35.535807 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:35.535296 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1a1e9e-8fc6dcb7-bs6sm" Nov 01 20:41:35.537533 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:35.537438 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-1a1e9e-66d4f8c5f-cr9fp" event={"ID":"8a9e967b-5887-4361-95ad-9a4122698833","Type":"ContainerStarted","Data":"964c4d7650f59a65aa7a22dc715d5f3763ebf12890a3c110838a4f3f43ca558c"} Nov 01 20:41:35.537533 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:35.537483 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-1a1e9e-66d4f8c5f-cr9fp" event={"ID":"8a9e967b-5887-4361-95ad-9a4122698833","Type":"ContainerStarted","Data":"ce683eca5367ccd60c9a8e9325bdbb3118685b5031ab0b2c4107d455fd3eb7c1"} Nov 01 20:41:35.541014 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:35.538162 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-1a1e9e-66d4f8c5f-cr9fp" Nov 01 20:41:35.676681 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:35.676570 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:35.701111 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:35.701072 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9" Nov 01 20:41:35.701449 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:35.701416 4484 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Nov 01 20:41:35.701548 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:35.701480 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9" podUID="3d20fc4c-5175-4154-8669-2494376150ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Nov 01 20:41:35.774689 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:35.774621 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-1a1e9e-66d4f8c5f-cr9fp" podStartSLOduration=1.7746028059999999 podStartE2EDuration="1.774602806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:41:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:41:35.774589493 +0000 UTC m=+646.552229655" watchObservedRunningTime="2026-11-01 20:41:35.774602806 +0000 UTC m=+646.552242941" Nov 01 20:41:35.902366 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:35.902303 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1a1e9e-8fc6dcb7-bs6sm" podStartSLOduration=1.9022895100000001 podStartE2EDuration="1.90228951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:41:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:41:35.900504142 +0000 UTC m=+646.678144313" watchObservedRunningTime="2026-11-01 20:41:35.90228951 +0000 UTC m=+646.679929645" Nov 01 20:41:35.979291 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:35.979217 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:36.485649 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:36.485612 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:36.700733 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:36.700697 4484 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Nov 01 20:41:36.700913 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:36.700748 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9" podUID="3d20fc4c-5175-4154-8669-2494376150ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Nov 01 20:41:37.146827 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:37.146798 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9" Nov 01 20:41:37.361405 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:37.361215 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-db0157-654558c4b7-vsbx9" Nov 01 20:41:37.361405 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:37.361306 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-db0157-6689cd59bc-nd7kc" Nov 01 20:41:37.700789 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:37.700747 4484 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Nov 01 20:41:37.701197 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:37.700810 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9" podUID="3d20fc4c-5175-4154-8669-2494376150ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Nov 01 20:41:38.700747 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:38.700711 4484 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Nov 01 20:41:38.700929 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:38.700767 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9" podUID="3d20fc4c-5175-4154-8669-2494376150ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Nov 01 20:41:38.947531 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:38.947494 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:39.119247 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:39.119210 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:39.303028 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:39.302973 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-c9ae21-7bc765fb9f-7tjsn"] Nov 01 20:41:39.331780 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:39.331741 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-c9ae21-7bc765fb9f-7tjsn" Nov 01 20:41:39.363193 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:39.363161 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-c9ae21-7bc765fb9f-7tjsn"] Nov 01 20:41:39.428100 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:39.403943 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lctpx\" (UniqueName: \"kubernetes.io/projected/7d955afb-8551-4657-9daa-c70b309757a6-kube-api-access-lctpx\") pod \"payload-pre-processing-e2e-worker-w5-c9ae21-7bc765fb9f-7tjsn\" (UID: \"7d955afb-8551-4657-9daa-c70b309757a6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-c9ae21-7bc765fb9f-7tjsn" Nov 01 20:41:39.428284 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:39.428204 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d955afb-8551-4657-9daa-c70b309757a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-c9ae21-7bc765fb9f-7tjsn\" (UID: \"7d955afb-8551-4657-9daa-c70b309757a6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-c9ae21-7bc765fb9f-7tjsn" Nov 01 20:41:39.428341 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:39.416594 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-c9ae21\"" Nov 01 20:41:39.428392 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:39.416755 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-c9ae21-dockercfg-vjk9c\"" Nov 01 20:41:39.516064 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:39.503899 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-c9ae21-6667d54cf5-twgq6"] Nov 01 20:41:39.564670 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:39.530618 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lctpx\" (UniqueName: \"kubernetes.io/projected/7d955afb-8551-4657-9daa-c70b309757a6-kube-api-access-lctpx\") pod \"payload-pre-processing-e2e-worker-w5-c9ae21-7bc765fb9f-7tjsn\" (UID: \"7d955afb-8551-4657-9daa-c70b309757a6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-c9ae21-7bc765fb9f-7tjsn" Nov 01 20:41:39.565058 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:39.565033 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d955afb-8551-4657-9daa-c70b309757a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-c9ae21-7bc765fb9f-7tjsn\" (UID: \"7d955afb-8551-4657-9daa-c70b309757a6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-c9ae21-7bc765fb9f-7tjsn" Nov 01 20:41:39.566368 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:39.566294 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d955afb-8551-4657-9daa-c70b309757a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-c9ae21-7bc765fb9f-7tjsn\" (UID: \"7d955afb-8551-4657-9daa-c70b309757a6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-c9ae21-7bc765fb9f-7tjsn" Nov 01 20:41:39.576573 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:39.576535 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-c9ae21-6667d54cf5-twgq6"] Nov 01 20:41:39.576757 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:39.576668 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-c9ae21-6667d54cf5-twgq6" Nov 01 20:41:39.588723 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:39.588698 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lctpx\" (UniqueName: \"kubernetes.io/projected/7d955afb-8551-4657-9daa-c70b309757a6-kube-api-access-lctpx\") pod \"payload-pre-processing-e2e-worker-w5-c9ae21-7bc765fb9f-7tjsn\" (UID: \"7d955afb-8551-4657-9daa-c70b309757a6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-c9ae21-7bc765fb9f-7tjsn" Nov 01 20:41:39.646177 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:39.646147 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-c9ae21-7bc765fb9f-7tjsn" Nov 01 20:41:39.667508 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:39.667194 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/824b88e5-a583-4489-9562-846fb877553d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-c9ae21-6667d54cf5-twgq6\" (UID: \"824b88e5-a583-4489-9562-846fb877553d\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-c9ae21-6667d54cf5-twgq6" Nov 01 20:41:39.667508 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:39.667324 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckd7p\" (UniqueName: \"kubernetes.io/projected/824b88e5-a583-4489-9562-846fb877553d-kube-api-access-ckd7p\") pod \"payload-processing-e2e-worker-w5-c9ae21-6667d54cf5-twgq6\" (UID: \"824b88e5-a583-4489-9562-846fb877553d\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-c9ae21-6667d54cf5-twgq6" Nov 01 20:41:39.700539 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:39.700508 4484 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Nov 01 20:41:39.700664 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:39.700578 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9" podUID="3d20fc4c-5175-4154-8669-2494376150ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Nov 01 20:41:39.768475 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:39.768426 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/824b88e5-a583-4489-9562-846fb877553d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-c9ae21-6667d54cf5-twgq6\" (UID: \"824b88e5-a583-4489-9562-846fb877553d\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-c9ae21-6667d54cf5-twgq6" Nov 01 20:41:39.769315 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:39.768510 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckd7p\" (UniqueName: \"kubernetes.io/projected/824b88e5-a583-4489-9562-846fb877553d-kube-api-access-ckd7p\") pod \"payload-processing-e2e-worker-w5-c9ae21-6667d54cf5-twgq6\" (UID: \"824b88e5-a583-4489-9562-846fb877553d\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-c9ae21-6667d54cf5-twgq6" Nov 01 20:41:39.769903 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:39.769879 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/824b88e5-a583-4489-9562-846fb877553d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-c9ae21-6667d54cf5-twgq6\" (UID: \"824b88e5-a583-4489-9562-846fb877553d\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-c9ae21-6667d54cf5-twgq6" Nov 01 20:41:39.832706 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:39.832651 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckd7p\" (UniqueName: \"kubernetes.io/projected/824b88e5-a583-4489-9562-846fb877553d-kube-api-access-ckd7p\") pod \"payload-processing-e2e-worker-w5-c9ae21-6667d54cf5-twgq6\" (UID: \"824b88e5-a583-4489-9562-846fb877553d\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-c9ae21-6667d54cf5-twgq6" Nov 01 20:41:39.931013 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:39.930955 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-c9ae21-6667d54cf5-twgq6" Nov 01 20:41:40.041055 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:40.041025 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-c9ae21-7bc765fb9f-7tjsn"] Nov 01 20:41:40.331114 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:40.318382 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-c9ae21-6667d54cf5-twgq6"] Nov 01 20:41:40.379240 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:40.379204 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:40.539895 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:40.539856 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:40.593448 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:40.593352 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-c9ae21-7bc765fb9f-7tjsn" event={"ID":"7d955afb-8551-4657-9daa-c70b309757a6","Type":"ContainerStarted","Data":"8598b20a8496f3d79d63ab7ca83c0b69a78470e99fee48cc983fbd4f761cce7c"} Nov 01 20:41:40.593625 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:40.593498 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-c9ae21-7bc765fb9f-7tjsn" event={"ID":"7d955afb-8551-4657-9daa-c70b309757a6","Type":"ContainerStarted","Data":"a7459c0b77a6d25996f67ed2e2b0f4e2e145f95c3a49a5c2ffd25b151a167773"} Nov 01 20:41:40.593625 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:40.593537 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-c9ae21-7bc765fb9f-7tjsn" Nov 01 20:41:40.595374 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:40.595345 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-c9ae21-6667d54cf5-twgq6" event={"ID":"824b88e5-a583-4489-9562-846fb877553d","Type":"ContainerStarted","Data":"69879df6e5a6ec6e02d154f04409e49cdd92d7f0183c21a22ba09a3df32601a6"} Nov 01 20:41:40.595527 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:40.595413 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-c9ae21-6667d54cf5-twgq6" Nov 01 20:41:40.595527 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:40.595438 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-c9ae21-6667d54cf5-twgq6" event={"ID":"824b88e5-a583-4489-9562-846fb877553d","Type":"ContainerStarted","Data":"2b0daa3cab291b4e6ee4d095cbfc6ba1be24ed529fceca6d01dcbc7698194289"} Nov 01 20:41:40.690862 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:40.690827 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:40.699901 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:40.699872 4484 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Nov 01 20:41:40.700058 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:40.699921 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9" podUID="3d20fc4c-5175-4154-8669-2494376150ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Nov 01 20:41:40.846437 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:40.846295 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-c9ae21-7bc765fb9f-7tjsn" podStartSLOduration=1.846271803 podStartE2EDuration="1.846271803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:41:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:41:40.727309426 +0000 UTC m=+651.504949562" watchObservedRunningTime="2026-11-01 20:41:40.846271803 +0000 UTC m=+651.623911940" Nov 01 20:41:41.397846 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:41.397818 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-208f2b-76b7b8c9f-fxd8f" Nov 01 20:41:41.513538 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:41.513479 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-c9ae21-6667d54cf5-twgq6" podStartSLOduration=2.513465027 podStartE2EDuration="2.513465027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:41:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:41:40.855901813 +0000 UTC m=+651.633541948" watchObservedRunningTime="2026-11-01 20:41:41.513465027 +0000 UTC m=+652.291105162" Nov 01 20:41:41.704186 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:41.704107 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9" Nov 01 20:41:41.705256 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:41.705233 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9" Nov 01 20:41:42.404085 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:42.404049 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-208f2b-7fb7755ddc-lx76l" Nov 01 20:41:43.612949 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:43.612910 4484 generic.go:356] "Generic (PLEG): container finished" podID="35f67043-6238-4386-a37a-3c2dbed1fae9" containerID="61b1906e39baee0d36b4f4ec168741c9a02e049517234025034b92d856b82356" exitCode=0 Nov 01 20:41:43.613445 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:43.612961 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr" event={"ID":"35f67043-6238-4386-a37a-3c2dbed1fae9","Type":"ContainerDied","Data":"61b1906e39baee0d36b4f4ec168741c9a02e049517234025034b92d856b82356"} Nov 01 20:41:43.613445 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:43.613356 4484 scope.go:122] "RemoveContainer" containerID="61b1906e39baee0d36b4f4ec168741c9a02e049517234025034b92d856b82356" Nov 01 20:41:43.614660 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:43.614624 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:41:43.614767 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:43.614708 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:41:43.614767 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:43.614753 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:41:44.645020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:44.644380 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr" event={"ID":"35f67043-6238-4386-a37a-3c2dbed1fae9","Type":"ContainerStarted","Data":"f39fcfb768baa4a8f9d9656dcdcc776029817091ff2e33c1c2c9d2dfa9d06eec"} Nov 01 20:41:44.666484 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:44.666455 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr" Nov 01 20:41:44.666713 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:44.666618 4484 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 01 20:41:44.666813 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:44.666798 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr" podUID="35f67043-6238-4386-a37a-3c2dbed1fae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 01 20:41:45.479387 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:45.479351 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:45.666277 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:45.666241 4484 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 01 20:41:45.666674 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:45.666291 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr" podUID="35f67043-6238-4386-a37a-3c2dbed1fae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 01 20:41:45.774661 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:41:45.774620 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 20:41:46.665334 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:46.665298 4484 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 01 20:41:46.665543 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:46.665351 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr" podUID="35f67043-6238-4386-a37a-3c2dbed1fae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 01 20:41:47.567577 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:47.567549 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-1a1e9e-66d4f8c5f-cr9fp" Nov 01 20:41:47.568445 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:47.568421 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1a1e9e-8fc6dcb7-bs6sm" Nov 01 20:41:47.666113 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:47.666064 4484 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 01 20:41:47.666413 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:47.666372 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr" podUID="35f67043-6238-4386-a37a-3c2dbed1fae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 01 20:41:48.666006 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:48.665956 4484 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 01 20:41:48.666522 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:48.666042 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr" podUID="35f67043-6238-4386-a37a-3c2dbed1fae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 01 20:41:49.666744 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:49.666708 4484 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 01 20:41:49.667217 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:49.667076 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr" podUID="35f67043-6238-4386-a37a-3c2dbed1fae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 01 20:41:50.359033 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:50.358952 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:50.670631 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:50.670535 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr" Nov 01 20:41:50.671274 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:50.671254 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr" Nov 01 20:41:50.671975 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:50.671949 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr" Nov 01 20:41:51.608262 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:51.607250 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-c9ae21-7bc765fb9f-7tjsn" Nov 01 20:41:51.613168 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:51.612867 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-c9ae21-6667d54cf5-twgq6" Nov 01 20:41:53.480142 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:53.480009 4484 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": read tcp 10.134.0.2:48894->10.134.0.39:15021: read: connection reset by peer" start-of-body= Nov 01 20:41:53.480142 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:53.480086 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw" podUID="e9c302ea-168a-4fa6-885f-2a3263835c79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": read tcp 10.134.0.2:48894->10.134.0.39:15021: read: connection reset by peer" Nov 01 20:41:53.717074 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:53.708772 4484 generic.go:356] "Generic (PLEG): container finished" podID="e9c302ea-168a-4fa6-885f-2a3263835c79" containerID="297e2449b94efbf6299679b7996c8245b7492d28246c60a767b6a54e6716a790" exitCode=0 Nov 01 20:41:53.717074 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:53.708940 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw" event={"ID":"e9c302ea-168a-4fa6-885f-2a3263835c79","Type":"ContainerDied","Data":"297e2449b94efbf6299679b7996c8245b7492d28246c60a767b6a54e6716a790"} Nov 01 20:41:53.717074 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:53.713488 4484 scope.go:122] "RemoveContainer" containerID="297e2449b94efbf6299679b7996c8245b7492d28246c60a767b6a54e6716a790" Nov 01 20:41:53.729324 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:53.729236 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:41:53.729455 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:53.729343 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:41:53.729455 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:53.729423 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:41:54.655060 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:54.654906 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:54.735829 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:54.735793 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw" event={"ID":"e9c302ea-168a-4fa6-885f-2a3263835c79","Type":"ContainerStarted","Data":"54b57e02f36025126bec7241c8f6c9529b485e6d3f662fdff7d589afcf3ab366"} Nov 01 20:41:54.974207 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:54.974124 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw" Nov 01 20:41:54.974360 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:54.974214 4484 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 01 20:41:54.974360 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:54.974246 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw" podUID="e9c302ea-168a-4fa6-885f-2a3263835c79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 01 20:41:55.252082 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:55.250825 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:55.426014 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:55.425963 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:55.592774 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:55.592683 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:55.756812 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:55.756787 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:55.974396 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:55.974308 4484 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 01 20:41:55.974396 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:55.974372 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw" podUID="e9c302ea-168a-4fa6-885f-2a3263835c79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 01 20:41:56.974751 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:56.974713 4484 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 01 20:41:56.975179 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:56.974767 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw" podUID="e9c302ea-168a-4fa6-885f-2a3263835c79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 01 20:41:57.974505 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:57.974468 4484 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 01 20:41:57.974680 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:57.974526 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw" podUID="e9c302ea-168a-4fa6-885f-2a3263835c79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 01 20:41:58.974910 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:58.974872 4484 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 01 20:41:58.975310 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:41:58.974937 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw" podUID="e9c302ea-168a-4fa6-885f-2a3263835c79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 01 20:42:00.024575 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:42:00.024524 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw" Nov 01 20:42:00.025429 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:42:00.025404 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw" Nov 01 20:42:00.027108 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:42:00.027086 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw" Nov 01 20:42:00.090680 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:42:00.090646 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:42:00.718588 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:42:00.718529 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:42:00.902302 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:42:00.902262 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:42:01.101280 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:42:01.101246 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:42:01.791645 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:42:01.791615 4484 generic.go:356] "Generic (PLEG): container finished" podID="3ae0b2c8-270b-4a3f-bb16-523874681ade" containerID="5f3068fefdcfccb633c8492083fc1e3420cd25dc941ed3797c287eeaadc6303d" exitCode=0 Nov 01 20:42:01.791883 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:42:01.791689 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv" event={"ID":"3ae0b2c8-270b-4a3f-bb16-523874681ade","Type":"ContainerDied","Data":"5f3068fefdcfccb633c8492083fc1e3420cd25dc941ed3797c287eeaadc6303d"} Nov 01 20:42:01.792256 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:42:01.792237 4484 scope.go:122] "RemoveContainer" containerID="5f3068fefdcfccb633c8492083fc1e3420cd25dc941ed3797c287eeaadc6303d" Nov 01 20:42:01.793553 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:42:01.793523 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:42:01.793625 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:42:01.793606 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:42:01.793669 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:42:01.793652 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:42:02.099106 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:42:02.099056 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:42:02.800792 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:42:02.800682 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv" event={"ID":"3ae0b2c8-270b-4a3f-bb16-523874681ade","Type":"ContainerStarted","Data":"acbed28624ebe42544acb96d055050fed39395495b39d480ce98a0590c69f879"} Nov 01 20:42:03.051226 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:42:03.051129 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv" Nov 01 20:42:03.052614 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:42:03.052584 4484 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 01 20:42:03.052739 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:42:03.052636 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv" podUID="3ae0b2c8-270b-4a3f-bb16-523874681ade" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 01 20:42:04.050817 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:42:04.050778 4484 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 01 20:42:04.051274 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:42:04.050835 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv" podUID="3ae0b2c8-270b-4a3f-bb16-523874681ade" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 01 20:42:05.050393 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:42:05.050353 4484 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 01 20:42:05.050580 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:42:05.050412 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv" podUID="3ae0b2c8-270b-4a3f-bb16-523874681ade" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 01 20:42:05.836828 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:42:05.836734 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:42:06.050617 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:42:06.050567 4484 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 01 20:42:06.050800 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:42:06.050643 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv" podUID="3ae0b2c8-270b-4a3f-bb16-523874681ade" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 01 20:42:07.051240 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:42:07.051200 4484 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 01 20:42:07.051651 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:42:07.051256 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv" podUID="3ae0b2c8-270b-4a3f-bb16-523874681ade" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 01 20:42:07.135453 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:42:07.135425 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv" Nov 01 20:42:08.105917 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:42:08.105881 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv" Nov 01 20:42:08.107467 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:42:08.107336 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv" Nov 01 20:42:10.767106 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:42:10.767073 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:42:15.765736 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:42:15.765698 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 20:42:15.981034 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:42:15.980985 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:42:16.145055 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:42:16.144945 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:42:16.551689 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:42:16.551654 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:42:16.720020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:42:16.717119 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:42:21.634044 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:42:21.633973 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:42:23.829598 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:42:23.829562 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:42:26.337286 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:42:26.337250 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:42:26.803783 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:42:26.803750 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:42:26.972008 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:42:26.971958 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:42:27.141323 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:42:27.141248 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:42:28.955595 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:42:28.955487 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:42:30.471432 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:42:30.471374 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:42:31.057704 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:42:31.057669 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:42:31.237817 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:42:31.237747 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:42:32.178841 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:42:32.178797 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:42:36.809100 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:42:36.809064 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:42:36.970019 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:42:36.969718 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:42:41.733272 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:42:41.733233 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:42:45.765970 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:42:45.765934 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 20:42:46.919546 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:42:46.919517 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:42:47.393402 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:42:47.393309 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:42:47.552513 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:42:47.552472 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:42:47.944186 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:42:47.944149 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:42:48.115650 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:42:48.115620 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:42:48.288723 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:42:48.288682 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:42:50.573622 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:42:50.573590 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:42:52.128821 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:42:52.128781 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:42:55.563777 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:42:55.563742 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:42:56.934673 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:42:56.934645 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:42:58.770650 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:42:58.770617 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:43:01.655802 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:43:01.655768 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:43:01.809671 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:43:01.809625 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:43:06.526661 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:43:06.526626 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:43:10.415619 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:43:10.415567 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:43:11.333578 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:43:11.333548 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:43:15.766536 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:43:15.766499 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 20:43:16.138508 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:43:16.138409 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:43:16.629148 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:43:16.629099 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg"] Nov 01 20:43:16.633050 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:43:16.633023 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" Nov 01 20:43:16.737911 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:43:16.737877 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-c9ae21\"/\"kube-root-ca.crt\"" Nov 01 20:43:16.737911 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:43:16.737893 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-c9ae21\"/\"simulator-w5-c9ae21-kserve-self-signed-certs\"" Nov 01 20:43:16.738173 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:43:16.737943 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-c9ae21\"/\"openshift-service-ca.crt\"" Nov 01 20:43:16.738173 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:43:16.738014 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-c9ae21\"/\"default-dockercfg-rfgc8\"" Nov 01 20:43:16.740312 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:43:16.740290 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg"] Nov 01 20:43:16.745200 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:43:16.745178 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:43:16.812126 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:43:16.805617 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17a9ac24-19cb-48c4-99b5-d74ce8aac0a1-tmp-dir\") pod \"simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg\" (UID: \"17a9ac24-19cb-48c4-99b5-d74ce8aac0a1\") " pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" Nov 01 20:43:16.812126 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:43:16.805664 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17a9ac24-19cb-48c4-99b5-d74ce8aac0a1-home\") pod \"simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg\" (UID: \"17a9ac24-19cb-48c4-99b5-d74ce8aac0a1\") " pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" Nov 01 20:43:16.812126 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:43:16.805721 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17a9ac24-19cb-48c4-99b5-d74ce8aac0a1-tls-certs\") pod \"simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg\" (UID: \"17a9ac24-19cb-48c4-99b5-d74ce8aac0a1\") " pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" Nov 01 20:43:16.812126 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:43:16.805823 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17a9ac24-19cb-48c4-99b5-d74ce8aac0a1-dshm\") pod \"simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg\" (UID: \"17a9ac24-19cb-48c4-99b5-d74ce8aac0a1\") " pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" Nov 01 20:43:16.812126 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:43:16.805879 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvttj\" (UniqueName: \"kubernetes.io/projected/17a9ac24-19cb-48c4-99b5-d74ce8aac0a1-kube-api-access-jvttj\") pod \"simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg\" (UID: \"17a9ac24-19cb-48c4-99b5-d74ce8aac0a1\") " pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" Nov 01 20:43:16.812126 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:43:16.805908 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17a9ac24-19cb-48c4-99b5-d74ce8aac0a1-model-cache\") pod \"simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg\" (UID: \"17a9ac24-19cb-48c4-99b5-d74ce8aac0a1\") " pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" Nov 01 20:43:16.906991 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:43:16.906911 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17a9ac24-19cb-48c4-99b5-d74ce8aac0a1-dshm\") pod \"simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg\" (UID: \"17a9ac24-19cb-48c4-99b5-d74ce8aac0a1\") " pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" Nov 01 20:43:16.906991 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:43:16.906974 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvttj\" (UniqueName: \"kubernetes.io/projected/17a9ac24-19cb-48c4-99b5-d74ce8aac0a1-kube-api-access-jvttj\") pod \"simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg\" (UID: \"17a9ac24-19cb-48c4-99b5-d74ce8aac0a1\") " pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" Nov 01 20:43:16.906991 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:43:16.907007 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17a9ac24-19cb-48c4-99b5-d74ce8aac0a1-model-cache\") pod \"simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg\" (UID: \"17a9ac24-19cb-48c4-99b5-d74ce8aac0a1\") " pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" Nov 01 20:43:16.907298 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:43:16.907081 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17a9ac24-19cb-48c4-99b5-d74ce8aac0a1-tmp-dir\") pod \"simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg\" (UID: \"17a9ac24-19cb-48c4-99b5-d74ce8aac0a1\") " pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" Nov 01 20:43:16.907298 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:43:16.907110 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17a9ac24-19cb-48c4-99b5-d74ce8aac0a1-home\") pod \"simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg\" (UID: \"17a9ac24-19cb-48c4-99b5-d74ce8aac0a1\") " pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" Nov 01 20:43:16.907298 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:43:16.907160 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17a9ac24-19cb-48c4-99b5-d74ce8aac0a1-tls-certs\") pod \"simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg\" (UID: \"17a9ac24-19cb-48c4-99b5-d74ce8aac0a1\") " pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" Nov 01 20:43:16.907492 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:43:16.907468 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17a9ac24-19cb-48c4-99b5-d74ce8aac0a1-model-cache\") pod \"simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg\" (UID: \"17a9ac24-19cb-48c4-99b5-d74ce8aac0a1\") " pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" Nov 01 20:43:16.907560 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:43:16.907521 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17a9ac24-19cb-48c4-99b5-d74ce8aac0a1-tmp-dir\") pod \"simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg\" (UID: \"17a9ac24-19cb-48c4-99b5-d74ce8aac0a1\") " pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" Nov 01 20:43:16.907656 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:43:16.907632 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17a9ac24-19cb-48c4-99b5-d74ce8aac0a1-home\") pod \"simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg\" (UID: \"17a9ac24-19cb-48c4-99b5-d74ce8aac0a1\") " pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" Nov 01 20:43:16.909717 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:43:16.909690 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17a9ac24-19cb-48c4-99b5-d74ce8aac0a1-dshm\") pod \"simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg\" (UID: \"17a9ac24-19cb-48c4-99b5-d74ce8aac0a1\") " pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" Nov 01 20:43:16.910317 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:43:16.910296 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17a9ac24-19cb-48c4-99b5-d74ce8aac0a1-tls-certs\") pod \"simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg\" (UID: \"17a9ac24-19cb-48c4-99b5-d74ce8aac0a1\") " pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" Nov 01 20:43:16.965831 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:43:16.965796 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvttj\" (UniqueName: \"kubernetes.io/projected/17a9ac24-19cb-48c4-99b5-d74ce8aac0a1-kube-api-access-jvttj\") pod \"simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg\" (UID: \"17a9ac24-19cb-48c4-99b5-d74ce8aac0a1\") " pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" Nov 01 20:43:17.245598 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:43:17.245568 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" Nov 01 20:43:17.619069 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:43:17.619040 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg"] Nov 01 20:43:17.620190 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:43:17.620164 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod17a9ac24_19cb_48c4_99b5_d74ce8aac0a1.slice/crio-5152d213b52aef5d73cc2fe3ed76f96a80bcc2558a4a4a9a534b8b349f44c4cf WatchSource:0}: Error finding container 5152d213b52aef5d73cc2fe3ed76f96a80bcc2558a4a4a9a534b8b349f44c4cf: Status 404 returned error can't find the container with id 5152d213b52aef5d73cc2fe3ed76f96a80bcc2558a4a4a9a534b8b349f44c4cf Nov 01 20:43:18.279067 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:43:18.279028 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" event={"ID":"17a9ac24-19cb-48c4-99b5-d74ce8aac0a1","Type":"ContainerStarted","Data":"640d3d99214489846e68eedfa38ef3f6ef234f4d1c14624c85c2cf6feaee34a7"} Nov 01 20:43:18.279444 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:43:18.279078 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" event={"ID":"17a9ac24-19cb-48c4-99b5-d74ce8aac0a1","Type":"ContainerStarted","Data":"5152d213b52aef5d73cc2fe3ed76f96a80bcc2558a4a4a9a534b8b349f44c4cf"} Nov 01 20:43:18.396505 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:43:18.396452 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" podStartSLOduration=2.396423293 podStartE2EDuration="2.396423293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:43:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:43:18.394264862 +0000 UTC m=+749.171904999" watchObservedRunningTime="2026-11-01 20:43:18.396423293 +0000 UTC m=+749.174063428" Nov 01 20:43:19.246080 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:43:19.246046 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" Nov 01 20:43:20.935613 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:43:20.935579 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:43:25.735817 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:43:25.735762 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:43:27.010503 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:43:27.010465 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:43:27.246612 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:43:27.246580 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" Nov 01 20:43:27.260019 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:43:27.259968 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" Nov 01 20:43:27.339444 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:43:27.339368 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" Nov 01 20:43:30.566883 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:43:30.566848 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:43:35.353740 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:43:35.353657 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:43:35.790769 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:43:35.790732 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:43:37.158363 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:43:37.158321 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:43:40.140545 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:43:40.140509 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:43:44.998640 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:43:44.998590 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:43:45.186423 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:43:45.186383 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:43:45.767139 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:43:45.767103 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 20:43:49.731561 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:43:49.731526 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:43:54.536442 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:43:54.536406 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:43:59.402947 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:43:59.402913 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:44:02.326957 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:02.326921 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:44:04.097900 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:04.097866 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:44:08.837194 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:08.837150 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:44:13.643818 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:13.643774 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:44:15.768883 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:44:15.768842 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 20:44:18.467022 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:18.466941 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:44:19.501160 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:19.501116 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:44:23.251463 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:23.251427 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:44:26.090670 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:26.090638 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4"] Nov 01 20:44:26.094653 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:26.094627 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" Nov 01 20:44:26.143243 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:26.143204 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/311f9699-6585-4a2c-8c20-e6b696b2f857-tmp-dir\") pod \"premium-w2-cd248f-kserve-66ddfc4c45-l74x4\" (UID: \"311f9699-6585-4a2c-8c20-e6b696b2f857\") " pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" Nov 01 20:44:26.143439 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:26.143271 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/311f9699-6585-4a2c-8c20-e6b696b2f857-model-cache\") pod \"premium-w2-cd248f-kserve-66ddfc4c45-l74x4\" (UID: \"311f9699-6585-4a2c-8c20-e6b696b2f857\") " pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" Nov 01 20:44:26.143439 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:26.143409 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/311f9699-6585-4a2c-8c20-e6b696b2f857-home\") pod \"premium-w2-cd248f-kserve-66ddfc4c45-l74x4\" (UID: \"311f9699-6585-4a2c-8c20-e6b696b2f857\") " pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" Nov 01 20:44:26.143574 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:26.143447 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/311f9699-6585-4a2c-8c20-e6b696b2f857-dshm\") pod \"premium-w2-cd248f-kserve-66ddfc4c45-l74x4\" (UID: \"311f9699-6585-4a2c-8c20-e6b696b2f857\") " pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" Nov 01 20:44:26.143574 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:26.143530 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qx9m\" (UniqueName: \"kubernetes.io/projected/311f9699-6585-4a2c-8c20-e6b696b2f857-kube-api-access-2qx9m\") pod \"premium-w2-cd248f-kserve-66ddfc4c45-l74x4\" (UID: \"311f9699-6585-4a2c-8c20-e6b696b2f857\") " pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" Nov 01 20:44:26.143574 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:26.143555 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/311f9699-6585-4a2c-8c20-e6b696b2f857-tls-certs\") pod \"premium-w2-cd248f-kserve-66ddfc4c45-l74x4\" (UID: \"311f9699-6585-4a2c-8c20-e6b696b2f857\") " pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" Nov 01 20:44:26.150350 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:26.150319 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4"] Nov 01 20:44:26.198639 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:26.198592 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-cd248f\"/\"openshift-service-ca.crt\"" Nov 01 20:44:26.201555 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:26.201528 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-cd248f\"/\"premium-w2-cd248f-kserve-self-signed-certs\"" Nov 01 20:44:26.201682 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:26.201557 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-cd248f\"/\"kube-root-ca.crt\"" Nov 01 20:44:26.201682 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:26.201621 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-cd248f\"/\"default-dockercfg-ck7jd\"" Nov 01 20:44:26.205365 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:26.205340 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:44:26.245163 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:26.245120 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qx9m\" (UniqueName: \"kubernetes.io/projected/311f9699-6585-4a2c-8c20-e6b696b2f857-kube-api-access-2qx9m\") pod \"premium-w2-cd248f-kserve-66ddfc4c45-l74x4\" (UID: \"311f9699-6585-4a2c-8c20-e6b696b2f857\") " pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" Nov 01 20:44:26.245370 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:26.245176 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/311f9699-6585-4a2c-8c20-e6b696b2f857-tls-certs\") pod \"premium-w2-cd248f-kserve-66ddfc4c45-l74x4\" (UID: \"311f9699-6585-4a2c-8c20-e6b696b2f857\") " pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" Nov 01 20:44:26.245370 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:26.245281 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/311f9699-6585-4a2c-8c20-e6b696b2f857-tmp-dir\") pod \"premium-w2-cd248f-kserve-66ddfc4c45-l74x4\" (UID: \"311f9699-6585-4a2c-8c20-e6b696b2f857\") " pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" Nov 01 20:44:26.245370 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:26.245359 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/311f9699-6585-4a2c-8c20-e6b696b2f857-model-cache\") pod \"premium-w2-cd248f-kserve-66ddfc4c45-l74x4\" (UID: \"311f9699-6585-4a2c-8c20-e6b696b2f857\") " pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" Nov 01 20:44:26.245534 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:26.245422 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/311f9699-6585-4a2c-8c20-e6b696b2f857-home\") pod \"premium-w2-cd248f-kserve-66ddfc4c45-l74x4\" (UID: \"311f9699-6585-4a2c-8c20-e6b696b2f857\") " pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" Nov 01 20:44:26.245534 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:26.245455 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/311f9699-6585-4a2c-8c20-e6b696b2f857-dshm\") pod \"premium-w2-cd248f-kserve-66ddfc4c45-l74x4\" (UID: \"311f9699-6585-4a2c-8c20-e6b696b2f857\") " pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" Nov 01 20:44:26.278041 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:26.246428 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/311f9699-6585-4a2c-8c20-e6b696b2f857-tmp-dir\") pod \"premium-w2-cd248f-kserve-66ddfc4c45-l74x4\" (UID: \"311f9699-6585-4a2c-8c20-e6b696b2f857\") " pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" Nov 01 20:44:26.278041 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:26.247483 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/311f9699-6585-4a2c-8c20-e6b696b2f857-model-cache\") pod \"premium-w2-cd248f-kserve-66ddfc4c45-l74x4\" (UID: \"311f9699-6585-4a2c-8c20-e6b696b2f857\") " pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" Nov 01 20:44:26.278041 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:26.247735 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/311f9699-6585-4a2c-8c20-e6b696b2f857-home\") pod \"premium-w2-cd248f-kserve-66ddfc4c45-l74x4\" (UID: \"311f9699-6585-4a2c-8c20-e6b696b2f857\") " pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" Nov 01 20:44:26.284297 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:26.284263 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/311f9699-6585-4a2c-8c20-e6b696b2f857-dshm\") pod \"premium-w2-cd248f-kserve-66ddfc4c45-l74x4\" (UID: \"311f9699-6585-4a2c-8c20-e6b696b2f857\") " pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" Nov 01 20:44:26.285618 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:26.285523 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/311f9699-6585-4a2c-8c20-e6b696b2f857-tls-certs\") pod \"premium-w2-cd248f-kserve-66ddfc4c45-l74x4\" (UID: \"311f9699-6585-4a2c-8c20-e6b696b2f857\") " pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" Nov 01 20:44:26.305658 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:26.305625 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qx9m\" (UniqueName: \"kubernetes.io/projected/311f9699-6585-4a2c-8c20-e6b696b2f857-kube-api-access-2qx9m\") pod \"premium-w2-cd248f-kserve-66ddfc4c45-l74x4\" (UID: \"311f9699-6585-4a2c-8c20-e6b696b2f857\") " pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" Nov 01 20:44:26.407269 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:26.407173 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" Nov 01 20:44:26.848742 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:26.848709 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4"] Nov 01 20:44:26.851202 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:44:26.851175 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod311f9699_6585_4a2c_8c20_e6b696b2f857.slice/crio-3811d2e3366588338bf66dc607ab50825f95a30b5da36c64290f6c96b9e54cfa WatchSource:0}: Error finding container 3811d2e3366588338bf66dc607ab50825f95a30b5da36c64290f6c96b9e54cfa: Status 404 returned error can't find the container with id 3811d2e3366588338bf66dc607ab50825f95a30b5da36c64290f6c96b9e54cfa Nov 01 20:44:27.732807 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:27.732770 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" event={"ID":"311f9699-6585-4a2c-8c20-e6b696b2f857","Type":"ContainerStarted","Data":"d79b4d0528c4d3b99396b7d9bbd4d9684197f2f16be9d8007a6081ae2f669555"} Nov 01 20:44:27.733221 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:27.732821 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" event={"ID":"311f9699-6585-4a2c-8c20-e6b696b2f857","Type":"ContainerStarted","Data":"3811d2e3366588338bf66dc607ab50825f95a30b5da36c64290f6c96b9e54cfa"} Nov 01 20:44:27.849616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:27.849566 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" podStartSLOduration=1.849550218 podStartE2EDuration="1.849550218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:44:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:44:27.849295948 +0000 UTC m=+818.626936083" watchObservedRunningTime="2026-11-01 20:44:27.849550218 +0000 UTC m=+818.627190373" Nov 01 20:44:28.051578 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:28.051544 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:44:28.407498 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:28.407412 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" Nov 01 20:44:32.844728 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:32.844686 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:44:36.407818 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:36.407616 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" Nov 01 20:44:36.421263 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:36.421232 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" Nov 01 20:44:36.791925 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:36.791895 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" Nov 01 20:44:37.503991 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:37.503956 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:44:37.662413 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:37.662381 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:44:42.373163 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:42.373123 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:44:43.219615 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:43.219575 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws"] Nov 01 20:44:43.223147 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:43.223130 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" Nov 01 20:44:43.304116 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:43.304079 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws"] Nov 01 20:44:43.331293 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:43.331255 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-1a1e9e\"/\"openshift-service-ca.crt\"" Nov 01 20:44:43.331442 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:43.331320 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-1a1e9e\"/\"premium-w4-1a1e9e-kserve-self-signed-certs\"" Nov 01 20:44:43.331527 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:43.331447 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-1a1e9e\"/\"kube-root-ca.crt\"" Nov 01 20:44:43.331588 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:43.331544 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-1a1e9e\"/\"default-dockercfg-rp528\"" Nov 01 20:44:43.354378 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:43.354355 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe8307db-264d-4b8a-b097-f180bc255a9e-home\") pod \"premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws\" (UID: \"fe8307db-264d-4b8a-b097-f180bc255a9e\") " pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" Nov 01 20:44:43.354520 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:43.354384 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe8307db-264d-4b8a-b097-f180bc255a9e-tls-certs\") pod \"premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws\" (UID: \"fe8307db-264d-4b8a-b097-f180bc255a9e\") " pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" Nov 01 20:44:43.354520 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:43.354475 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe8307db-264d-4b8a-b097-f180bc255a9e-tmp-dir\") pod \"premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws\" (UID: \"fe8307db-264d-4b8a-b097-f180bc255a9e\") " pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" Nov 01 20:44:43.354520 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:43.354508 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe8307db-264d-4b8a-b097-f180bc255a9e-model-cache\") pod \"premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws\" (UID: \"fe8307db-264d-4b8a-b097-f180bc255a9e\") " pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" Nov 01 20:44:43.354684 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:43.354646 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p542x\" (UniqueName: \"kubernetes.io/projected/fe8307db-264d-4b8a-b097-f180bc255a9e-kube-api-access-p542x\") pod \"premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws\" (UID: \"fe8307db-264d-4b8a-b097-f180bc255a9e\") " pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" Nov 01 20:44:43.354810 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:43.354787 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe8307db-264d-4b8a-b097-f180bc255a9e-dshm\") pod \"premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws\" (UID: \"fe8307db-264d-4b8a-b097-f180bc255a9e\") " pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" Nov 01 20:44:43.394202 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:43.394109 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:44:43.455943 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:43.455852 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe8307db-264d-4b8a-b097-f180bc255a9e-home\") pod \"premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws\" (UID: \"fe8307db-264d-4b8a-b097-f180bc255a9e\") " pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" Nov 01 20:44:43.456159 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:43.455970 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe8307db-264d-4b8a-b097-f180bc255a9e-tls-certs\") pod \"premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws\" (UID: \"fe8307db-264d-4b8a-b097-f180bc255a9e\") " pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" Nov 01 20:44:43.456159 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:43.456064 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe8307db-264d-4b8a-b097-f180bc255a9e-tmp-dir\") pod \"premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws\" (UID: \"fe8307db-264d-4b8a-b097-f180bc255a9e\") " pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" Nov 01 20:44:43.456159 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:43.456099 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe8307db-264d-4b8a-b097-f180bc255a9e-model-cache\") pod \"premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws\" (UID: \"fe8307db-264d-4b8a-b097-f180bc255a9e\") " pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" Nov 01 20:44:43.456389 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:43.456177 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p542x\" (UniqueName: \"kubernetes.io/projected/fe8307db-264d-4b8a-b097-f180bc255a9e-kube-api-access-p542x\") pod \"premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws\" (UID: \"fe8307db-264d-4b8a-b097-f180bc255a9e\") " pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" Nov 01 20:44:43.456389 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:43.456330 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe8307db-264d-4b8a-b097-f180bc255a9e-dshm\") pod \"premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws\" (UID: \"fe8307db-264d-4b8a-b097-f180bc255a9e\") " pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" Nov 01 20:44:43.456389 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:43.456356 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe8307db-264d-4b8a-b097-f180bc255a9e-home\") pod \"premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws\" (UID: \"fe8307db-264d-4b8a-b097-f180bc255a9e\") " pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" Nov 01 20:44:43.456547 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:43.456404 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe8307db-264d-4b8a-b097-f180bc255a9e-tmp-dir\") pod \"premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws\" (UID: \"fe8307db-264d-4b8a-b097-f180bc255a9e\") " pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" Nov 01 20:44:43.456585 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:43.456541 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe8307db-264d-4b8a-b097-f180bc255a9e-model-cache\") pod \"premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws\" (UID: \"fe8307db-264d-4b8a-b097-f180bc255a9e\") " pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" Nov 01 20:44:43.458484 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:43.458459 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe8307db-264d-4b8a-b097-f180bc255a9e-dshm\") pod \"premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws\" (UID: \"fe8307db-264d-4b8a-b097-f180bc255a9e\") " pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" Nov 01 20:44:43.458484 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:43.458474 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe8307db-264d-4b8a-b097-f180bc255a9e-tls-certs\") pod \"premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws\" (UID: \"fe8307db-264d-4b8a-b097-f180bc255a9e\") " pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" Nov 01 20:44:43.513743 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:43.513713 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p542x\" (UniqueName: \"kubernetes.io/projected/fe8307db-264d-4b8a-b097-f180bc255a9e-kube-api-access-p542x\") pod \"premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws\" (UID: \"fe8307db-264d-4b8a-b097-f180bc255a9e\") " pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" Nov 01 20:44:43.535640 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:43.535610 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" Nov 01 20:44:43.923661 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:43.923631 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws"] Nov 01 20:44:43.924961 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:44:43.924937 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe8307db_264d_4b8a_b097_f180bc255a9e.slice/crio-ce4d6d7c36f55f57858e9ce63198be9f53a8c46b22dad9aebd173cca6c3300bd WatchSource:0}: Error finding container ce4d6d7c36f55f57858e9ce63198be9f53a8c46b22dad9aebd173cca6c3300bd: Status 404 returned error can't find the container with id ce4d6d7c36f55f57858e9ce63198be9f53a8c46b22dad9aebd173cca6c3300bd Nov 01 20:44:44.522179 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:44.522137 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b8846bc-cwg7c"] Nov 01 20:44:44.526143 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:44.526122 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b8846bc-cwg7c" Nov 01 20:44:44.582199 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:44.582158 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b8846bc-cwg7c"] Nov 01 20:44:44.629558 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:44.629522 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:44:44.629802 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:44.629525 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:44:44.629926 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:44.629523 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:44:44.629926 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:44.629604 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 20:44:44.672337 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:44.672289 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d3a7b07-765e-4eec-8948-ff6366bb6c9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b8846bc-cwg7c\" (UID: \"9d3a7b07-765e-4eec-8948-ff6366bb6c9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b8846bc-cwg7c" Nov 01 20:44:44.672529 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:44.672351 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d3a7b07-765e-4eec-8948-ff6366bb6c9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b8846bc-cwg7c\" (UID: \"9d3a7b07-765e-4eec-8948-ff6366bb6c9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b8846bc-cwg7c" Nov 01 20:44:44.672594 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:44.672542 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzwmn\" (UniqueName: \"kubernetes.io/projected/9d3a7b07-765e-4eec-8948-ff6366bb6c9a-kube-api-access-rzwmn\") pod \"maas-api-revoke-sub-5b8846bc-cwg7c\" (UID: \"9d3a7b07-765e-4eec-8948-ff6366bb6c9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b8846bc-cwg7c" Nov 01 20:44:44.781108 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:44.777685 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d3a7b07-765e-4eec-8948-ff6366bb6c9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b8846bc-cwg7c\" (UID: \"9d3a7b07-765e-4eec-8948-ff6366bb6c9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b8846bc-cwg7c" Nov 01 20:44:44.781108 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:44.777746 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d3a7b07-765e-4eec-8948-ff6366bb6c9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b8846bc-cwg7c\" (UID: \"9d3a7b07-765e-4eec-8948-ff6366bb6c9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b8846bc-cwg7c" Nov 01 20:44:44.781108 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:44.777822 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzwmn\" (UniqueName: \"kubernetes.io/projected/9d3a7b07-765e-4eec-8948-ff6366bb6c9a-kube-api-access-rzwmn\") pod \"maas-api-revoke-sub-5b8846bc-cwg7c\" (UID: \"9d3a7b07-765e-4eec-8948-ff6366bb6c9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b8846bc-cwg7c" Nov 01 20:44:44.781108 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:44.779676 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d3a7b07-765e-4eec-8948-ff6366bb6c9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b8846bc-cwg7c\" (UID: \"9d3a7b07-765e-4eec-8948-ff6366bb6c9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b8846bc-cwg7c" Nov 01 20:44:44.819602 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:44.815480 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d3a7b07-765e-4eec-8948-ff6366bb6c9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b8846bc-cwg7c\" (UID: \"9d3a7b07-765e-4eec-8948-ff6366bb6c9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b8846bc-cwg7c" Nov 01 20:44:44.841712 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:44.841675 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" event={"ID":"fe8307db-264d-4b8a-b097-f180bc255a9e","Type":"ContainerStarted","Data":"b5fb24eb8274831d9a71f6e345a01ef166c31017df29796bd931accc2d74eb85"} Nov 01 20:44:44.841943 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:44.841926 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" event={"ID":"fe8307db-264d-4b8a-b097-f180bc255a9e","Type":"ContainerStarted","Data":"ce4d6d7c36f55f57858e9ce63198be9f53a8c46b22dad9aebd173cca6c3300bd"} Nov 01 20:44:44.857712 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:44.856656 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzwmn\" (UniqueName: \"kubernetes.io/projected/9d3a7b07-765e-4eec-8948-ff6366bb6c9a-kube-api-access-rzwmn\") pod \"maas-api-revoke-sub-5b8846bc-cwg7c\" (UID: \"9d3a7b07-765e-4eec-8948-ff6366bb6c9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b8846bc-cwg7c" Nov 01 20:44:44.963516 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:44.963449 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" podStartSLOduration=1.9634304679999999 podStartE2EDuration="1.963430468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:44:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:44:44.958896402 +0000 UTC m=+835.736536540" watchObservedRunningTime="2026-11-01 20:44:44.963430468 +0000 UTC m=+835.741070604" Nov 01 20:44:45.143820 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:45.143716 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b8846bc-cwg7c" Nov 01 20:44:45.537213 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:45.536849 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" Nov 01 20:44:45.537635 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:45.537530 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b8846bc-cwg7c"] Nov 01 20:44:45.542440 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:44:45.542403 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d3a7b07_765e_4eec_8948_ff6366bb6c9a.slice/crio-ed45fb6a08f9f1c7ae592714d012e9ccd73f253ba74f462b1a14561274c203cc WatchSource:0}: Error finding container ed45fb6a08f9f1c7ae592714d012e9ccd73f253ba74f462b1a14561274c203cc: Status 404 returned error can't find the container with id ed45fb6a08f9f1c7ae592714d012e9ccd73f253ba74f462b1a14561274c203cc Nov 01 20:44:45.768223 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:44:45.768173 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 20:44:45.848941 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:45.848859 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b8846bc-cwg7c" event={"ID":"9d3a7b07-765e-4eec-8948-ff6366bb6c9a","Type":"ContainerStarted","Data":"ed45fb6a08f9f1c7ae592714d012e9ccd73f253ba74f462b1a14561274c203cc"} Nov 01 20:44:47.233405 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:47.233377 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:44:51.804562 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:51.804526 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:44:52.133265 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:52.133170 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:44:53.536149 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:53.536118 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" Nov 01 20:44:53.549048 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:53.549017 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" Nov 01 20:44:53.898564 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:53.898520 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b8846bc-cwg7c" event={"ID":"9d3a7b07-765e-4eec-8948-ff6366bb6c9a","Type":"ContainerStarted","Data":"06ffa97ef20777ec7985cf9af9c8d2186c891dc6bf6e55ef12120f009d8b4f27"} Nov 01 20:44:53.910564 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:53.910538 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" Nov 01 20:44:54.066037 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:54.065972 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b8846bc-cwg7c" podStartSLOduration=2.5370252989999997 podStartE2EDuration="10.065958342s" totalImagesPullingTime="7.528933043s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:44:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:44:54.013154133 +0000 UTC m=+844.790794269" watchObservedRunningTime="2026-11-01 20:44:54.065958342 +0000 UTC m=+844.843598477" Nov 01 20:44:54.906295 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:54.906261 4484 generic.go:356] "Generic (PLEG): container finished" podID="9d3a7b07-765e-4eec-8948-ff6366bb6c9a" containerID="06ffa97ef20777ec7985cf9af9c8d2186c891dc6bf6e55ef12120f009d8b4f27" exitCode=0 Nov 01 20:44:54.906809 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:54.906343 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b8846bc-cwg7c" event={"ID":"9d3a7b07-765e-4eec-8948-ff6366bb6c9a","Type":"ContainerDied","Data":"06ffa97ef20777ec7985cf9af9c8d2186c891dc6bf6e55ef12120f009d8b4f27"} Nov 01 20:44:56.063334 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:56.063302 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b8846bc-cwg7c" Nov 01 20:44:56.126791 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:56.126764 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d3a7b07-765e-4eec-8948-ff6366bb6c9a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d3a7b07-765e-4eec-8948-ff6366bb6c9a-maas-api-xks-ca\") pod \"9d3a7b07-765e-4eec-8948-ff6366bb6c9a\" (UID: \"9d3a7b07-765e-4eec-8948-ff6366bb6c9a\") " Nov 01 20:44:56.126951 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:56.126824 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d3a7b07-765e-4eec-8948-ff6366bb6c9a-kube-api-access-rzwmn\" (UniqueName: \"kubernetes.io/projected/9d3a7b07-765e-4eec-8948-ff6366bb6c9a-kube-api-access-rzwmn\") pod \"9d3a7b07-765e-4eec-8948-ff6366bb6c9a\" (UID: \"9d3a7b07-765e-4eec-8948-ff6366bb6c9a\") " Nov 01 20:44:56.126951 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:56.126855 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d3a7b07-765e-4eec-8948-ff6366bb6c9a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d3a7b07-765e-4eec-8948-ff6366bb6c9a-maas-api-openshift-ca\") pod \"9d3a7b07-765e-4eec-8948-ff6366bb6c9a\" (UID: \"9d3a7b07-765e-4eec-8948-ff6366bb6c9a\") " Nov 01 20:44:56.127190 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:56.127168 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d3a7b07-765e-4eec-8948-ff6366bb6c9a-maas-api-openshift-ca" pod "9d3a7b07-765e-4eec-8948-ff6366bb6c9a" (UID: "9d3a7b07-765e-4eec-8948-ff6366bb6c9a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:44:56.127294 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:56.127278 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d3a7b07-765e-4eec-8948-ff6366bb6c9a-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:44:56.128904 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:56.128878 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d3a7b07-765e-4eec-8948-ff6366bb6c9a-maas-api-xks-ca" pod "9d3a7b07-765e-4eec-8948-ff6366bb6c9a" (UID: "9d3a7b07-765e-4eec-8948-ff6366bb6c9a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:44:56.129029 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:56.128943 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d3a7b07-765e-4eec-8948-ff6366bb6c9a-kube-api-access-rzwmn" pod "9d3a7b07-765e-4eec-8948-ff6366bb6c9a" (UID: "9d3a7b07-765e-4eec-8948-ff6366bb6c9a"). InnerVolumeSpecName "kube-api-access-rzwmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:44:56.229011 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:56.228916 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzwmn\" (UniqueName: \"kubernetes.io/projected/9d3a7b07-765e-4eec-8948-ff6366bb6c9a-kube-api-access-rzwmn\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:44:56.229011 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:56.228947 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d3a7b07-765e-4eec-8948-ff6366bb6c9a-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:44:56.738331 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:56.738296 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06fecfbb-7wr8m"] Nov 01 20:44:56.738864 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:56.738847 4484 cpu_manager.go:414] "containerMap: removing container" podUID="9d3a7b07-765e-4eec-8948-ff6366bb6c9a" containerName="revoke-keys" Nov 01 20:44:56.738929 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:56.738865 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d3a7b07-765e-4eec-8948-ff6366bb6c9a" containerName="revoke-keys" Nov 01 20:44:56.739001 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:56.738983 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d3a7b07-765e-4eec-8948-ff6366bb6c9a" containerName="revoke-keys" Nov 01 20:44:56.742690 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:56.742667 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fecfbb-7wr8m" Nov 01 20:44:56.808764 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:56.808723 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06fecfbb-7wr8m"] Nov 01 20:44:56.835064 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:56.835020 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04242230-db36-48bc-b020-06b79d6ddab4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06fecfbb-7wr8m\" (UID: \"04242230-db36-48bc-b020-06b79d6ddab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fecfbb-7wr8m" Nov 01 20:44:56.835064 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:56.835063 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st2cl\" (UniqueName: \"kubernetes.io/projected/04242230-db36-48bc-b020-06b79d6ddab4-kube-api-access-st2cl\") pod \"maas-api-revoke-sub-06fecfbb-7wr8m\" (UID: \"04242230-db36-48bc-b020-06b79d6ddab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fecfbb-7wr8m" Nov 01 20:44:56.835281 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:56.835171 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04242230-db36-48bc-b020-06b79d6ddab4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06fecfbb-7wr8m\" (UID: \"04242230-db36-48bc-b020-06b79d6ddab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fecfbb-7wr8m" Nov 01 20:44:56.919173 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:56.919147 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b8846bc-cwg7c" Nov 01 20:44:56.919356 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:56.919146 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b8846bc-cwg7c" event={"ID":"9d3a7b07-765e-4eec-8948-ff6366bb6c9a","Type":"ContainerDied","Data":"ed45fb6a08f9f1c7ae592714d012e9ccd73f253ba74f462b1a14561274c203cc"} Nov 01 20:44:56.919356 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:56.919273 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed45fb6a08f9f1c7ae592714d012e9ccd73f253ba74f462b1a14561274c203cc" Nov 01 20:44:56.936654 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:56.936626 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04242230-db36-48bc-b020-06b79d6ddab4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06fecfbb-7wr8m\" (UID: \"04242230-db36-48bc-b020-06b79d6ddab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fecfbb-7wr8m" Nov 01 20:44:56.936783 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:56.936658 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st2cl\" (UniqueName: \"kubernetes.io/projected/04242230-db36-48bc-b020-06b79d6ddab4-kube-api-access-st2cl\") pod \"maas-api-revoke-sub-06fecfbb-7wr8m\" (UID: \"04242230-db36-48bc-b020-06b79d6ddab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fecfbb-7wr8m" Nov 01 20:44:56.936783 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:56.936710 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04242230-db36-48bc-b020-06b79d6ddab4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06fecfbb-7wr8m\" (UID: \"04242230-db36-48bc-b020-06b79d6ddab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fecfbb-7wr8m" Nov 01 20:44:56.937447 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:56.937421 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04242230-db36-48bc-b020-06b79d6ddab4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06fecfbb-7wr8m\" (UID: \"04242230-db36-48bc-b020-06b79d6ddab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fecfbb-7wr8m" Nov 01 20:44:56.938894 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:56.938872 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04242230-db36-48bc-b020-06b79d6ddab4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06fecfbb-7wr8m\" (UID: \"04242230-db36-48bc-b020-06b79d6ddab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fecfbb-7wr8m" Nov 01 20:44:56.944464 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:56.944441 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:44:56.994623 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:56.994543 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-st2cl\" (UniqueName: \"kubernetes.io/projected/04242230-db36-48bc-b020-06b79d6ddab4-kube-api-access-st2cl\") pod \"maas-api-revoke-sub-06fecfbb-7wr8m\" (UID: \"04242230-db36-48bc-b020-06b79d6ddab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fecfbb-7wr8m" Nov 01 20:44:57.057456 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:57.057425 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fecfbb-7wr8m" Nov 01 20:44:57.428924 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:57.428896 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06fecfbb-7wr8m"] Nov 01 20:44:57.430675 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:44:57.430648 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04242230_db36_48bc_b020_06b79d6ddab4.slice/crio-5678d03397b90025f08f3cdd19e726ac303ed1324e5e5e4fd2cf689b24d0a9a5 WatchSource:0}: Error finding container 5678d03397b90025f08f3cdd19e726ac303ed1324e5e5e4fd2cf689b24d0a9a5: Status 404 returned error can't find the container with id 5678d03397b90025f08f3cdd19e726ac303ed1324e5e5e4fd2cf689b24d0a9a5 Nov 01 20:44:57.926143 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:57.926105 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fecfbb-7wr8m" event={"ID":"04242230-db36-48bc-b020-06b79d6ddab4","Type":"ContainerStarted","Data":"0876d9565c2f13a9c2e32b006db9feed08d6be664a86909fef10f3dc7b07ad86"} Nov 01 20:44:57.926310 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:57.926157 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fecfbb-7wr8m" event={"ID":"04242230-db36-48bc-b020-06b79d6ddab4","Type":"ContainerStarted","Data":"5678d03397b90025f08f3cdd19e726ac303ed1324e5e5e4fd2cf689b24d0a9a5"} Nov 01 20:44:58.086284 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:58.086229 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fecfbb-7wr8m" podStartSLOduration=2.086213222 podStartE2EDuration="2.086213222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:44:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:44:58.08446274 +0000 UTC m=+848.862102854" watchObservedRunningTime="2026-11-01 20:44:58.086213222 +0000 UTC m=+848.863853348" Nov 01 20:44:58.940418 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:58.940383 4484 generic.go:356] "Generic (PLEG): container finished" podID="04242230-db36-48bc-b020-06b79d6ddab4" containerID="0876d9565c2f13a9c2e32b006db9feed08d6be664a86909fef10f3dc7b07ad86" exitCode=0 Nov 01 20:44:58.940802 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:44:58.940468 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fecfbb-7wr8m" event={"ID":"04242230-db36-48bc-b020-06b79d6ddab4","Type":"ContainerDied","Data":"0876d9565c2f13a9c2e32b006db9feed08d6be664a86909fef10f3dc7b07ad86"} Nov 01 20:45:00.165242 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.165200 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892765-wkqgl"] Nov 01 20:45:00.169340 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.169318 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892765-xp8qs"] Nov 01 20:45:00.169732 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.169476 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892765-wkqgl" Nov 01 20:45:00.173512 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.173494 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fecfbb-7wr8m" Nov 01 20:45:00.174397 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.174380 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892765-4z8cw"] Nov 01 20:45:00.174513 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.174482 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-xp8qs" Nov 01 20:45:00.174933 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.174917 4484 cpu_manager.go:414] "containerMap: removing container" podUID="04242230-db36-48bc-b020-06b79d6ddab4" containerName="revoke-keys" Nov 01 20:45:00.174990 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.174934 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04242230-db36-48bc-b020-06b79d6ddab4" containerName="revoke-keys" Nov 01 20:45:00.175100 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.175080 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="04242230-db36-48bc-b020-06b79d6ddab4" containerName="revoke-keys" Nov 01 20:45:00.179142 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.179124 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892765-4z8cw" Nov 01 20:45:00.214340 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.214298 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892765-r4rtj"] Nov 01 20:45:00.218333 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.218311 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892765-jkvj9"] Nov 01 20:45:00.218478 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.218436 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892765-r4rtj" Nov 01 20:45:00.222421 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.222400 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892765-jkvj9" Nov 01 20:45:00.267536 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.267487 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892765-wkqgl"] Nov 01 20:45:00.273775 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.271119 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892765-xp8qs"] Nov 01 20:45:00.274723 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.274695 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892765-4z8cw"] Nov 01 20:45:00.286139 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.286106 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04242230-db36-48bc-b020-06b79d6ddab4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04242230-db36-48bc-b020-06b79d6ddab4-maas-api-openshift-ca\") pod \"04242230-db36-48bc-b020-06b79d6ddab4\" (UID: \"04242230-db36-48bc-b020-06b79d6ddab4\") " Nov 01 20:45:00.286312 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.286155 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04242230-db36-48bc-b020-06b79d6ddab4-kube-api-access-st2cl\" (UniqueName: \"kubernetes.io/projected/04242230-db36-48bc-b020-06b79d6ddab4-kube-api-access-st2cl\") pod \"04242230-db36-48bc-b020-06b79d6ddab4\" (UID: \"04242230-db36-48bc-b020-06b79d6ddab4\") " Nov 01 20:45:00.286384 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.286319 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04242230-db36-48bc-b020-06b79d6ddab4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04242230-db36-48bc-b020-06b79d6ddab4-maas-api-xks-ca\") pod \"04242230-db36-48bc-b020-06b79d6ddab4\" (UID: \"04242230-db36-48bc-b020-06b79d6ddab4\") " Nov 01 20:45:00.286840 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.286558 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04242230-db36-48bc-b020-06b79d6ddab4-maas-api-openshift-ca" pod "04242230-db36-48bc-b020-06b79d6ddab4" (UID: "04242230-db36-48bc-b020-06b79d6ddab4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:45:00.287303 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.287276 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db922223-a186-42a9-9a5b-c44258bbb793-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-cd248f-29892765-r4rtj\" (UID: \"db922223-a186-42a9-9a5b-c44258bbb793\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892765-r4rtj" Nov 01 20:45:00.287406 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.287329 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d22905a-1b38-4c16-81fc-66f1cefd6c7e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-db0157-29892765-wkqgl\" (UID: \"3d22905a-1b38-4c16-81fc-66f1cefd6c7e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892765-wkqgl" Nov 01 20:45:00.287545 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.287504 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fnmv\" (UniqueName: \"kubernetes.io/projected/143424a0-981e-428f-ae6f-9c33ea0d5836-kube-api-access-9fnmv\") pod \"maas-api-key-cleanup-29892765-xp8qs\" (UID: \"143424a0-981e-428f-ae6f-9c33ea0d5836\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-xp8qs" Nov 01 20:45:00.287545 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.287534 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f16b797d-6930-414c-9957-bff940a16153-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892765-4z8cw\" (UID: \"f16b797d-6930-414c-9957-bff940a16153\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892765-4z8cw" Nov 01 20:45:00.287675 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.287557 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4964cf9-925d-4258-b8fe-2cc516b79487-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-208f2b-29892765-jkvj9\" (UID: \"b4964cf9-925d-4258-b8fe-2cc516b79487\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892765-jkvj9" Nov 01 20:45:00.287675 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.287581 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8lhr\" (UniqueName: \"kubernetes.io/projected/b4964cf9-925d-4258-b8fe-2cc516b79487-kube-api-access-p8lhr\") pod \"maas-api-key-cleanup-e2e-worker-w0-208f2b-29892765-jkvj9\" (UID: \"b4964cf9-925d-4258-b8fe-2cc516b79487\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892765-jkvj9" Nov 01 20:45:00.287675 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.287641 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8thx\" (UniqueName: \"kubernetes.io/projected/3d22905a-1b38-4c16-81fc-66f1cefd6c7e-kube-api-access-d8thx\") pod \"maas-api-key-cleanup-e2e-worker-w1-db0157-29892765-wkqgl\" (UID: \"3d22905a-1b38-4c16-81fc-66f1cefd6c7e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892765-wkqgl" Nov 01 20:45:00.287870 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.287844 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/143424a0-981e-428f-ae6f-9c33ea0d5836-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892765-xp8qs\" (UID: \"143424a0-981e-428f-ae6f-9c33ea0d5836\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-xp8qs" Nov 01 20:45:00.287948 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.287885 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvfpb\" (UniqueName: \"kubernetes.io/projected/db922223-a186-42a9-9a5b-c44258bbb793-kube-api-access-kvfpb\") pod \"maas-api-key-cleanup-e2e-worker-w2-cd248f-29892765-r4rtj\" (UID: \"db922223-a186-42a9-9a5b-c44258bbb793\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892765-r4rtj" Nov 01 20:45:00.288036 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.287930 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4h2k\" (UniqueName: \"kubernetes.io/projected/f16b797d-6930-414c-9957-bff940a16153-kube-api-access-m4h2k\") pod \"maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892765-4z8cw\" (UID: \"f16b797d-6930-414c-9957-bff940a16153\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892765-4z8cw" Nov 01 20:45:00.288219 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.288192 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04242230-db36-48bc-b020-06b79d6ddab4-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:00.296168 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.296113 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04242230-db36-48bc-b020-06b79d6ddab4-maas-api-xks-ca" pod "04242230-db36-48bc-b020-06b79d6ddab4" (UID: "04242230-db36-48bc-b020-06b79d6ddab4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:45:00.296168 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.296133 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04242230-db36-48bc-b020-06b79d6ddab4-kube-api-access-st2cl" pod "04242230-db36-48bc-b020-06b79d6ddab4" (UID: "04242230-db36-48bc-b020-06b79d6ddab4"). InnerVolumeSpecName "kube-api-access-st2cl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:45:00.334531 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.334493 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892765-r4rtj"] Nov 01 20:45:00.335642 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.335620 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892765-jkvj9"] Nov 01 20:45:00.389361 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.389323 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/143424a0-981e-428f-ae6f-9c33ea0d5836-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892765-xp8qs\" (UID: \"143424a0-981e-428f-ae6f-9c33ea0d5836\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-xp8qs" Nov 01 20:45:00.389361 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.389363 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvfpb\" (UniqueName: \"kubernetes.io/projected/db922223-a186-42a9-9a5b-c44258bbb793-kube-api-access-kvfpb\") pod \"maas-api-key-cleanup-e2e-worker-w2-cd248f-29892765-r4rtj\" (UID: \"db922223-a186-42a9-9a5b-c44258bbb793\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892765-r4rtj" Nov 01 20:45:00.389578 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.389406 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4h2k\" (UniqueName: \"kubernetes.io/projected/f16b797d-6930-414c-9957-bff940a16153-kube-api-access-m4h2k\") pod \"maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892765-4z8cw\" (UID: \"f16b797d-6930-414c-9957-bff940a16153\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892765-4z8cw" Nov 01 20:45:00.389578 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.389440 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db922223-a186-42a9-9a5b-c44258bbb793-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-cd248f-29892765-r4rtj\" (UID: \"db922223-a186-42a9-9a5b-c44258bbb793\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892765-r4rtj" Nov 01 20:45:00.389578 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.389468 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d22905a-1b38-4c16-81fc-66f1cefd6c7e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-db0157-29892765-wkqgl\" (UID: \"3d22905a-1b38-4c16-81fc-66f1cefd6c7e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892765-wkqgl" Nov 01 20:45:00.389578 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.389574 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fnmv\" (UniqueName: \"kubernetes.io/projected/143424a0-981e-428f-ae6f-9c33ea0d5836-kube-api-access-9fnmv\") pod \"maas-api-key-cleanup-29892765-xp8qs\" (UID: \"143424a0-981e-428f-ae6f-9c33ea0d5836\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-xp8qs" Nov 01 20:45:00.389704 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.389600 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f16b797d-6930-414c-9957-bff940a16153-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892765-4z8cw\" (UID: \"f16b797d-6930-414c-9957-bff940a16153\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892765-4z8cw" Nov 01 20:45:00.389704 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.389624 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4964cf9-925d-4258-b8fe-2cc516b79487-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-208f2b-29892765-jkvj9\" (UID: \"b4964cf9-925d-4258-b8fe-2cc516b79487\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892765-jkvj9" Nov 01 20:45:00.389704 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.389649 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8lhr\" (UniqueName: \"kubernetes.io/projected/b4964cf9-925d-4258-b8fe-2cc516b79487-kube-api-access-p8lhr\") pod \"maas-api-key-cleanup-e2e-worker-w0-208f2b-29892765-jkvj9\" (UID: \"b4964cf9-925d-4258-b8fe-2cc516b79487\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892765-jkvj9" Nov 01 20:45:00.389799 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.389706 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8thx\" (UniqueName: \"kubernetes.io/projected/3d22905a-1b38-4c16-81fc-66f1cefd6c7e-kube-api-access-d8thx\") pod \"maas-api-key-cleanup-e2e-worker-w1-db0157-29892765-wkqgl\" (UID: \"3d22905a-1b38-4c16-81fc-66f1cefd6c7e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892765-wkqgl" Nov 01 20:45:00.389832 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.389802 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04242230-db36-48bc-b020-06b79d6ddab4-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:00.389832 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.389821 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st2cl\" (UniqueName: \"kubernetes.io/projected/04242230-db36-48bc-b020-06b79d6ddab4-kube-api-access-st2cl\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:00.390325 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.390300 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/143424a0-981e-428f-ae6f-9c33ea0d5836-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892765-xp8qs\" (UID: \"143424a0-981e-428f-ae6f-9c33ea0d5836\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-xp8qs" Nov 01 20:45:00.390427 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.390407 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d22905a-1b38-4c16-81fc-66f1cefd6c7e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-db0157-29892765-wkqgl\" (UID: \"3d22905a-1b38-4c16-81fc-66f1cefd6c7e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892765-wkqgl" Nov 01 20:45:00.390584 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.390562 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f16b797d-6930-414c-9957-bff940a16153-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892765-4z8cw\" (UID: \"f16b797d-6930-414c-9957-bff940a16153\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892765-4z8cw" Nov 01 20:45:00.390790 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.390770 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4964cf9-925d-4258-b8fe-2cc516b79487-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-208f2b-29892765-jkvj9\" (UID: \"b4964cf9-925d-4258-b8fe-2cc516b79487\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892765-jkvj9" Nov 01 20:45:00.390989 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.390972 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db922223-a186-42a9-9a5b-c44258bbb793-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-cd248f-29892765-r4rtj\" (UID: \"db922223-a186-42a9-9a5b-c44258bbb793\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892765-r4rtj" Nov 01 20:45:00.482076 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.481598 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8thx\" (UniqueName: \"kubernetes.io/projected/3d22905a-1b38-4c16-81fc-66f1cefd6c7e-kube-api-access-d8thx\") pod \"maas-api-key-cleanup-e2e-worker-w1-db0157-29892765-wkqgl\" (UID: \"3d22905a-1b38-4c16-81fc-66f1cefd6c7e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892765-wkqgl" Nov 01 20:45:00.506018 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.493492 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892765-wkqgl" Nov 01 20:45:00.506018 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.501480 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fnmv\" (UniqueName: \"kubernetes.io/projected/143424a0-981e-428f-ae6f-9c33ea0d5836-kube-api-access-9fnmv\") pod \"maas-api-key-cleanup-29892765-xp8qs\" (UID: \"143424a0-981e-428f-ae6f-9c33ea0d5836\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-xp8qs" Nov 01 20:45:00.515066 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.510752 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvfpb\" (UniqueName: \"kubernetes.io/projected/db922223-a186-42a9-9a5b-c44258bbb793-kube-api-access-kvfpb\") pod \"maas-api-key-cleanup-e2e-worker-w2-cd248f-29892765-r4rtj\" (UID: \"db922223-a186-42a9-9a5b-c44258bbb793\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892765-r4rtj" Nov 01 20:45:00.515066 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.511529 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4h2k\" (UniqueName: \"kubernetes.io/projected/f16b797d-6930-414c-9957-bff940a16153-kube-api-access-m4h2k\") pod \"maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892765-4z8cw\" (UID: \"f16b797d-6930-414c-9957-bff940a16153\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892765-4z8cw" Nov 01 20:45:00.527572 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.525787 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8lhr\" (UniqueName: \"kubernetes.io/projected/b4964cf9-925d-4258-b8fe-2cc516b79487-kube-api-access-p8lhr\") pod \"maas-api-key-cleanup-e2e-worker-w0-208f2b-29892765-jkvj9\" (UID: \"b4964cf9-925d-4258-b8fe-2cc516b79487\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892765-jkvj9" Nov 01 20:45:00.551022 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.550502 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892765-r4rtj" Nov 01 20:45:00.552677 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.552587 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892765-jkvj9" Nov 01 20:45:00.796481 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.796131 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-xp8qs" Nov 01 20:45:00.816876 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.816836 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892765-4z8cw" Nov 01 20:45:00.967751 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.967717 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fecfbb-7wr8m" event={"ID":"04242230-db36-48bc-b020-06b79d6ddab4","Type":"ContainerDied","Data":"5678d03397b90025f08f3cdd19e726ac303ed1324e5e5e4fd2cf689b24d0a9a5"} Nov 01 20:45:00.967867 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.967784 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5678d03397b90025f08f3cdd19e726ac303ed1324e5e5e4fd2cf689b24d0a9a5" Nov 01 20:45:00.967867 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:00.967861 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fecfbb-7wr8m" Nov 01 20:45:01.007422 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:01.006709 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892765-r4rtj"] Nov 01 20:45:01.007422 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:45:01.006883 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb922223_a186_42a9_9a5b_c44258bbb793.slice/crio-50ee8bc15a1a14d42867295189beed8c39f0da2fb61da62fd3c6ddba314e6d92 WatchSource:0}: Error finding container 50ee8bc15a1a14d42867295189beed8c39f0da2fb61da62fd3c6ddba314e6d92: Status 404 returned error can't find the container with id 50ee8bc15a1a14d42867295189beed8c39f0da2fb61da62fd3c6ddba314e6d92 Nov 01 20:45:01.021861 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:01.021828 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892765-wkqgl"] Nov 01 20:45:01.055216 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:01.055169 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892765-jkvj9"] Nov 01 20:45:01.269581 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:01.269551 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892765-xp8qs"] Nov 01 20:45:01.273638 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:45:01.273610 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod143424a0_981e_428f_ae6f_9c33ea0d5836.slice/crio-8c506cec3ae8048d661385d42f9bbcb5f092b3fba65f0e6423899e1f5b9c0f24 WatchSource:0}: Error finding container 8c506cec3ae8048d661385d42f9bbcb5f092b3fba65f0e6423899e1f5b9c0f24: Status 404 returned error can't find the container with id 8c506cec3ae8048d661385d42f9bbcb5f092b3fba65f0e6423899e1f5b9c0f24 Nov 01 20:45:01.285363 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:01.285338 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892765-4z8cw"] Nov 01 20:45:01.290649 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:45:01.290611 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf16b797d_6930_414c_9957_bff940a16153.slice/crio-0197e28c3a5a4e419429103ddcb63853dac0825da1d98875e02075a0161b4cf1 WatchSource:0}: Error finding container 0197e28c3a5a4e419429103ddcb63853dac0825da1d98875e02075a0161b4cf1: Status 404 returned error can't find the container with id 0197e28c3a5a4e419429103ddcb63853dac0825da1d98875e02075a0161b4cf1 Nov 01 20:45:01.749769 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:01.749730 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:01.978114 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:01.977442 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892765-4z8cw" event={"ID":"f16b797d-6930-414c-9957-bff940a16153","Type":"ContainerStarted","Data":"090a2d869db9e976fa5754e53314dee6d77cd84b65024903841aba2901cfa7ef"} Nov 01 20:45:01.978114 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:01.977506 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892765-4z8cw" event={"ID":"f16b797d-6930-414c-9957-bff940a16153","Type":"ContainerStarted","Data":"0197e28c3a5a4e419429103ddcb63853dac0825da1d98875e02075a0161b4cf1"} Nov 01 20:45:01.979321 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:01.979293 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892765-jkvj9" event={"ID":"b4964cf9-925d-4258-b8fe-2cc516b79487","Type":"ContainerStarted","Data":"f797e80722a0a5664575fb4d38c07986081fe92699d9a7daf43a22775858dfbd"} Nov 01 20:45:01.979533 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:01.979496 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892765-jkvj9" event={"ID":"b4964cf9-925d-4258-b8fe-2cc516b79487","Type":"ContainerStarted","Data":"38770816bacdc75002d1a6d490bde25bfb6d819ff690cc3a483981cdb380de20"} Nov 01 20:45:01.983298 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:01.983266 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892765-wkqgl" event={"ID":"3d22905a-1b38-4c16-81fc-66f1cefd6c7e","Type":"ContainerStarted","Data":"a0bb565365e0843d1fe3da45a16b6f50dd24c1bd1a29bcde3d9704aa9f2cd369"} Nov 01 20:45:01.983436 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:01.983317 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892765-wkqgl" event={"ID":"3d22905a-1b38-4c16-81fc-66f1cefd6c7e","Type":"ContainerStarted","Data":"2c92e7d808fd27b6f664cb770084f9aea1d245da1eeeed1c97470cb2f958d169"} Nov 01 20:45:01.985061 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:01.985023 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-xp8qs" event={"ID":"143424a0-981e-428f-ae6f-9c33ea0d5836","Type":"ContainerStarted","Data":"ddc7d94397a2f397fe5c5365f454465626e0fd4c3f43c5e7f476aa28064c3586"} Nov 01 20:45:01.985186 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:01.985072 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-xp8qs" event={"ID":"143424a0-981e-428f-ae6f-9c33ea0d5836","Type":"ContainerStarted","Data":"8c506cec3ae8048d661385d42f9bbcb5f092b3fba65f0e6423899e1f5b9c0f24"} Nov 01 20:45:01.987053 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:01.986957 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892765-r4rtj" event={"ID":"db922223-a186-42a9-9a5b-c44258bbb793","Type":"ContainerStarted","Data":"0a618699c067ce2876559bc2ccf8ac6f640b1b72e3ad393d075162fcf1315009"} Nov 01 20:45:01.987243 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:01.987202 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892765-r4rtj" event={"ID":"db922223-a186-42a9-9a5b-c44258bbb793","Type":"ContainerStarted","Data":"50ee8bc15a1a14d42867295189beed8c39f0da2fb61da62fd3c6ddba314e6d92"} Nov 01 20:45:02.093172 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:02.093063 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892765-4z8cw" podStartSLOduration=2.093049586 podStartE2EDuration="2.093049586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:45:02.091244144 +0000 UTC m=+852.868884279" watchObservedRunningTime="2026-11-01 20:45:02.093049586 +0000 UTC m=+852.870689712" Nov 01 20:45:02.209777 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:02.209528 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892765-wkqgl" podStartSLOduration=2.209509304 podStartE2EDuration="2.209509304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:45:02.208416783 +0000 UTC m=+852.986056918" watchObservedRunningTime="2026-11-01 20:45:02.209509304 +0000 UTC m=+852.987149439" Nov 01 20:45:02.325071 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:02.324988 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-xp8qs" podStartSLOduration=2.324975263 podStartE2EDuration="2.324975263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:45:02.324475414 +0000 UTC m=+853.102115583" watchObservedRunningTime="2026-11-01 20:45:02.324975263 +0000 UTC m=+853.102615397" Nov 01 20:45:02.442965 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:02.442901 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892765-r4rtj" podStartSLOduration=2.442879999 podStartE2EDuration="2.442879999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:45:02.438967911 +0000 UTC m=+853.216608045" watchObservedRunningTime="2026-11-01 20:45:02.442879999 +0000 UTC m=+853.220520135" Nov 01 20:45:02.557187 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:02.557116 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892765-jkvj9" podStartSLOduration=2.557095145 podStartE2EDuration="2.557095145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:45:02.554457286 +0000 UTC m=+853.332097421" watchObservedRunningTime="2026-11-01 20:45:02.557095145 +0000 UTC m=+853.334735282" Nov 01 20:45:02.995277 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:02.995241 4484 generic.go:356] "Generic (PLEG): container finished" podID="f16b797d-6930-414c-9957-bff940a16153" containerID="090a2d869db9e976fa5754e53314dee6d77cd84b65024903841aba2901cfa7ef" exitCode=60 Nov 01 20:45:02.995589 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:02.995327 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892765-4z8cw" event={"ID":"f16b797d-6930-414c-9957-bff940a16153","Type":"ContainerDied","Data":"090a2d869db9e976fa5754e53314dee6d77cd84b65024903841aba2901cfa7ef"} Nov 01 20:45:02.995693 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:02.995610 4484 scope.go:122] "RemoveContainer" containerID="090a2d869db9e976fa5754e53314dee6d77cd84b65024903841aba2901cfa7ef" Nov 01 20:45:02.997279 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:02.997248 4484 generic.go:356] "Generic (PLEG): container finished" podID="b4964cf9-925d-4258-b8fe-2cc516b79487" containerID="f797e80722a0a5664575fb4d38c07986081fe92699d9a7daf43a22775858dfbd" exitCode=60 Nov 01 20:45:02.997400 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:02.997317 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892765-jkvj9" event={"ID":"b4964cf9-925d-4258-b8fe-2cc516b79487","Type":"ContainerDied","Data":"f797e80722a0a5664575fb4d38c07986081fe92699d9a7daf43a22775858dfbd"} Nov 01 20:45:02.997624 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:02.997608 4484 scope.go:122] "RemoveContainer" containerID="f797e80722a0a5664575fb4d38c07986081fe92699d9a7daf43a22775858dfbd" Nov 01 20:45:03.000202 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:03.000178 4484 generic.go:356] "Generic (PLEG): container finished" podID="3d22905a-1b38-4c16-81fc-66f1cefd6c7e" containerID="a0bb565365e0843d1fe3da45a16b6f50dd24c1bd1a29bcde3d9704aa9f2cd369" exitCode=60 Nov 01 20:45:03.000311 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:03.000289 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892765-wkqgl" event={"ID":"3d22905a-1b38-4c16-81fc-66f1cefd6c7e","Type":"ContainerDied","Data":"a0bb565365e0843d1fe3da45a16b6f50dd24c1bd1a29bcde3d9704aa9f2cd369"} Nov 01 20:45:03.000468 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:03.000450 4484 scope.go:122] "RemoveContainer" containerID="a0bb565365e0843d1fe3da45a16b6f50dd24c1bd1a29bcde3d9704aa9f2cd369" Nov 01 20:45:03.003292 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:03.003267 4484 generic.go:356] "Generic (PLEG): container finished" podID="143424a0-981e-428f-ae6f-9c33ea0d5836" containerID="ddc7d94397a2f397fe5c5365f454465626e0fd4c3f43c5e7f476aa28064c3586" exitCode=60 Nov 01 20:45:03.003404 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:03.003347 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-xp8qs" event={"ID":"143424a0-981e-428f-ae6f-9c33ea0d5836","Type":"ContainerDied","Data":"ddc7d94397a2f397fe5c5365f454465626e0fd4c3f43c5e7f476aa28064c3586"} Nov 01 20:45:03.003556 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:03.003541 4484 scope.go:122] "RemoveContainer" containerID="ddc7d94397a2f397fe5c5365f454465626e0fd4c3f43c5e7f476aa28064c3586" Nov 01 20:45:03.005214 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:03.005179 4484 generic.go:356] "Generic (PLEG): container finished" podID="db922223-a186-42a9-9a5b-c44258bbb793" containerID="0a618699c067ce2876559bc2ccf8ac6f640b1b72e3ad393d075162fcf1315009" exitCode=60 Nov 01 20:45:03.005302 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:03.005265 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892765-r4rtj" event={"ID":"db922223-a186-42a9-9a5b-c44258bbb793","Type":"ContainerDied","Data":"0a618699c067ce2876559bc2ccf8ac6f640b1b72e3ad393d075162fcf1315009"} Nov 01 20:45:03.005434 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:03.005418 4484 scope.go:122] "RemoveContainer" containerID="0a618699c067ce2876559bc2ccf8ac6f640b1b72e3ad393d075162fcf1315009" Nov 01 20:45:03.071277 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:03.071246 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:03.238475 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:03.238441 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:04.016085 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:04.016052 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892765-wkqgl" event={"ID":"3d22905a-1b38-4c16-81fc-66f1cefd6c7e","Type":"ContainerStarted","Data":"1b98ae9085a508a3427ad2ad9896b22f00d379ab9fa6a19a89f99af2581524a3"} Nov 01 20:45:04.019062 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:04.019032 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-xp8qs" event={"ID":"143424a0-981e-428f-ae6f-9c33ea0d5836","Type":"ContainerStarted","Data":"a272bdefe7c2e539e321a9f3641700431ad83c5c33d55a2e8fbc4c4c41834dfc"} Nov 01 20:45:04.019381 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:04.019363 4484 scope.go:122] "RemoveContainer" containerID="a272bdefe7c2e539e321a9f3641700431ad83c5c33d55a2e8fbc4c4c41834dfc" Nov 01 20:45:04.019696 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:45:04.019664 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892765-xp8qs_odh-ai-gateway-infra(143424a0-981e-428f-ae6f-9c33ea0d5836)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-xp8qs" podUID="143424a0-981e-428f-ae6f-9c33ea0d5836" Nov 01 20:45:04.021201 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:04.021173 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892765-r4rtj" event={"ID":"db922223-a186-42a9-9a5b-c44258bbb793","Type":"ContainerStarted","Data":"25f09ad4c34f658ecc6fbca4c3e8a245dd13712874875d385cb5e2bc18fd0917"} Nov 01 20:45:04.021416 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:04.021398 4484 scope.go:122] "RemoveContainer" containerID="25f09ad4c34f658ecc6fbca4c3e8a245dd13712874875d385cb5e2bc18fd0917" Nov 01 20:45:04.021678 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:45:04.021656 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-cd248f-29892765-r4rtj_odh-ai-gateway-infra(db922223-a186-42a9-9a5b-c44258bbb793)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892765-r4rtj" podUID="db922223-a186-42a9-9a5b-c44258bbb793" Nov 01 20:45:04.023234 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:04.023208 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892765-4z8cw" event={"ID":"f16b797d-6930-414c-9957-bff940a16153","Type":"ContainerStarted","Data":"5ea1d56f393bd22c3a866ab36141cf3a21d1cdf539dffdd32219566ce8e20d77"} Nov 01 20:45:04.025049 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:04.025027 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892765-jkvj9" event={"ID":"b4964cf9-925d-4258-b8fe-2cc516b79487","Type":"ContainerStarted","Data":"4bad70676e67ccdc8b5abee8ba0b94562af0316274d5e379c80aac8fa53e1e9b"} Nov 01 20:45:04.025308 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:04.025291 4484 scope.go:122] "RemoveContainer" containerID="4bad70676e67ccdc8b5abee8ba0b94562af0316274d5e379c80aac8fa53e1e9b" Nov 01 20:45:04.025585 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:45:04.025562 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-208f2b-29892765-jkvj9_odh-ai-gateway-infra(b4964cf9-925d-4258-b8fe-2cc516b79487)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892765-jkvj9" podUID="b4964cf9-925d-4258-b8fe-2cc516b79487" Nov 01 20:45:05.032923 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:05.032883 4484 generic.go:356] "Generic (PLEG): container finished" podID="f16b797d-6930-414c-9957-bff940a16153" containerID="5ea1d56f393bd22c3a866ab36141cf3a21d1cdf539dffdd32219566ce8e20d77" exitCode=60 Nov 01 20:45:05.033433 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:05.032960 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892765-4z8cw" event={"ID":"f16b797d-6930-414c-9957-bff940a16153","Type":"ContainerDied","Data":"5ea1d56f393bd22c3a866ab36141cf3a21d1cdf539dffdd32219566ce8e20d77"} Nov 01 20:45:05.033433 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:05.033088 4484 scope.go:122] "RemoveContainer" containerID="090a2d869db9e976fa5754e53314dee6d77cd84b65024903841aba2901cfa7ef" Nov 01 20:45:05.033433 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:05.033287 4484 scope.go:122] "RemoveContainer" containerID="5ea1d56f393bd22c3a866ab36141cf3a21d1cdf539dffdd32219566ce8e20d77" Nov 01 20:45:05.033630 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:45:05.033583 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892765-4z8cw_odh-ai-gateway-infra(f16b797d-6930-414c-9957-bff940a16153)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892765-4z8cw" podUID="f16b797d-6930-414c-9957-bff940a16153" Nov 01 20:45:05.034942 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:05.034918 4484 generic.go:356] "Generic (PLEG): container finished" podID="b4964cf9-925d-4258-b8fe-2cc516b79487" containerID="4bad70676e67ccdc8b5abee8ba0b94562af0316274d5e379c80aac8fa53e1e9b" exitCode=60 Nov 01 20:45:05.035080 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:05.035031 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892765-jkvj9" event={"ID":"b4964cf9-925d-4258-b8fe-2cc516b79487","Type":"ContainerDied","Data":"4bad70676e67ccdc8b5abee8ba0b94562af0316274d5e379c80aac8fa53e1e9b"} Nov 01 20:45:05.035294 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:05.035277 4484 scope.go:122] "RemoveContainer" containerID="4bad70676e67ccdc8b5abee8ba0b94562af0316274d5e379c80aac8fa53e1e9b" Nov 01 20:45:05.035560 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:45:05.035536 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-208f2b-29892765-jkvj9_odh-ai-gateway-infra(b4964cf9-925d-4258-b8fe-2cc516b79487)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892765-jkvj9" podUID="b4964cf9-925d-4258-b8fe-2cc516b79487" Nov 01 20:45:05.036764 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:05.036745 4484 generic.go:356] "Generic (PLEG): container finished" podID="3d22905a-1b38-4c16-81fc-66f1cefd6c7e" containerID="1b98ae9085a508a3427ad2ad9896b22f00d379ab9fa6a19a89f99af2581524a3" exitCode=60 Nov 01 20:45:05.036850 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:05.036798 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892765-wkqgl" event={"ID":"3d22905a-1b38-4c16-81fc-66f1cefd6c7e","Type":"ContainerDied","Data":"1b98ae9085a508a3427ad2ad9896b22f00d379ab9fa6a19a89f99af2581524a3"} Nov 01 20:45:05.037078 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:05.037060 4484 scope.go:122] "RemoveContainer" containerID="1b98ae9085a508a3427ad2ad9896b22f00d379ab9fa6a19a89f99af2581524a3" Nov 01 20:45:05.037341 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:45:05.037324 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-db0157-29892765-wkqgl_odh-ai-gateway-infra(3d22905a-1b38-4c16-81fc-66f1cefd6c7e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892765-wkqgl" podUID="3d22905a-1b38-4c16-81fc-66f1cefd6c7e" Nov 01 20:45:05.038619 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:05.038589 4484 generic.go:356] "Generic (PLEG): container finished" podID="143424a0-981e-428f-ae6f-9c33ea0d5836" containerID="a272bdefe7c2e539e321a9f3641700431ad83c5c33d55a2e8fbc4c4c41834dfc" exitCode=60 Nov 01 20:45:05.038705 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:05.038645 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-xp8qs" event={"ID":"143424a0-981e-428f-ae6f-9c33ea0d5836","Type":"ContainerDied","Data":"a272bdefe7c2e539e321a9f3641700431ad83c5c33d55a2e8fbc4c4c41834dfc"} Nov 01 20:45:05.038914 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:05.038902 4484 scope.go:122] "RemoveContainer" containerID="a272bdefe7c2e539e321a9f3641700431ad83c5c33d55a2e8fbc4c4c41834dfc" Nov 01 20:45:05.039182 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:45:05.039159 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892765-xp8qs_odh-ai-gateway-infra(143424a0-981e-428f-ae6f-9c33ea0d5836)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-xp8qs" podUID="143424a0-981e-428f-ae6f-9c33ea0d5836" Nov 01 20:45:05.040743 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:05.040721 4484 generic.go:356] "Generic (PLEG): container finished" podID="db922223-a186-42a9-9a5b-c44258bbb793" containerID="25f09ad4c34f658ecc6fbca4c3e8a245dd13712874875d385cb5e2bc18fd0917" exitCode=60 Nov 01 20:45:05.040839 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:05.040801 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892765-r4rtj" event={"ID":"db922223-a186-42a9-9a5b-c44258bbb793","Type":"ContainerDied","Data":"25f09ad4c34f658ecc6fbca4c3e8a245dd13712874875d385cb5e2bc18fd0917"} Nov 01 20:45:05.040976 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:05.040962 4484 scope.go:122] "RemoveContainer" containerID="25f09ad4c34f658ecc6fbca4c3e8a245dd13712874875d385cb5e2bc18fd0917" Nov 01 20:45:05.041173 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:45:05.041156 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-cd248f-29892765-r4rtj_odh-ai-gateway-infra(db922223-a186-42a9-9a5b-c44258bbb793)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892765-r4rtj" podUID="db922223-a186-42a9-9a5b-c44258bbb793" Nov 01 20:45:05.045640 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:05.045625 4484 scope.go:122] "RemoveContainer" containerID="f797e80722a0a5664575fb4d38c07986081fe92699d9a7daf43a22775858dfbd" Nov 01 20:45:05.058118 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:05.058094 4484 scope.go:122] "RemoveContainer" containerID="a0bb565365e0843d1fe3da45a16b6f50dd24c1bd1a29bcde3d9704aa9f2cd369" Nov 01 20:45:05.071107 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:05.071069 4484 scope.go:122] "RemoveContainer" containerID="ddc7d94397a2f397fe5c5365f454465626e0fd4c3f43c5e7f476aa28064c3586" Nov 01 20:45:05.083655 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:05.083634 4484 scope.go:122] "RemoveContainer" containerID="0a618699c067ce2876559bc2ccf8ac6f640b1b72e3ad393d075162fcf1315009" Nov 01 20:45:06.049399 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:06.049372 4484 scope.go:122] "RemoveContainer" containerID="1b98ae9085a508a3427ad2ad9896b22f00d379ab9fa6a19a89f99af2581524a3" Nov 01 20:45:06.049831 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:45:06.049668 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-db0157-29892765-wkqgl_odh-ai-gateway-infra(3d22905a-1b38-4c16-81fc-66f1cefd6c7e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892765-wkqgl" podUID="3d22905a-1b38-4c16-81fc-66f1cefd6c7e" Nov 01 20:45:06.051004 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:06.050969 4484 scope.go:122] "RemoveContainer" containerID="a272bdefe7c2e539e321a9f3641700431ad83c5c33d55a2e8fbc4c4c41834dfc" Nov 01 20:45:06.051280 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:45:06.051257 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892765-xp8qs_odh-ai-gateway-infra(143424a0-981e-428f-ae6f-9c33ea0d5836)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-xp8qs" podUID="143424a0-981e-428f-ae6f-9c33ea0d5836" Nov 01 20:45:06.052559 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:06.052534 4484 scope.go:122] "RemoveContainer" containerID="25f09ad4c34f658ecc6fbca4c3e8a245dd13712874875d385cb5e2bc18fd0917" Nov 01 20:45:06.052796 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:45:06.052775 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-cd248f-29892765-r4rtj_odh-ai-gateway-infra(db922223-a186-42a9-9a5b-c44258bbb793)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892765-r4rtj" podUID="db922223-a186-42a9-9a5b-c44258bbb793" Nov 01 20:45:06.054036 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:06.054019 4484 scope.go:122] "RemoveContainer" containerID="5ea1d56f393bd22c3a866ab36141cf3a21d1cdf539dffdd32219566ce8e20d77" Nov 01 20:45:06.054284 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:45:06.054260 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892765-4z8cw_odh-ai-gateway-infra(f16b797d-6930-414c-9957-bff940a16153)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892765-4z8cw" podUID="f16b797d-6930-414c-9957-bff940a16153" Nov 01 20:45:06.055451 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:06.055438 4484 scope.go:122] "RemoveContainer" containerID="4bad70676e67ccdc8b5abee8ba0b94562af0316274d5e379c80aac8fa53e1e9b" Nov 01 20:45:06.055650 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:45:06.055634 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-208f2b-29892765-jkvj9_odh-ai-gateway-infra(b4964cf9-925d-4258-b8fe-2cc516b79487)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892765-jkvj9" podUID="b4964cf9-925d-4258-b8fe-2cc516b79487" Nov 01 20:45:06.551556 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:06.551527 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:09.969003 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:09.968954 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8f6bbf1-6ljkf"] Nov 01 20:45:09.975553 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:09.975528 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f6bbf1-6ljkf" Nov 01 20:45:10.025269 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:10.025214 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt26l\" (UniqueName: \"kubernetes.io/projected/226c28bf-b17c-4eae-819a-03fb580dc870-kube-api-access-rt26l\") pod \"maas-api-revoke-sub-e8f6bbf1-6ljkf\" (UID: \"226c28bf-b17c-4eae-819a-03fb580dc870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f6bbf1-6ljkf" Nov 01 20:45:10.025472 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:10.025417 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/226c28bf-b17c-4eae-819a-03fb580dc870-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8f6bbf1-6ljkf\" (UID: \"226c28bf-b17c-4eae-819a-03fb580dc870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f6bbf1-6ljkf" Nov 01 20:45:10.025472 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:10.025459 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/226c28bf-b17c-4eae-819a-03fb580dc870-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8f6bbf1-6ljkf\" (UID: \"226c28bf-b17c-4eae-819a-03fb580dc870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f6bbf1-6ljkf" Nov 01 20:45:10.030187 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:10.030161 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8f6bbf1-6ljkf"] Nov 01 20:45:10.077027 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:10.076977 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:45:10.126500 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:10.126464 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/226c28bf-b17c-4eae-819a-03fb580dc870-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8f6bbf1-6ljkf\" (UID: \"226c28bf-b17c-4eae-819a-03fb580dc870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f6bbf1-6ljkf" Nov 01 20:45:10.126697 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:10.126555 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt26l\" (UniqueName: \"kubernetes.io/projected/226c28bf-b17c-4eae-819a-03fb580dc870-kube-api-access-rt26l\") pod \"maas-api-revoke-sub-e8f6bbf1-6ljkf\" (UID: \"226c28bf-b17c-4eae-819a-03fb580dc870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f6bbf1-6ljkf" Nov 01 20:45:10.126767 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:10.126712 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/226c28bf-b17c-4eae-819a-03fb580dc870-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8f6bbf1-6ljkf\" (UID: \"226c28bf-b17c-4eae-819a-03fb580dc870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f6bbf1-6ljkf" Nov 01 20:45:10.127372 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:10.127348 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/226c28bf-b17c-4eae-819a-03fb580dc870-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8f6bbf1-6ljkf\" (UID: \"226c28bf-b17c-4eae-819a-03fb580dc870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f6bbf1-6ljkf" Nov 01 20:45:10.129006 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:10.128974 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/226c28bf-b17c-4eae-819a-03fb580dc870-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8f6bbf1-6ljkf\" (UID: \"226c28bf-b17c-4eae-819a-03fb580dc870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f6bbf1-6ljkf" Nov 01 20:45:10.186471 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:10.186437 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt26l\" (UniqueName: \"kubernetes.io/projected/226c28bf-b17c-4eae-819a-03fb580dc870-kube-api-access-rt26l\") pod \"maas-api-revoke-sub-e8f6bbf1-6ljkf\" (UID: \"226c28bf-b17c-4eae-819a-03fb580dc870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f6bbf1-6ljkf" Nov 01 20:45:10.290033 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:10.289976 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f6bbf1-6ljkf" Nov 01 20:45:10.665930 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:10.665894 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8f6bbf1-6ljkf"] Nov 01 20:45:10.666486 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:45:10.666460 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod226c28bf_b17c_4eae_819a_03fb580dc870.slice/crio-298c03bd22721e746e07035f402dd6284cc497dae39aa42e51deecd15e5d245d WatchSource:0}: Error finding container 298c03bd22721e746e07035f402dd6284cc497dae39aa42e51deecd15e5d245d: Status 404 returned error can't find the container with id 298c03bd22721e746e07035f402dd6284cc497dae39aa42e51deecd15e5d245d Nov 01 20:45:11.086721 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:11.086681 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f6bbf1-6ljkf" event={"ID":"226c28bf-b17c-4eae-819a-03fb580dc870","Type":"ContainerStarted","Data":"38364e9a9caa053c7725f945528c63de0a4954b68063236647bb05c742a1ffbe"} Nov 01 20:45:11.086721 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:11.086745 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f6bbf1-6ljkf" event={"ID":"226c28bf-b17c-4eae-819a-03fb580dc870","Type":"ContainerStarted","Data":"298c03bd22721e746e07035f402dd6284cc497dae39aa42e51deecd15e5d245d"} Nov 01 20:45:11.202214 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:11.202106 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f6bbf1-6ljkf" podStartSLOduration=2.202091943 podStartE2EDuration="2.202091943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:45:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:45:11.201445928 +0000 UTC m=+861.979086098" watchObservedRunningTime="2026-11-01 20:45:11.202091943 +0000 UTC m=+861.979732077" Nov 01 20:45:11.418480 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:11.418450 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:12.094430 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:12.094396 4484 generic.go:356] "Generic (PLEG): container finished" podID="226c28bf-b17c-4eae-819a-03fb580dc870" containerID="38364e9a9caa053c7725f945528c63de0a4954b68063236647bb05c742a1ffbe" exitCode=0 Nov 01 20:45:12.094843 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:12.094491 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f6bbf1-6ljkf" event={"ID":"226c28bf-b17c-4eae-819a-03fb580dc870","Type":"ContainerDied","Data":"38364e9a9caa053c7725f945528c63de0a4954b68063236647bb05c742a1ffbe"} Nov 01 20:45:13.257502 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:13.257474 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f6bbf1-6ljkf" Nov 01 20:45:13.377900 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:13.377810 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/226c28bf-b17c-4eae-819a-03fb580dc870-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/226c28bf-b17c-4eae-819a-03fb580dc870-maas-api-openshift-ca\") pod \"226c28bf-b17c-4eae-819a-03fb580dc870\" (UID: \"226c28bf-b17c-4eae-819a-03fb580dc870\") " Nov 01 20:45:13.377900 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:13.377865 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/226c28bf-b17c-4eae-819a-03fb580dc870-kube-api-access-rt26l\" (UniqueName: \"kubernetes.io/projected/226c28bf-b17c-4eae-819a-03fb580dc870-kube-api-access-rt26l\") pod \"226c28bf-b17c-4eae-819a-03fb580dc870\" (UID: \"226c28bf-b17c-4eae-819a-03fb580dc870\") " Nov 01 20:45:13.378142 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:13.377910 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/226c28bf-b17c-4eae-819a-03fb580dc870-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/226c28bf-b17c-4eae-819a-03fb580dc870-maas-api-xks-ca\") pod \"226c28bf-b17c-4eae-819a-03fb580dc870\" (UID: \"226c28bf-b17c-4eae-819a-03fb580dc870\") " Nov 01 20:45:13.378248 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:13.378219 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/226c28bf-b17c-4eae-819a-03fb580dc870-maas-api-openshift-ca" pod "226c28bf-b17c-4eae-819a-03fb580dc870" (UID: "226c28bf-b17c-4eae-819a-03fb580dc870"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:45:13.378425 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:13.378404 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/226c28bf-b17c-4eae-819a-03fb580dc870-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:13.381150 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:13.381119 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/226c28bf-b17c-4eae-819a-03fb580dc870-kube-api-access-rt26l" pod "226c28bf-b17c-4eae-819a-03fb580dc870" (UID: "226c28bf-b17c-4eae-819a-03fb580dc870"). InnerVolumeSpecName "kube-api-access-rt26l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:45:13.381333 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:13.381197 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/226c28bf-b17c-4eae-819a-03fb580dc870-maas-api-xks-ca" pod "226c28bf-b17c-4eae-819a-03fb580dc870" (UID: "226c28bf-b17c-4eae-819a-03fb580dc870"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:45:13.479542 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:13.479496 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt26l\" (UniqueName: \"kubernetes.io/projected/226c28bf-b17c-4eae-819a-03fb580dc870-kube-api-access-rt26l\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:13.479542 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:13.479537 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/226c28bf-b17c-4eae-819a-03fb580dc870-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:14.108903 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:14.108860 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f6bbf1-6ljkf" Nov 01 20:45:14.108903 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:14.108892 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f6bbf1-6ljkf" event={"ID":"226c28bf-b17c-4eae-819a-03fb580dc870","Type":"ContainerDied","Data":"298c03bd22721e746e07035f402dd6284cc497dae39aa42e51deecd15e5d245d"} Nov 01 20:45:14.109147 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:14.108939 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="298c03bd22721e746e07035f402dd6284cc497dae39aa42e51deecd15e5d245d" Nov 01 20:45:15.364683 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:15.364653 4484 scope.go:122] "RemoveContainer" containerID="25f09ad4c34f658ecc6fbca4c3e8a245dd13712874875d385cb5e2bc18fd0917" Nov 01 20:45:15.777632 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:45:15.777594 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 20:45:16.123391 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:16.123361 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892765-r4rtj" event={"ID":"db922223-a186-42a9-9a5b-c44258bbb793","Type":"ContainerStarted","Data":"083618f1c29254275a201f47de8980f6b475481661b8a5a693e42a44bace91c9"} Nov 01 20:45:16.123782 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:16.123764 4484 scope.go:122] "RemoveContainer" containerID="083618f1c29254275a201f47de8980f6b475481661b8a5a693e42a44bace91c9" Nov 01 20:45:16.124097 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:45:16.124078 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-cd248f-29892765-r4rtj_odh-ai-gateway-infra(db922223-a186-42a9-9a5b-c44258bbb793)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892765-r4rtj" podUID="db922223-a186-42a9-9a5b-c44258bbb793" Nov 01 20:45:16.241627 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:16.241592 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:16.742562 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:16.742529 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a7a217d-w4jzg"] Nov 01 20:45:16.743170 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:16.743155 4484 cpu_manager.go:414] "containerMap: removing container" podUID="226c28bf-b17c-4eae-819a-03fb580dc870" containerName="revoke-keys" Nov 01 20:45:16.743226 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:16.743171 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="226c28bf-b17c-4eae-819a-03fb580dc870" containerName="revoke-keys" Nov 01 20:45:16.743293 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:16.743262 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="226c28bf-b17c-4eae-819a-03fb580dc870" containerName="revoke-keys" Nov 01 20:45:16.746615 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:16.746593 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7a217d-w4jzg" Nov 01 20:45:16.800476 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:16.800443 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:45:16.802543 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:16.802515 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a7a217d-w4jzg"] Nov 01 20:45:16.819312 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:16.819267 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dee25581-08e6-472d-b589-ccd44b94bd5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a7a217d-w4jzg\" (UID: \"dee25581-08e6-472d-b589-ccd44b94bd5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7a217d-w4jzg" Nov 01 20:45:16.819312 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:16.819303 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dee25581-08e6-472d-b589-ccd44b94bd5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a7a217d-w4jzg\" (UID: \"dee25581-08e6-472d-b589-ccd44b94bd5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7a217d-w4jzg" Nov 01 20:45:16.819312 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:16.819318 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr8m7\" (UniqueName: \"kubernetes.io/projected/dee25581-08e6-472d-b589-ccd44b94bd5f-kube-api-access-fr8m7\") pod \"maas-api-revoke-sub-2a7a217d-w4jzg\" (UID: \"dee25581-08e6-472d-b589-ccd44b94bd5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7a217d-w4jzg" Nov 01 20:45:16.920320 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:16.920287 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dee25581-08e6-472d-b589-ccd44b94bd5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a7a217d-w4jzg\" (UID: \"dee25581-08e6-472d-b589-ccd44b94bd5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7a217d-w4jzg" Nov 01 20:45:16.920506 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:16.920327 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dee25581-08e6-472d-b589-ccd44b94bd5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a7a217d-w4jzg\" (UID: \"dee25581-08e6-472d-b589-ccd44b94bd5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7a217d-w4jzg" Nov 01 20:45:16.920506 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:16.920353 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr8m7\" (UniqueName: \"kubernetes.io/projected/dee25581-08e6-472d-b589-ccd44b94bd5f-kube-api-access-fr8m7\") pod \"maas-api-revoke-sub-2a7a217d-w4jzg\" (UID: \"dee25581-08e6-472d-b589-ccd44b94bd5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7a217d-w4jzg" Nov 01 20:45:16.920926 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:16.920894 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dee25581-08e6-472d-b589-ccd44b94bd5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a7a217d-w4jzg\" (UID: \"dee25581-08e6-472d-b589-ccd44b94bd5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7a217d-w4jzg" Nov 01 20:45:16.922521 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:16.922503 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dee25581-08e6-472d-b589-ccd44b94bd5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a7a217d-w4jzg\" (UID: \"dee25581-08e6-472d-b589-ccd44b94bd5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7a217d-w4jzg" Nov 01 20:45:16.978183 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:16.978154 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr8m7\" (UniqueName: \"kubernetes.io/projected/dee25581-08e6-472d-b589-ccd44b94bd5f-kube-api-access-fr8m7\") pod \"maas-api-revoke-sub-2a7a217d-w4jzg\" (UID: \"dee25581-08e6-472d-b589-ccd44b94bd5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7a217d-w4jzg" Nov 01 20:45:17.060404 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:17.060310 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7a217d-w4jzg" Nov 01 20:45:17.152121 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:17.152078 4484 generic.go:356] "Generic (PLEG): container finished" podID="db922223-a186-42a9-9a5b-c44258bbb793" containerID="083618f1c29254275a201f47de8980f6b475481661b8a5a693e42a44bace91c9" exitCode=60 Nov 01 20:45:17.152317 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:17.152147 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892765-r4rtj" event={"ID":"db922223-a186-42a9-9a5b-c44258bbb793","Type":"ContainerDied","Data":"083618f1c29254275a201f47de8980f6b475481661b8a5a693e42a44bace91c9"} Nov 01 20:45:17.152317 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:17.152236 4484 scope.go:122] "RemoveContainer" containerID="25f09ad4c34f658ecc6fbca4c3e8a245dd13712874875d385cb5e2bc18fd0917" Nov 01 20:45:17.152941 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:17.152919 4484 scope.go:122] "RemoveContainer" containerID="083618f1c29254275a201f47de8980f6b475481661b8a5a693e42a44bace91c9" Nov 01 20:45:17.153481 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:45:17.153453 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-cd248f-29892765-r4rtj_odh-ai-gateway-infra(db922223-a186-42a9-9a5b-c44258bbb793)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892765-r4rtj" podUID="db922223-a186-42a9-9a5b-c44258bbb793" Nov 01 20:45:17.251249 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:17.251216 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892765-r4rtj"] Nov 01 20:45:17.365223 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:17.365144 4484 scope.go:122] "RemoveContainer" containerID="4bad70676e67ccdc8b5abee8ba0b94562af0316274d5e379c80aac8fa53e1e9b" Nov 01 20:45:17.434042 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:17.434018 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a7a217d-w4jzg"] Nov 01 20:45:17.435077 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:45:17.435051 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddee25581_08e6_472d_b589_ccd44b94bd5f.slice/crio-6a13b4fb90c657e2a7c5228d7f283d59774b94eee51ea3d0fc8b28f8b2197c13 WatchSource:0}: Error finding container 6a13b4fb90c657e2a7c5228d7f283d59774b94eee51ea3d0fc8b28f8b2197c13: Status 404 returned error can't find the container with id 6a13b4fb90c657e2a7c5228d7f283d59774b94eee51ea3d0fc8b28f8b2197c13 Nov 01 20:45:18.161390 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:18.161359 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7a217d-w4jzg" event={"ID":"dee25581-08e6-472d-b589-ccd44b94bd5f","Type":"ContainerStarted","Data":"5889a5b844671e651c59fe07269fba75e3198fe99b1a33fa8e629cb0f39c190a"} Nov 01 20:45:18.161805 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:18.161425 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7a217d-w4jzg" event={"ID":"dee25581-08e6-472d-b589-ccd44b94bd5f","Type":"ContainerStarted","Data":"6a13b4fb90c657e2a7c5228d7f283d59774b94eee51ea3d0fc8b28f8b2197c13"} Nov 01 20:45:18.163358 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:18.163330 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892765-jkvj9" event={"ID":"b4964cf9-925d-4258-b8fe-2cc516b79487","Type":"ContainerStarted","Data":"7e1b94d2b6c6891f3afb241734cd9bb50c64029905524d97d06488a7cbd3ff4b"} Nov 01 20:45:18.365191 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:18.365159 4484 scope.go:122] "RemoveContainer" containerID="1b98ae9085a508a3427ad2ad9896b22f00d379ab9fa6a19a89f99af2581524a3" Nov 01 20:45:18.365512 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:18.365494 4484 scope.go:122] "RemoveContainer" containerID="a272bdefe7c2e539e321a9f3641700431ad83c5c33d55a2e8fbc4c4c41834dfc" Nov 01 20:45:18.365587 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:18.365578 4484 scope.go:122] "RemoveContainer" containerID="5ea1d56f393bd22c3a866ab36141cf3a21d1cdf539dffdd32219566ce8e20d77" Nov 01 20:45:18.399457 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:18.399350 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7a217d-w4jzg" podStartSLOduration=2.399329215 podStartE2EDuration="2.399329215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:45:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:45:18.394884064 +0000 UTC m=+869.172524197" watchObservedRunningTime="2026-11-01 20:45:18.399329215 +0000 UTC m=+869.176969351" Nov 01 20:45:18.423989 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:18.423861 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892765-r4rtj" Nov 01 20:45:18.538702 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:18.538668 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db922223-a186-42a9-9a5b-c44258bbb793-kube-api-access-kvfpb\" (UniqueName: \"kubernetes.io/projected/db922223-a186-42a9-9a5b-c44258bbb793-kube-api-access-kvfpb\") pod \"db922223-a186-42a9-9a5b-c44258bbb793\" (UID: \"db922223-a186-42a9-9a5b-c44258bbb793\") " Nov 01 20:45:18.538862 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:18.538805 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db922223-a186-42a9-9a5b-c44258bbb793-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db922223-a186-42a9-9a5b-c44258bbb793-maas-api-service-ca\") pod \"db922223-a186-42a9-9a5b-c44258bbb793\" (UID: \"db922223-a186-42a9-9a5b-c44258bbb793\") " Nov 01 20:45:18.539212 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:18.539183 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db922223-a186-42a9-9a5b-c44258bbb793-maas-api-service-ca" pod "db922223-a186-42a9-9a5b-c44258bbb793" (UID: "db922223-a186-42a9-9a5b-c44258bbb793"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:45:18.540781 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:18.540759 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db922223-a186-42a9-9a5b-c44258bbb793-kube-api-access-kvfpb" pod "db922223-a186-42a9-9a5b-c44258bbb793" (UID: "db922223-a186-42a9-9a5b-c44258bbb793"). InnerVolumeSpecName "kube-api-access-kvfpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:45:18.642079 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:18.640427 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvfpb\" (UniqueName: \"kubernetes.io/projected/db922223-a186-42a9-9a5b-c44258bbb793-kube-api-access-kvfpb\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:18.642079 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:18.640458 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db922223-a186-42a9-9a5b-c44258bbb793-maas-api-service-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:19.171011 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:19.170963 4484 generic.go:356] "Generic (PLEG): container finished" podID="dee25581-08e6-472d-b589-ccd44b94bd5f" containerID="5889a5b844671e651c59fe07269fba75e3198fe99b1a33fa8e629cb0f39c190a" exitCode=0 Nov 01 20:45:19.171482 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:19.171048 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7a217d-w4jzg" event={"ID":"dee25581-08e6-472d-b589-ccd44b94bd5f","Type":"ContainerDied","Data":"5889a5b844671e651c59fe07269fba75e3198fe99b1a33fa8e629cb0f39c190a"} Nov 01 20:45:19.172932 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:19.172909 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892765-4z8cw" event={"ID":"f16b797d-6930-414c-9957-bff940a16153","Type":"ContainerStarted","Data":"c263852a79d189863924bd7aabf764085963a5807bc36ebc99b05b2a92f043dd"} Nov 01 20:45:19.174694 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:19.174673 4484 generic.go:356] "Generic (PLEG): container finished" podID="b4964cf9-925d-4258-b8fe-2cc516b79487" containerID="7e1b94d2b6c6891f3afb241734cd9bb50c64029905524d97d06488a7cbd3ff4b" exitCode=60 Nov 01 20:45:19.174796 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:19.174745 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892765-jkvj9" event={"ID":"b4964cf9-925d-4258-b8fe-2cc516b79487","Type":"ContainerDied","Data":"7e1b94d2b6c6891f3afb241734cd9bb50c64029905524d97d06488a7cbd3ff4b"} Nov 01 20:45:19.174851 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:19.174798 4484 scope.go:122] "RemoveContainer" containerID="4bad70676e67ccdc8b5abee8ba0b94562af0316274d5e379c80aac8fa53e1e9b" Nov 01 20:45:19.175144 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:19.175128 4484 scope.go:122] "RemoveContainer" containerID="7e1b94d2b6c6891f3afb241734cd9bb50c64029905524d97d06488a7cbd3ff4b" Nov 01 20:45:19.175426 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:45:19.175396 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-208f2b-29892765-jkvj9_odh-ai-gateway-infra(b4964cf9-925d-4258-b8fe-2cc516b79487)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892765-jkvj9" podUID="b4964cf9-925d-4258-b8fe-2cc516b79487" Nov 01 20:45:19.176662 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:19.176642 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892765-wkqgl" event={"ID":"3d22905a-1b38-4c16-81fc-66f1cefd6c7e","Type":"ContainerStarted","Data":"8152c3e41237a959269d6b0673f5053f6214a3f800b70143f2d41c00fa6fa7a6"} Nov 01 20:45:19.178418 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:19.178398 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-xp8qs" event={"ID":"143424a0-981e-428f-ae6f-9c33ea0d5836","Type":"ContainerStarted","Data":"49cbede220db73253581448f07e3f4e15e24bcdcdd441aebcf37980d4489a95f"} Nov 01 20:45:19.179687 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:19.179668 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892765-r4rtj" Nov 01 20:45:19.179687 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:19.179684 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892765-r4rtj" event={"ID":"db922223-a186-42a9-9a5b-c44258bbb793","Type":"ContainerDied","Data":"50ee8bc15a1a14d42867295189beed8c39f0da2fb61da62fd3c6ddba314e6d92"} Nov 01 20:45:19.187179 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:19.187146 4484 scope.go:122] "RemoveContainer" containerID="083618f1c29254275a201f47de8980f6b475481661b8a5a693e42a44bace91c9" Nov 01 20:45:19.524231 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:19.524198 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892765-jkvj9"] Nov 01 20:45:19.856675 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:19.856580 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892765-r4rtj"] Nov 01 20:45:19.860108 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:19.860078 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892765-r4rtj"] Nov 01 20:45:20.189228 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:20.189145 4484 generic.go:356] "Generic (PLEG): container finished" podID="143424a0-981e-428f-ae6f-9c33ea0d5836" containerID="49cbede220db73253581448f07e3f4e15e24bcdcdd441aebcf37980d4489a95f" exitCode=60 Nov 01 20:45:20.189690 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:20.189216 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-xp8qs" event={"ID":"143424a0-981e-428f-ae6f-9c33ea0d5836","Type":"ContainerDied","Data":"49cbede220db73253581448f07e3f4e15e24bcdcdd441aebcf37980d4489a95f"} Nov 01 20:45:20.189690 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:20.189316 4484 scope.go:122] "RemoveContainer" containerID="a272bdefe7c2e539e321a9f3641700431ad83c5c33d55a2e8fbc4c4c41834dfc" Nov 01 20:45:20.189802 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:20.189756 4484 scope.go:122] "RemoveContainer" containerID="49cbede220db73253581448f07e3f4e15e24bcdcdd441aebcf37980d4489a95f" Nov 01 20:45:20.190092 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:45:20.190066 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29892765-xp8qs_odh-ai-gateway-infra(143424a0-981e-428f-ae6f-9c33ea0d5836)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-xp8qs" podUID="143424a0-981e-428f-ae6f-9c33ea0d5836" Nov 01 20:45:20.192375 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:20.192352 4484 generic.go:356] "Generic (PLEG): container finished" podID="f16b797d-6930-414c-9957-bff940a16153" containerID="c263852a79d189863924bd7aabf764085963a5807bc36ebc99b05b2a92f043dd" exitCode=60 Nov 01 20:45:20.192470 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:20.192423 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892765-4z8cw" event={"ID":"f16b797d-6930-414c-9957-bff940a16153","Type":"ContainerDied","Data":"c263852a79d189863924bd7aabf764085963a5807bc36ebc99b05b2a92f043dd"} Nov 01 20:45:20.192752 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:20.192733 4484 scope.go:122] "RemoveContainer" containerID="c263852a79d189863924bd7aabf764085963a5807bc36ebc99b05b2a92f043dd" Nov 01 20:45:20.193024 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:45:20.192981 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892765-4z8cw_odh-ai-gateway-infra(f16b797d-6930-414c-9957-bff940a16153)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892765-4z8cw" podUID="f16b797d-6930-414c-9957-bff940a16153" Nov 01 20:45:20.196789 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:20.196765 4484 generic.go:356] "Generic (PLEG): container finished" podID="3d22905a-1b38-4c16-81fc-66f1cefd6c7e" containerID="8152c3e41237a959269d6b0673f5053f6214a3f800b70143f2d41c00fa6fa7a6" exitCode=60 Nov 01 20:45:20.196883 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:20.196845 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892765-wkqgl" event={"ID":"3d22905a-1b38-4c16-81fc-66f1cefd6c7e","Type":"ContainerDied","Data":"8152c3e41237a959269d6b0673f5053f6214a3f800b70143f2d41c00fa6fa7a6"} Nov 01 20:45:20.197342 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:20.197326 4484 scope.go:122] "RemoveContainer" containerID="8152c3e41237a959269d6b0673f5053f6214a3f800b70143f2d41c00fa6fa7a6" Nov 01 20:45:20.197633 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:45:20.197612 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-db0157-29892765-wkqgl_odh-ai-gateway-infra(3d22905a-1b38-4c16-81fc-66f1cefd6c7e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892765-wkqgl" podUID="3d22905a-1b38-4c16-81fc-66f1cefd6c7e" Nov 01 20:45:20.208715 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:20.208691 4484 scope.go:122] "RemoveContainer" containerID="5ea1d56f393bd22c3a866ab36141cf3a21d1cdf539dffdd32219566ce8e20d77" Nov 01 20:45:20.225183 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:20.225154 4484 scope.go:122] "RemoveContainer" containerID="1b98ae9085a508a3427ad2ad9896b22f00d379ab9fa6a19a89f99af2581524a3" Nov 01 20:45:20.375714 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:20.375687 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892765-jkvj9" Nov 01 20:45:20.427487 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:20.427461 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7a217d-w4jzg" Nov 01 20:45:20.463709 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:20.463615 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4964cf9-925d-4258-b8fe-2cc516b79487-kube-api-access-p8lhr\" (UniqueName: \"kubernetes.io/projected/b4964cf9-925d-4258-b8fe-2cc516b79487-kube-api-access-p8lhr\") pod \"b4964cf9-925d-4258-b8fe-2cc516b79487\" (UID: \"b4964cf9-925d-4258-b8fe-2cc516b79487\") " Nov 01 20:45:20.463885 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:20.463735 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4964cf9-925d-4258-b8fe-2cc516b79487-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4964cf9-925d-4258-b8fe-2cc516b79487-maas-api-service-ca\") pod \"b4964cf9-925d-4258-b8fe-2cc516b79487\" (UID: \"b4964cf9-925d-4258-b8fe-2cc516b79487\") " Nov 01 20:45:20.464118 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:20.464096 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4964cf9-925d-4258-b8fe-2cc516b79487-maas-api-service-ca" pod "b4964cf9-925d-4258-b8fe-2cc516b79487" (UID: "b4964cf9-925d-4258-b8fe-2cc516b79487"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:45:20.464261 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:20.464244 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4964cf9-925d-4258-b8fe-2cc516b79487-maas-api-service-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:20.465929 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:20.465905 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4964cf9-925d-4258-b8fe-2cc516b79487-kube-api-access-p8lhr" pod "b4964cf9-925d-4258-b8fe-2cc516b79487" (UID: "b4964cf9-925d-4258-b8fe-2cc516b79487"). InnerVolumeSpecName "kube-api-access-p8lhr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:45:20.525364 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:20.525329 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892765-4z8cw"] Nov 01 20:45:20.565216 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:20.565178 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dee25581-08e6-472d-b589-ccd44b94bd5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dee25581-08e6-472d-b589-ccd44b94bd5f-maas-api-openshift-ca\") pod \"dee25581-08e6-472d-b589-ccd44b94bd5f\" (UID: \"dee25581-08e6-472d-b589-ccd44b94bd5f\") " Nov 01 20:45:20.565394 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:20.565351 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dee25581-08e6-472d-b589-ccd44b94bd5f-kube-api-access-fr8m7\" (UniqueName: \"kubernetes.io/projected/dee25581-08e6-472d-b589-ccd44b94bd5f-kube-api-access-fr8m7\") pod \"dee25581-08e6-472d-b589-ccd44b94bd5f\" (UID: \"dee25581-08e6-472d-b589-ccd44b94bd5f\") " Nov 01 20:45:20.565479 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:20.565409 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dee25581-08e6-472d-b589-ccd44b94bd5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dee25581-08e6-472d-b589-ccd44b94bd5f-maas-api-xks-ca\") pod \"dee25581-08e6-472d-b589-ccd44b94bd5f\" (UID: \"dee25581-08e6-472d-b589-ccd44b94bd5f\") " Nov 01 20:45:20.565590 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:20.565567 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dee25581-08e6-472d-b589-ccd44b94bd5f-maas-api-openshift-ca" pod "dee25581-08e6-472d-b589-ccd44b94bd5f" (UID: "dee25581-08e6-472d-b589-ccd44b94bd5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:45:20.565810 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:20.565796 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dee25581-08e6-472d-b589-ccd44b94bd5f-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:20.565878 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:20.565813 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8lhr\" (UniqueName: \"kubernetes.io/projected/b4964cf9-925d-4258-b8fe-2cc516b79487-kube-api-access-p8lhr\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:20.567435 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:20.567410 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dee25581-08e6-472d-b589-ccd44b94bd5f-maas-api-xks-ca" pod "dee25581-08e6-472d-b589-ccd44b94bd5f" (UID: "dee25581-08e6-472d-b589-ccd44b94bd5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:45:20.567586 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:20.567509 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dee25581-08e6-472d-b589-ccd44b94bd5f-kube-api-access-fr8m7" pod "dee25581-08e6-472d-b589-ccd44b94bd5f" (UID: "dee25581-08e6-472d-b589-ccd44b94bd5f"). InnerVolumeSpecName "kube-api-access-fr8m7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:45:20.641488 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:20.641450 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892765-wkqgl"] Nov 01 20:45:20.666447 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:20.666412 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr8m7\" (UniqueName: \"kubernetes.io/projected/dee25581-08e6-472d-b589-ccd44b94bd5f-kube-api-access-fr8m7\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:20.666447 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:20.666441 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dee25581-08e6-472d-b589-ccd44b94bd5f-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:20.754423 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:20.754388 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892765-xp8qs"] Nov 01 20:45:21.041617 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.041526 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:21.204803 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.204769 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7a217d-w4jzg" Nov 01 20:45:21.205314 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.204769 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7a217d-w4jzg" event={"ID":"dee25581-08e6-472d-b589-ccd44b94bd5f","Type":"ContainerDied","Data":"6a13b4fb90c657e2a7c5228d7f283d59774b94eee51ea3d0fc8b28f8b2197c13"} Nov 01 20:45:21.205314 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.204906 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a13b4fb90c657e2a7c5228d7f283d59774b94eee51ea3d0fc8b28f8b2197c13" Nov 01 20:45:21.208263 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.208236 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892765-jkvj9" Nov 01 20:45:21.208380 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.208262 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892765-jkvj9" event={"ID":"b4964cf9-925d-4258-b8fe-2cc516b79487","Type":"ContainerDied","Data":"38770816bacdc75002d1a6d490bde25bfb6d819ff690cc3a483981cdb380de20"} Nov 01 20:45:21.208380 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.208314 4484 scope.go:122] "RemoveContainer" containerID="7e1b94d2b6c6891f3afb241734cd9bb50c64029905524d97d06488a7cbd3ff4b" Nov 01 20:45:21.307950 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.307875 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2e93c2f-nnclz"] Nov 01 20:45:21.309284 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.309257 4484 cpu_manager.go:414] "containerMap: removing container" podUID="b4964cf9-925d-4258-b8fe-2cc516b79487" containerName="cleanup" Nov 01 20:45:21.309284 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.309281 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4964cf9-925d-4258-b8fe-2cc516b79487" containerName="cleanup" Nov 01 20:45:21.309456 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.309308 4484 cpu_manager.go:414] "containerMap: removing container" podUID="dee25581-08e6-472d-b589-ccd44b94bd5f" containerName="revoke-keys" Nov 01 20:45:21.309456 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.309315 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dee25581-08e6-472d-b589-ccd44b94bd5f" containerName="revoke-keys" Nov 01 20:45:21.309456 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.309342 4484 cpu_manager.go:414] "containerMap: removing container" podUID="db922223-a186-42a9-9a5b-c44258bbb793" containerName="cleanup" Nov 01 20:45:21.309456 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.309349 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db922223-a186-42a9-9a5b-c44258bbb793" containerName="cleanup" Nov 01 20:45:21.309456 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.309366 4484 cpu_manager.go:414] "containerMap: removing container" podUID="db922223-a186-42a9-9a5b-c44258bbb793" containerName="cleanup" Nov 01 20:45:21.309456 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.309374 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db922223-a186-42a9-9a5b-c44258bbb793" containerName="cleanup" Nov 01 20:45:21.309456 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.309400 4484 cpu_manager.go:414] "containerMap: removing container" podUID="b4964cf9-925d-4258-b8fe-2cc516b79487" containerName="cleanup" Nov 01 20:45:21.309456 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.309407 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4964cf9-925d-4258-b8fe-2cc516b79487" containerName="cleanup" Nov 01 20:45:21.309835 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.309555 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="db922223-a186-42a9-9a5b-c44258bbb793" containerName="cleanup" Nov 01 20:45:21.309835 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.309571 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="db922223-a186-42a9-9a5b-c44258bbb793" containerName="cleanup" Nov 01 20:45:21.309835 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.309581 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4964cf9-925d-4258-b8fe-2cc516b79487" containerName="cleanup" Nov 01 20:45:21.309835 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.309591 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4964cf9-925d-4258-b8fe-2cc516b79487" containerName="cleanup" Nov 01 20:45:21.309835 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.309604 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4964cf9-925d-4258-b8fe-2cc516b79487" containerName="cleanup" Nov 01 20:45:21.309835 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.309616 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="dee25581-08e6-472d-b589-ccd44b94bd5f" containerName="revoke-keys" Nov 01 20:45:21.315127 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.315097 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e93c2f-nnclz" Nov 01 20:45:21.372088 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.371672 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:45:21.372088 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.371739 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2e93c2f-nnclz"] Nov 01 20:45:21.381241 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.381213 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pwtl\" (UniqueName: \"kubernetes.io/projected/d87c9034-5ef9-4323-be7b-040921d4bf35-kube-api-access-5pwtl\") pod \"maas-api-revoke-sub-c2e93c2f-nnclz\" (UID: \"d87c9034-5ef9-4323-be7b-040921d4bf35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e93c2f-nnclz" Nov 01 20:45:21.381348 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.381309 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d87c9034-5ef9-4323-be7b-040921d4bf35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2e93c2f-nnclz\" (UID: \"d87c9034-5ef9-4323-be7b-040921d4bf35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e93c2f-nnclz" Nov 01 20:45:21.381609 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.381574 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d87c9034-5ef9-4323-be7b-040921d4bf35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2e93c2f-nnclz\" (UID: \"d87c9034-5ef9-4323-be7b-040921d4bf35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e93c2f-nnclz" Nov 01 20:45:21.383641 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.383619 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db922223-a186-42a9-9a5b-c44258bbb793" path="/var/lib/kubelet/pods/db922223-a186-42a9-9a5b-c44258bbb793/volumes" Nov 01 20:45:21.426952 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.426919 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892765-4z8cw" Nov 01 20:45:21.429811 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.429790 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-xp8qs" Nov 01 20:45:21.433732 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.433711 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892765-wkqgl" Nov 01 20:45:21.482877 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.482844 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d87c9034-5ef9-4323-be7b-040921d4bf35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2e93c2f-nnclz\" (UID: \"d87c9034-5ef9-4323-be7b-040921d4bf35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e93c2f-nnclz" Nov 01 20:45:21.483115 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.482892 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pwtl\" (UniqueName: \"kubernetes.io/projected/d87c9034-5ef9-4323-be7b-040921d4bf35-kube-api-access-5pwtl\") pod \"maas-api-revoke-sub-c2e93c2f-nnclz\" (UID: \"d87c9034-5ef9-4323-be7b-040921d4bf35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e93c2f-nnclz" Nov 01 20:45:21.483115 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.482937 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d87c9034-5ef9-4323-be7b-040921d4bf35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2e93c2f-nnclz\" (UID: \"d87c9034-5ef9-4323-be7b-040921d4bf35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e93c2f-nnclz" Nov 01 20:45:21.483627 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.483603 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d87c9034-5ef9-4323-be7b-040921d4bf35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2e93c2f-nnclz\" (UID: \"d87c9034-5ef9-4323-be7b-040921d4bf35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e93c2f-nnclz" Nov 01 20:45:21.485221 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.485201 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d87c9034-5ef9-4323-be7b-040921d4bf35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2e93c2f-nnclz\" (UID: \"d87c9034-5ef9-4323-be7b-040921d4bf35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e93c2f-nnclz" Nov 01 20:45:21.540489 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.540461 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pwtl\" (UniqueName: \"kubernetes.io/projected/d87c9034-5ef9-4323-be7b-040921d4bf35-kube-api-access-5pwtl\") pod \"maas-api-revoke-sub-c2e93c2f-nnclz\" (UID: \"d87c9034-5ef9-4323-be7b-040921d4bf35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e93c2f-nnclz" Nov 01 20:45:21.583713 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.583621 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f16b797d-6930-414c-9957-bff940a16153-kube-api-access-m4h2k\" (UniqueName: \"kubernetes.io/projected/f16b797d-6930-414c-9957-bff940a16153-kube-api-access-m4h2k\") pod \"f16b797d-6930-414c-9957-bff940a16153\" (UID: \"f16b797d-6930-414c-9957-bff940a16153\") " Nov 01 20:45:21.583713 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.583687 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/143424a0-981e-428f-ae6f-9c33ea0d5836-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/143424a0-981e-428f-ae6f-9c33ea0d5836-maas-api-service-ca\") pod \"143424a0-981e-428f-ae6f-9c33ea0d5836\" (UID: \"143424a0-981e-428f-ae6f-9c33ea0d5836\") " Nov 01 20:45:21.583873 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.583849 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d22905a-1b38-4c16-81fc-66f1cefd6c7e-kube-api-access-d8thx\" (UniqueName: \"kubernetes.io/projected/3d22905a-1b38-4c16-81fc-66f1cefd6c7e-kube-api-access-d8thx\") pod \"3d22905a-1b38-4c16-81fc-66f1cefd6c7e\" (UID: \"3d22905a-1b38-4c16-81fc-66f1cefd6c7e\") " Nov 01 20:45:21.583935 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.583893 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d22905a-1b38-4c16-81fc-66f1cefd6c7e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d22905a-1b38-4c16-81fc-66f1cefd6c7e-maas-api-service-ca\") pod \"3d22905a-1b38-4c16-81fc-66f1cefd6c7e\" (UID: \"3d22905a-1b38-4c16-81fc-66f1cefd6c7e\") " Nov 01 20:45:21.583935 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.583920 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f16b797d-6930-414c-9957-bff940a16153-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f16b797d-6930-414c-9957-bff940a16153-maas-api-service-ca\") pod \"f16b797d-6930-414c-9957-bff940a16153\" (UID: \"f16b797d-6930-414c-9957-bff940a16153\") " Nov 01 20:45:21.584070 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.583949 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/143424a0-981e-428f-ae6f-9c33ea0d5836-kube-api-access-9fnmv\" (UniqueName: \"kubernetes.io/projected/143424a0-981e-428f-ae6f-9c33ea0d5836-kube-api-access-9fnmv\") pod \"143424a0-981e-428f-ae6f-9c33ea0d5836\" (UID: \"143424a0-981e-428f-ae6f-9c33ea0d5836\") " Nov 01 20:45:21.584330 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.584299 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d22905a-1b38-4c16-81fc-66f1cefd6c7e-maas-api-service-ca" pod "3d22905a-1b38-4c16-81fc-66f1cefd6c7e" (UID: "3d22905a-1b38-4c16-81fc-66f1cefd6c7e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:45:21.584407 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.584311 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f16b797d-6930-414c-9957-bff940a16153-maas-api-service-ca" pod "f16b797d-6930-414c-9957-bff940a16153" (UID: "f16b797d-6930-414c-9957-bff940a16153"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:45:21.584466 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.584439 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/143424a0-981e-428f-ae6f-9c33ea0d5836-maas-api-service-ca" pod "143424a0-981e-428f-ae6f-9c33ea0d5836" (UID: "143424a0-981e-428f-ae6f-9c33ea0d5836"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:45:21.584555 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.584540 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/143424a0-981e-428f-ae6f-9c33ea0d5836-maas-api-service-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:21.584638 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.584557 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d22905a-1b38-4c16-81fc-66f1cefd6c7e-maas-api-service-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:21.584638 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.584570 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f16b797d-6930-414c-9957-bff940a16153-maas-api-service-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:21.585719 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.585693 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f16b797d-6930-414c-9957-bff940a16153-kube-api-access-m4h2k" pod "f16b797d-6930-414c-9957-bff940a16153" (UID: "f16b797d-6930-414c-9957-bff940a16153"). InnerVolumeSpecName "kube-api-access-m4h2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:45:21.586162 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.586135 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d22905a-1b38-4c16-81fc-66f1cefd6c7e-kube-api-access-d8thx" pod "3d22905a-1b38-4c16-81fc-66f1cefd6c7e" (UID: "3d22905a-1b38-4c16-81fc-66f1cefd6c7e"). InnerVolumeSpecName "kube-api-access-d8thx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:45:21.586249 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.586168 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/143424a0-981e-428f-ae6f-9c33ea0d5836-kube-api-access-9fnmv" pod "143424a0-981e-428f-ae6f-9c33ea0d5836" (UID: "143424a0-981e-428f-ae6f-9c33ea0d5836"). InnerVolumeSpecName "kube-api-access-9fnmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:45:21.634914 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.634879 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e93c2f-nnclz" Nov 01 20:45:21.686199 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.686138 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8thx\" (UniqueName: \"kubernetes.io/projected/3d22905a-1b38-4c16-81fc-66f1cefd6c7e-kube-api-access-d8thx\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:21.686199 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.686167 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fnmv\" (UniqueName: \"kubernetes.io/projected/143424a0-981e-428f-ae6f-9c33ea0d5836-kube-api-access-9fnmv\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:21.686199 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.686177 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4h2k\" (UniqueName: \"kubernetes.io/projected/f16b797d-6930-414c-9957-bff940a16153-kube-api-access-m4h2k\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:21.717806 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.717770 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892765-jkvj9"] Nov 01 20:45:21.721326 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:21.721299 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-208f2b-29892765-jkvj9"] Nov 01 20:45:22.010393 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:22.010367 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2e93c2f-nnclz"] Nov 01 20:45:22.011343 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:45:22.011315 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd87c9034_5ef9_4323_be7b_040921d4bf35.slice/crio-ce826f5f4f75d3a1a61e02ef6b351ef1c9228a752c86b5db384301890f85a6c9 WatchSource:0}: Error finding container ce826f5f4f75d3a1a61e02ef6b351ef1c9228a752c86b5db384301890f85a6c9: Status 404 returned error can't find the container with id ce826f5f4f75d3a1a61e02ef6b351ef1c9228a752c86b5db384301890f85a6c9 Nov 01 20:45:22.221087 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:22.221056 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892765-4z8cw" Nov 01 20:45:22.221473 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:22.221049 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892765-4z8cw" event={"ID":"f16b797d-6930-414c-9957-bff940a16153","Type":"ContainerDied","Data":"0197e28c3a5a4e419429103ddcb63853dac0825da1d98875e02075a0161b4cf1"} Nov 01 20:45:22.221473 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:22.221290 4484 scope.go:122] "RemoveContainer" containerID="c263852a79d189863924bd7aabf764085963a5807bc36ebc99b05b2a92f043dd" Nov 01 20:45:22.223535 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:22.223516 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892765-wkqgl" Nov 01 20:45:22.223535 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:22.223524 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892765-wkqgl" event={"ID":"3d22905a-1b38-4c16-81fc-66f1cefd6c7e","Type":"ContainerDied","Data":"2c92e7d808fd27b6f664cb770084f9aea1d245da1eeeed1c97470cb2f958d169"} Nov 01 20:45:22.224881 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:22.224853 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e93c2f-nnclz" event={"ID":"d87c9034-5ef9-4323-be7b-040921d4bf35","Type":"ContainerStarted","Data":"ce826f5f4f75d3a1a61e02ef6b351ef1c9228a752c86b5db384301890f85a6c9"} Nov 01 20:45:22.226413 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:22.226393 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-xp8qs" Nov 01 20:45:22.226514 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:22.226414 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-xp8qs" event={"ID":"143424a0-981e-428f-ae6f-9c33ea0d5836","Type":"ContainerDied","Data":"8c506cec3ae8048d661385d42f9bbcb5f092b3fba65f0e6423899e1f5b9c0f24"} Nov 01 20:45:22.235956 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:22.235297 4484 scope.go:122] "RemoveContainer" containerID="8152c3e41237a959269d6b0673f5053f6214a3f800b70143f2d41c00fa6fa7a6" Nov 01 20:45:22.245624 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:22.245605 4484 scope.go:122] "RemoveContainer" containerID="49cbede220db73253581448f07e3f4e15e24bcdcdd441aebcf37980d4489a95f" Nov 01 20:45:22.389523 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:22.389491 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892765-4z8cw"] Nov 01 20:45:22.392175 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:22.392148 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892765-4z8cw"] Nov 01 20:45:22.502618 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:22.502579 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892765-wkqgl"] Nov 01 20:45:22.506417 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:22.506385 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892765-wkqgl"] Nov 01 20:45:22.615539 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:22.615452 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892765-xp8qs"] Nov 01 20:45:22.617953 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:22.617935 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892765-xp8qs"] Nov 01 20:45:23.037411 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.037371 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms"] Nov 01 20:45:23.038264 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.038242 4484 cpu_manager.go:414] "containerMap: removing container" podUID="b4964cf9-925d-4258-b8fe-2cc516b79487" containerName="cleanup" Nov 01 20:45:23.038373 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.038266 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4964cf9-925d-4258-b8fe-2cc516b79487" containerName="cleanup" Nov 01 20:45:23.038373 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.038287 4484 cpu_manager.go:414] "containerMap: removing container" podUID="143424a0-981e-428f-ae6f-9c33ea0d5836" containerName="cleanup" Nov 01 20:45:23.038373 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.038296 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="143424a0-981e-428f-ae6f-9c33ea0d5836" containerName="cleanup" Nov 01 20:45:23.038373 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.038310 4484 cpu_manager.go:414] "containerMap: removing container" podUID="143424a0-981e-428f-ae6f-9c33ea0d5836" containerName="cleanup" Nov 01 20:45:23.038373 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.038319 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="143424a0-981e-428f-ae6f-9c33ea0d5836" containerName="cleanup" Nov 01 20:45:23.038373 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.038334 4484 cpu_manager.go:414] "containerMap: removing container" podUID="f16b797d-6930-414c-9957-bff940a16153" containerName="cleanup" Nov 01 20:45:23.038373 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.038341 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f16b797d-6930-414c-9957-bff940a16153" containerName="cleanup" Nov 01 20:45:23.038373 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.038356 4484 cpu_manager.go:414] "containerMap: removing container" podUID="3d22905a-1b38-4c16-81fc-66f1cefd6c7e" containerName="cleanup" Nov 01 20:45:23.038373 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.038363 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d22905a-1b38-4c16-81fc-66f1cefd6c7e" containerName="cleanup" Nov 01 20:45:23.038794 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.038384 4484 cpu_manager.go:414] "containerMap: removing container" podUID="3d22905a-1b38-4c16-81fc-66f1cefd6c7e" containerName="cleanup" Nov 01 20:45:23.038794 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.038389 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d22905a-1b38-4c16-81fc-66f1cefd6c7e" containerName="cleanup" Nov 01 20:45:23.038794 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.038400 4484 cpu_manager.go:414] "containerMap: removing container" podUID="f16b797d-6930-414c-9957-bff940a16153" containerName="cleanup" Nov 01 20:45:23.038794 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.038404 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f16b797d-6930-414c-9957-bff940a16153" containerName="cleanup" Nov 01 20:45:23.038794 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.038422 4484 cpu_manager.go:414] "containerMap: removing container" podUID="db922223-a186-42a9-9a5b-c44258bbb793" containerName="cleanup" Nov 01 20:45:23.038794 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.038427 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db922223-a186-42a9-9a5b-c44258bbb793" containerName="cleanup" Nov 01 20:45:23.038794 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.038539 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="f16b797d-6930-414c-9957-bff940a16153" containerName="cleanup" Nov 01 20:45:23.038794 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.038550 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="143424a0-981e-428f-ae6f-9c33ea0d5836" containerName="cleanup" Nov 01 20:45:23.038794 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.038556 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="143424a0-981e-428f-ae6f-9c33ea0d5836" containerName="cleanup" Nov 01 20:45:23.038794 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.038562 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="f16b797d-6930-414c-9957-bff940a16153" containerName="cleanup" Nov 01 20:45:23.038794 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.038568 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d22905a-1b38-4c16-81fc-66f1cefd6c7e" containerName="cleanup" Nov 01 20:45:23.038794 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.038575 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d22905a-1b38-4c16-81fc-66f1cefd6c7e" containerName="cleanup" Nov 01 20:45:23.038794 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.038583 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="db922223-a186-42a9-9a5b-c44258bbb793" containerName="cleanup" Nov 01 20:45:23.041956 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.041938 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms" Nov 01 20:45:23.097979 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.097936 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-c9ae21-openshift-default-dockercfg-k8hz9\"" Nov 01 20:45:23.101521 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.101478 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms"] Nov 01 20:45:23.201676 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.201600 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bc9b444-1915-4ba3-a7f7-789549742e85-istio-data\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms\" (UID: \"6bc9b444-1915-4ba3-a7f7-789549742e85\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms" Nov 01 20:45:23.201676 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.201645 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc9b444-1915-4ba3-a7f7-789549742e85-workload-socket\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms\" (UID: \"6bc9b444-1915-4ba3-a7f7-789549742e85\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms" Nov 01 20:45:23.201859 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.201715 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bc9b444-1915-4ba3-a7f7-789549742e85-istiod-ca-cert\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms\" (UID: \"6bc9b444-1915-4ba3-a7f7-789549742e85\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms" Nov 01 20:45:23.201859 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.201750 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z4rj\" (UniqueName: \"kubernetes.io/projected/6bc9b444-1915-4ba3-a7f7-789549742e85-kube-api-access-4z4rj\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms\" (UID: \"6bc9b444-1915-4ba3-a7f7-789549742e85\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms" Nov 01 20:45:23.201859 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.201799 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc9b444-1915-4ba3-a7f7-789549742e85-credential-socket\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms\" (UID: \"6bc9b444-1915-4ba3-a7f7-789549742e85\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms" Nov 01 20:45:23.201982 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.201883 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bc9b444-1915-4ba3-a7f7-789549742e85-istio-token\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms\" (UID: \"6bc9b444-1915-4ba3-a7f7-789549742e85\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms" Nov 01 20:45:23.201982 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.201914 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bc9b444-1915-4ba3-a7f7-789549742e85-istio-envoy\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms\" (UID: \"6bc9b444-1915-4ba3-a7f7-789549742e85\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms" Nov 01 20:45:23.201982 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.201937 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bc9b444-1915-4ba3-a7f7-789549742e85-workload-certs\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms\" (UID: \"6bc9b444-1915-4ba3-a7f7-789549742e85\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms" Nov 01 20:45:23.202122 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.202067 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bc9b444-1915-4ba3-a7f7-789549742e85-istio-podinfo\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms\" (UID: \"6bc9b444-1915-4ba3-a7f7-789549742e85\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms" Nov 01 20:45:23.236883 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.236846 4484 generic.go:356] "Generic (PLEG): container finished" podID="d87c9034-5ef9-4323-be7b-040921d4bf35" containerID="312bf148a6b8e8c89912c215d67b686484ce8f8b16a69e7a8a96fd5430685806" exitCode=0 Nov 01 20:45:23.237337 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.236921 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e93c2f-nnclz" event={"ID":"d87c9034-5ef9-4323-be7b-040921d4bf35","Type":"ContainerDied","Data":"312bf148a6b8e8c89912c215d67b686484ce8f8b16a69e7a8a96fd5430685806"} Nov 01 20:45:23.303362 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.303328 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bc9b444-1915-4ba3-a7f7-789549742e85-istio-data\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms\" (UID: \"6bc9b444-1915-4ba3-a7f7-789549742e85\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms" Nov 01 20:45:23.303362 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.303366 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc9b444-1915-4ba3-a7f7-789549742e85-workload-socket\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms\" (UID: \"6bc9b444-1915-4ba3-a7f7-789549742e85\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms" Nov 01 20:45:23.303598 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.303388 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bc9b444-1915-4ba3-a7f7-789549742e85-istiod-ca-cert\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms\" (UID: \"6bc9b444-1915-4ba3-a7f7-789549742e85\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms" Nov 01 20:45:23.303598 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.303417 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z4rj\" (UniqueName: \"kubernetes.io/projected/6bc9b444-1915-4ba3-a7f7-789549742e85-kube-api-access-4z4rj\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms\" (UID: \"6bc9b444-1915-4ba3-a7f7-789549742e85\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms" Nov 01 20:45:23.303598 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.303455 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc9b444-1915-4ba3-a7f7-789549742e85-credential-socket\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms\" (UID: \"6bc9b444-1915-4ba3-a7f7-789549742e85\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms" Nov 01 20:45:23.303598 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.303476 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bc9b444-1915-4ba3-a7f7-789549742e85-istio-token\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms\" (UID: \"6bc9b444-1915-4ba3-a7f7-789549742e85\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms" Nov 01 20:45:23.303598 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.303493 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bc9b444-1915-4ba3-a7f7-789549742e85-istio-envoy\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms\" (UID: \"6bc9b444-1915-4ba3-a7f7-789549742e85\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms" Nov 01 20:45:23.303853 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.303593 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bc9b444-1915-4ba3-a7f7-789549742e85-workload-certs\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms\" (UID: \"6bc9b444-1915-4ba3-a7f7-789549742e85\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms" Nov 01 20:45:23.303853 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.303697 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bc9b444-1915-4ba3-a7f7-789549742e85-istio-podinfo\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms\" (UID: \"6bc9b444-1915-4ba3-a7f7-789549742e85\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms" Nov 01 20:45:23.304097 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.304041 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc9b444-1915-4ba3-a7f7-789549742e85-workload-socket\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms\" (UID: \"6bc9b444-1915-4ba3-a7f7-789549742e85\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms" Nov 01 20:45:23.304207 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.304162 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bc9b444-1915-4ba3-a7f7-789549742e85-istio-data\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms\" (UID: \"6bc9b444-1915-4ba3-a7f7-789549742e85\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms" Nov 01 20:45:23.304270 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.304207 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bc9b444-1915-4ba3-a7f7-789549742e85-workload-certs\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms\" (UID: \"6bc9b444-1915-4ba3-a7f7-789549742e85\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms" Nov 01 20:45:23.304270 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.304228 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc9b444-1915-4ba3-a7f7-789549742e85-credential-socket\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms\" (UID: \"6bc9b444-1915-4ba3-a7f7-789549742e85\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms" Nov 01 20:45:23.304444 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.304423 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bc9b444-1915-4ba3-a7f7-789549742e85-istiod-ca-cert\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms\" (UID: \"6bc9b444-1915-4ba3-a7f7-789549742e85\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms" Nov 01 20:45:23.305854 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.305828 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bc9b444-1915-4ba3-a7f7-789549742e85-istio-envoy\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms\" (UID: \"6bc9b444-1915-4ba3-a7f7-789549742e85\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms" Nov 01 20:45:23.306074 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.306057 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bc9b444-1915-4ba3-a7f7-789549742e85-istio-podinfo\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms\" (UID: \"6bc9b444-1915-4ba3-a7f7-789549742e85\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms" Nov 01 20:45:23.361643 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.361615 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bc9b444-1915-4ba3-a7f7-789549742e85-istio-token\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms\" (UID: \"6bc9b444-1915-4ba3-a7f7-789549742e85\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms" Nov 01 20:45:23.361903 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.361879 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z4rj\" (UniqueName: \"kubernetes.io/projected/6bc9b444-1915-4ba3-a7f7-789549742e85-kube-api-access-4z4rj\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms\" (UID: \"6bc9b444-1915-4ba3-a7f7-789549742e85\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms" Nov 01 20:45:23.363577 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.363554 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms" Nov 01 20:45:23.371505 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.371478 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="143424a0-981e-428f-ae6f-9c33ea0d5836" path="/var/lib/kubelet/pods/143424a0-981e-428f-ae6f-9c33ea0d5836/volumes" Nov 01 20:45:23.371892 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.371878 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d22905a-1b38-4c16-81fc-66f1cefd6c7e" path="/var/lib/kubelet/pods/3d22905a-1b38-4c16-81fc-66f1cefd6c7e/volumes" Nov 01 20:45:23.372339 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.372317 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4964cf9-925d-4258-b8fe-2cc516b79487" path="/var/lib/kubelet/pods/b4964cf9-925d-4258-b8fe-2cc516b79487/volumes" Nov 01 20:45:23.373130 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.373113 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f16b797d-6930-414c-9957-bff940a16153" path="/var/lib/kubelet/pods/f16b797d-6930-414c-9957-bff940a16153/volumes" Nov 01 20:45:23.843462 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.843436 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms"] Nov 01 20:45:23.844046 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:45:23.844018 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bc9b444_1915_4ba3_a7f7_789549742e85.slice/crio-df922b6b099c935f2d2a523ee78ef063829255e07f1104e5dd5044f8cfc72163 WatchSource:0}: Error finding container df922b6b099c935f2d2a523ee78ef063829255e07f1104e5dd5044f8cfc72163: Status 404 returned error can't find the container with id df922b6b099c935f2d2a523ee78ef063829255e07f1104e5dd5044f8cfc72163 Nov 01 20:45:23.846531 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.846497 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:45:23.846640 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.846578 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:45:23.846640 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:23.846631 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:45:24.254443 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:24.254405 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms" event={"ID":"6bc9b444-1915-4ba3-a7f7-789549742e85","Type":"ContainerStarted","Data":"e8cdf47147083f8aab89976bba8b2812b6ada4417df3e325740374d4965cbe8d"} Nov 01 20:45:24.255063 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:24.255027 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms" event={"ID":"6bc9b444-1915-4ba3-a7f7-789549742e85","Type":"ContainerStarted","Data":"df922b6b099c935f2d2a523ee78ef063829255e07f1104e5dd5044f8cfc72163"} Nov 01 20:45:24.498094 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:24.498061 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms" Nov 01 20:45:24.502067 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:24.502029 4484 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 01 20:45:24.502327 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:24.502289 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms" podUID="6bc9b444-1915-4ba3-a7f7-789549742e85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 01 20:45:24.570616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:24.570579 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e93c2f-nnclz" Nov 01 20:45:24.627970 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:24.627059 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d87c9034-5ef9-4323-be7b-040921d4bf35-kube-api-access-5pwtl\" (UniqueName: \"kubernetes.io/projected/d87c9034-5ef9-4323-be7b-040921d4bf35-kube-api-access-5pwtl\") pod \"d87c9034-5ef9-4323-be7b-040921d4bf35\" (UID: \"d87c9034-5ef9-4323-be7b-040921d4bf35\") " Nov 01 20:45:24.627970 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:24.627125 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d87c9034-5ef9-4323-be7b-040921d4bf35-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d87c9034-5ef9-4323-be7b-040921d4bf35-maas-api-xks-ca\") pod \"d87c9034-5ef9-4323-be7b-040921d4bf35\" (UID: \"d87c9034-5ef9-4323-be7b-040921d4bf35\") " Nov 01 20:45:24.627970 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:24.627181 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d87c9034-5ef9-4323-be7b-040921d4bf35-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d87c9034-5ef9-4323-be7b-040921d4bf35-maas-api-openshift-ca\") pod \"d87c9034-5ef9-4323-be7b-040921d4bf35\" (UID: \"d87c9034-5ef9-4323-be7b-040921d4bf35\") " Nov 01 20:45:24.628798 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:24.628477 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d87c9034-5ef9-4323-be7b-040921d4bf35-maas-api-openshift-ca" pod "d87c9034-5ef9-4323-be7b-040921d4bf35" (UID: "d87c9034-5ef9-4323-be7b-040921d4bf35"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:45:24.641030 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:24.638973 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d87c9034-5ef9-4323-be7b-040921d4bf35-kube-api-access-5pwtl" pod "d87c9034-5ef9-4323-be7b-040921d4bf35" (UID: "d87c9034-5ef9-4323-be7b-040921d4bf35"). InnerVolumeSpecName "kube-api-access-5pwtl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:45:24.641223 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:24.641032 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d87c9034-5ef9-4323-be7b-040921d4bf35-maas-api-xks-ca" pod "d87c9034-5ef9-4323-be7b-040921d4bf35" (UID: "d87c9034-5ef9-4323-be7b-040921d4bf35"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:45:24.706021 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:24.694060 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms" podStartSLOduration=2.69403754 podStartE2EDuration="2.69403754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:45:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:45:24.480656955 +0000 UTC m=+875.258297093" watchObservedRunningTime="2026-11-01 20:45:24.69403754 +0000 UTC m=+875.471677679" Nov 01 20:45:24.736019 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:24.729733 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pwtl\" (UniqueName: \"kubernetes.io/projected/d87c9034-5ef9-4323-be7b-040921d4bf35-kube-api-access-5pwtl\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:24.736019 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:24.729768 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d87c9034-5ef9-4323-be7b-040921d4bf35-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:24.736019 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:24.729783 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d87c9034-5ef9-4323-be7b-040921d4bf35-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:25.278022 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:25.265855 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e93c2f-nnclz" Nov 01 20:45:25.278022 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:25.270623 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e93c2f-nnclz" event={"ID":"d87c9034-5ef9-4323-be7b-040921d4bf35","Type":"ContainerDied","Data":"ce826f5f4f75d3a1a61e02ef6b351ef1c9228a752c86b5db384301890f85a6c9"} Nov 01 20:45:25.278022 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:25.270726 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce826f5f4f75d3a1a61e02ef6b351ef1c9228a752c86b5db384301890f85a6c9" Nov 01 20:45:25.383020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:25.380233 4484 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 01 20:45:25.383020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:25.382133 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms" podUID="6bc9b444-1915-4ba3-a7f7-789549742e85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 01 20:45:26.053019 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:26.052411 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:26.366942 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:26.366851 4484 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 01 20:45:26.367512 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:26.367480 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms" podUID="6bc9b444-1915-4ba3-a7f7-789549742e85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 01 20:45:27.364761 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:27.364725 4484 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 01 20:45:27.364917 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:27.364795 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms" podUID="6bc9b444-1915-4ba3-a7f7-789549742e85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 01 20:45:27.533040 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:27.532980 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0902bc45-ptppj"] Nov 01 20:45:27.533905 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:27.533883 4484 cpu_manager.go:414] "containerMap: removing container" podUID="3d22905a-1b38-4c16-81fc-66f1cefd6c7e" containerName="cleanup" Nov 01 20:45:27.533944 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:27.533909 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d22905a-1b38-4c16-81fc-66f1cefd6c7e" containerName="cleanup" Nov 01 20:45:27.533975 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:27.533963 4484 cpu_manager.go:414] "containerMap: removing container" podUID="d87c9034-5ef9-4323-be7b-040921d4bf35" containerName="revoke-keys" Nov 01 20:45:27.534021 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:27.533971 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d87c9034-5ef9-4323-be7b-040921d4bf35" containerName="revoke-keys" Nov 01 20:45:27.534021 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:27.534016 4484 cpu_manager.go:414] "containerMap: removing container" podUID="143424a0-981e-428f-ae6f-9c33ea0d5836" containerName="cleanup" Nov 01 20:45:27.534090 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:27.534024 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="143424a0-981e-428f-ae6f-9c33ea0d5836" containerName="cleanup" Nov 01 20:45:27.534090 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:27.534039 4484 cpu_manager.go:414] "containerMap: removing container" podUID="f16b797d-6930-414c-9957-bff940a16153" containerName="cleanup" Nov 01 20:45:27.534090 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:27.534046 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f16b797d-6930-414c-9957-bff940a16153" containerName="cleanup" Nov 01 20:45:27.534235 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:27.534221 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="d87c9034-5ef9-4323-be7b-040921d4bf35" containerName="revoke-keys" Nov 01 20:45:27.534268 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:27.534239 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="f16b797d-6930-414c-9957-bff940a16153" containerName="cleanup" Nov 01 20:45:27.534268 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:27.534252 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="143424a0-981e-428f-ae6f-9c33ea0d5836" containerName="cleanup" Nov 01 20:45:27.534331 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:27.534267 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d22905a-1b38-4c16-81fc-66f1cefd6c7e" containerName="cleanup" Nov 01 20:45:27.538366 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:27.538338 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0902bc45-ptppj" Nov 01 20:45:27.571901 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:27.571865 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5780b494-dc4c-47a7-b2df-52af36460639-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0902bc45-ptppj\" (UID: \"5780b494-dc4c-47a7-b2df-52af36460639\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0902bc45-ptppj" Nov 01 20:45:27.572107 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:27.571914 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5780b494-dc4c-47a7-b2df-52af36460639-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0902bc45-ptppj\" (UID: \"5780b494-dc4c-47a7-b2df-52af36460639\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0902bc45-ptppj" Nov 01 20:45:27.572107 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:27.571933 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n54w5\" (UniqueName: \"kubernetes.io/projected/5780b494-dc4c-47a7-b2df-52af36460639-kube-api-access-n54w5\") pod \"maas-api-revoke-sub-0902bc45-ptppj\" (UID: \"5780b494-dc4c-47a7-b2df-52af36460639\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0902bc45-ptppj" Nov 01 20:45:27.594088 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:27.594050 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0902bc45-ptppj"] Nov 01 20:45:27.595826 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:27.595802 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:45:27.640411 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:27.640327 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:45:27.640798 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:27.640327 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 20:45:27.640970 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:27.640725 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:45:27.674029 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:27.673756 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5780b494-dc4c-47a7-b2df-52af36460639-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0902bc45-ptppj\" (UID: \"5780b494-dc4c-47a7-b2df-52af36460639\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0902bc45-ptppj" Nov 01 20:45:27.675545 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:27.674378 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5780b494-dc4c-47a7-b2df-52af36460639-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0902bc45-ptppj\" (UID: \"5780b494-dc4c-47a7-b2df-52af36460639\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0902bc45-ptppj" Nov 01 20:45:27.675545 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:27.675149 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5780b494-dc4c-47a7-b2df-52af36460639-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0902bc45-ptppj\" (UID: \"5780b494-dc4c-47a7-b2df-52af36460639\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0902bc45-ptppj" Nov 01 20:45:27.675545 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:27.675265 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n54w5\" (UniqueName: \"kubernetes.io/projected/5780b494-dc4c-47a7-b2df-52af36460639-kube-api-access-n54w5\") pod \"maas-api-revoke-sub-0902bc45-ptppj\" (UID: \"5780b494-dc4c-47a7-b2df-52af36460639\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0902bc45-ptppj" Nov 01 20:45:27.682024 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:27.681732 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5780b494-dc4c-47a7-b2df-52af36460639-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0902bc45-ptppj\" (UID: \"5780b494-dc4c-47a7-b2df-52af36460639\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0902bc45-ptppj" Nov 01 20:45:27.733871 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:27.733833 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n54w5\" (UniqueName: \"kubernetes.io/projected/5780b494-dc4c-47a7-b2df-52af36460639-kube-api-access-n54w5\") pod \"maas-api-revoke-sub-0902bc45-ptppj\" (UID: \"5780b494-dc4c-47a7-b2df-52af36460639\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0902bc45-ptppj" Nov 01 20:45:27.862017 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:27.860275 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0902bc45-ptppj" Nov 01 20:45:28.364301 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:28.364265 4484 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 01 20:45:28.364484 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:28.364343 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms" podUID="6bc9b444-1915-4ba3-a7f7-789549742e85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 01 20:45:28.422057 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:28.419775 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0902bc45-ptppj"] Nov 01 20:45:29.343461 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:29.343418 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0902bc45-ptppj" event={"ID":"5780b494-dc4c-47a7-b2df-52af36460639","Type":"ContainerStarted","Data":"ab015295258d73e4d54c396fe05bf46f76537e0b764aef0d8ea3d3b556ce1df1"} Nov 01 20:45:29.344051 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:29.343488 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0902bc45-ptppj" event={"ID":"5780b494-dc4c-47a7-b2df-52af36460639","Type":"ContainerStarted","Data":"8a7b67c15877d71bc5b77b7708f7e3dd8f56059eb2900d84c91b79ff3a64d8af"} Nov 01 20:45:29.364952 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:29.364914 4484 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 01 20:45:29.365119 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:29.364969 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms" podUID="6bc9b444-1915-4ba3-a7f7-789549742e85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 01 20:45:30.351919 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:30.351877 4484 generic.go:356] "Generic (PLEG): container finished" podID="5780b494-dc4c-47a7-b2df-52af36460639" containerID="ab015295258d73e4d54c396fe05bf46f76537e0b764aef0d8ea3d3b556ce1df1" exitCode=0 Nov 01 20:45:30.352322 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:30.351961 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0902bc45-ptppj" event={"ID":"5780b494-dc4c-47a7-b2df-52af36460639","Type":"ContainerDied","Data":"ab015295258d73e4d54c396fe05bf46f76537e0b764aef0d8ea3d3b556ce1df1"} Nov 01 20:45:30.364444 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:30.364403 4484 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 01 20:45:30.364596 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:30.364487 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms" podUID="6bc9b444-1915-4ba3-a7f7-789549742e85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 01 20:45:30.931204 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:30.931163 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:31.374044 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:31.374015 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms" Nov 01 20:45:31.374476 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:31.374389 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms" Nov 01 20:45:31.375218 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:31.375203 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms" Nov 01 20:45:31.506650 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:31.506623 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0902bc45-ptppj" Nov 01 20:45:31.590211 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:31.590185 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5780b494-dc4c-47a7-b2df-52af36460639-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5780b494-dc4c-47a7-b2df-52af36460639-maas-api-openshift-ca\") pod \"5780b494-dc4c-47a7-b2df-52af36460639\" (UID: \"5780b494-dc4c-47a7-b2df-52af36460639\") " Nov 01 20:45:31.590393 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:31.590289 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5780b494-dc4c-47a7-b2df-52af36460639-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5780b494-dc4c-47a7-b2df-52af36460639-maas-api-xks-ca\") pod \"5780b494-dc4c-47a7-b2df-52af36460639\" (UID: \"5780b494-dc4c-47a7-b2df-52af36460639\") " Nov 01 20:45:31.590506 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:31.590486 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5780b494-dc4c-47a7-b2df-52af36460639-kube-api-access-n54w5\" (UniqueName: \"kubernetes.io/projected/5780b494-dc4c-47a7-b2df-52af36460639-kube-api-access-n54w5\") pod \"5780b494-dc4c-47a7-b2df-52af36460639\" (UID: \"5780b494-dc4c-47a7-b2df-52af36460639\") " Nov 01 20:45:31.590559 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:31.590534 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5780b494-dc4c-47a7-b2df-52af36460639-maas-api-openshift-ca" pod "5780b494-dc4c-47a7-b2df-52af36460639" (UID: "5780b494-dc4c-47a7-b2df-52af36460639"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:45:31.590886 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:31.590866 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5780b494-dc4c-47a7-b2df-52af36460639-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:31.592423 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:31.592397 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5780b494-dc4c-47a7-b2df-52af36460639-kube-api-access-n54w5" pod "5780b494-dc4c-47a7-b2df-52af36460639" (UID: "5780b494-dc4c-47a7-b2df-52af36460639"). InnerVolumeSpecName "kube-api-access-n54w5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:45:31.592496 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:31.592453 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5780b494-dc4c-47a7-b2df-52af36460639-maas-api-xks-ca" pod "5780b494-dc4c-47a7-b2df-52af36460639" (UID: "5780b494-dc4c-47a7-b2df-52af36460639"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:45:31.692799 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:31.692692 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n54w5\" (UniqueName: \"kubernetes.io/projected/5780b494-dc4c-47a7-b2df-52af36460639-kube-api-access-n54w5\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:31.692799 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:31.692729 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5780b494-dc4c-47a7-b2df-52af36460639-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:32.367337 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:32.367307 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0902bc45-ptppj" Nov 01 20:45:32.367512 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:32.367351 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0902bc45-ptppj" event={"ID":"5780b494-dc4c-47a7-b2df-52af36460639","Type":"ContainerDied","Data":"8a7b67c15877d71bc5b77b7708f7e3dd8f56059eb2900d84c91b79ff3a64d8af"} Nov 01 20:45:32.367512 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:32.367393 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a7b67c15877d71bc5b77b7708f7e3dd8f56059eb2900d84c91b79ff3a64d8af" Nov 01 20:45:35.744831 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:35.744794 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:37.850317 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:37.850279 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:40.527701 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:40.527668 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:45.423018 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:45.419580 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:45.695077 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:45.694751 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-438e9c5e-rt5dz"] Nov 01 20:45:45.708016 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:45.695633 4484 cpu_manager.go:414] "containerMap: removing container" podUID="5780b494-dc4c-47a7-b2df-52af36460639" containerName="revoke-keys" Nov 01 20:45:45.708016 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:45.695657 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5780b494-dc4c-47a7-b2df-52af36460639" containerName="revoke-keys" Nov 01 20:45:45.708016 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:45.695828 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="5780b494-dc4c-47a7-b2df-52af36460639" containerName="revoke-keys" Nov 01 20:45:45.708016 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:45.701197 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-438e9c5e-rt5dz" Nov 01 20:45:45.797019 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:45.766319 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwslm\" (UniqueName: \"kubernetes.io/projected/ee3f4ee2-4fa0-4db0-8bc0-19a591f90f1e-kube-api-access-lwslm\") pod \"maas-api-revoke-sub-438e9c5e-rt5dz\" (UID: \"ee3f4ee2-4fa0-4db0-8bc0-19a591f90f1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-438e9c5e-rt5dz" Nov 01 20:45:45.797436 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:45.778961 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-438e9c5e-rt5dz"] Nov 01 20:45:45.797900 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:45.797739 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee3f4ee2-4fa0-4db0-8bc0-19a591f90f1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-438e9c5e-rt5dz\" (UID: \"ee3f4ee2-4fa0-4db0-8bc0-19a591f90f1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-438e9c5e-rt5dz" Nov 01 20:45:45.798166 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:45.798149 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee3f4ee2-4fa0-4db0-8bc0-19a591f90f1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-438e9c5e-rt5dz\" (UID: \"ee3f4ee2-4fa0-4db0-8bc0-19a591f90f1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-438e9c5e-rt5dz" Nov 01 20:45:45.800536 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:45.800510 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 20:45:45.800709 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:45.800695 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:45:45.806028 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:45.801503 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:45:45.827046 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:45.822200 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:45:45.902022 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:45.860304 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e72e83b8-8wq9j"] Nov 01 20:45:45.902022 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:45.894728 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72e83b8-8wq9j" Nov 01 20:45:45.911032 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:45.902453 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee3f4ee2-4fa0-4db0-8bc0-19a591f90f1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-438e9c5e-rt5dz\" (UID: \"ee3f4ee2-4fa0-4db0-8bc0-19a591f90f1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-438e9c5e-rt5dz" Nov 01 20:45:45.911032 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:45.902574 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwslm\" (UniqueName: \"kubernetes.io/projected/ee3f4ee2-4fa0-4db0-8bc0-19a591f90f1e-kube-api-access-lwslm\") pod \"maas-api-revoke-sub-438e9c5e-rt5dz\" (UID: \"ee3f4ee2-4fa0-4db0-8bc0-19a591f90f1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-438e9c5e-rt5dz" Nov 01 20:45:45.911032 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:45.902724 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee3f4ee2-4fa0-4db0-8bc0-19a591f90f1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-438e9c5e-rt5dz\" (UID: \"ee3f4ee2-4fa0-4db0-8bc0-19a591f90f1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-438e9c5e-rt5dz" Nov 01 20:45:45.982073 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:45.970276 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee3f4ee2-4fa0-4db0-8bc0-19a591f90f1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-438e9c5e-rt5dz\" (UID: \"ee3f4ee2-4fa0-4db0-8bc0-19a591f90f1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-438e9c5e-rt5dz" Nov 01 20:45:45.982073 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:45:45.979690 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 20:45:45.989099 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:45.985334 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee3f4ee2-4fa0-4db0-8bc0-19a591f90f1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-438e9c5e-rt5dz\" (UID: \"ee3f4ee2-4fa0-4db0-8bc0-19a591f90f1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-438e9c5e-rt5dz" Nov 01 20:45:46.007795 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:46.007755 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwslm\" (UniqueName: \"kubernetes.io/projected/ee3f4ee2-4fa0-4db0-8bc0-19a591f90f1e-kube-api-access-lwslm\") pod \"maas-api-revoke-sub-438e9c5e-rt5dz\" (UID: \"ee3f4ee2-4fa0-4db0-8bc0-19a591f90f1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-438e9c5e-rt5dz" Nov 01 20:45:46.008770 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:46.008658 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zqzd\" (UniqueName: \"kubernetes.io/projected/9d51ed2a-f4ab-434a-af8b-ebf29ed29198-kube-api-access-4zqzd\") pod \"maas-api-revoke-sub-e72e83b8-8wq9j\" (UID: \"9d51ed2a-f4ab-434a-af8b-ebf29ed29198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72e83b8-8wq9j" Nov 01 20:45:46.009057 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:46.009035 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d51ed2a-f4ab-434a-af8b-ebf29ed29198-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e72e83b8-8wq9j\" (UID: \"9d51ed2a-f4ab-434a-af8b-ebf29ed29198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72e83b8-8wq9j" Nov 01 20:45:46.009303 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:46.009284 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d51ed2a-f4ab-434a-af8b-ebf29ed29198-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e72e83b8-8wq9j\" (UID: \"9d51ed2a-f4ab-434a-af8b-ebf29ed29198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72e83b8-8wq9j" Nov 01 20:45:46.087109 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:46.078095 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-438e9c5e-rt5dz" Nov 01 20:45:46.112387 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:46.112355 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zqzd\" (UniqueName: \"kubernetes.io/projected/9d51ed2a-f4ab-434a-af8b-ebf29ed29198-kube-api-access-4zqzd\") pod \"maas-api-revoke-sub-e72e83b8-8wq9j\" (UID: \"9d51ed2a-f4ab-434a-af8b-ebf29ed29198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72e83b8-8wq9j" Nov 01 20:45:46.121378 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:46.120063 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e72e83b8-8wq9j"] Nov 01 20:45:46.121610 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:46.121591 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d51ed2a-f4ab-434a-af8b-ebf29ed29198-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e72e83b8-8wq9j\" (UID: \"9d51ed2a-f4ab-434a-af8b-ebf29ed29198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72e83b8-8wq9j" Nov 01 20:45:46.122362 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:46.122341 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d51ed2a-f4ab-434a-af8b-ebf29ed29198-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e72e83b8-8wq9j\" (UID: \"9d51ed2a-f4ab-434a-af8b-ebf29ed29198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72e83b8-8wq9j" Nov 01 20:45:46.123405 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:46.123192 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d51ed2a-f4ab-434a-af8b-ebf29ed29198-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e72e83b8-8wq9j\" (UID: \"9d51ed2a-f4ab-434a-af8b-ebf29ed29198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72e83b8-8wq9j" Nov 01 20:45:46.129261 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:46.125344 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1184d79b-27xvv"] Nov 01 20:45:46.151900 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:46.151827 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d51ed2a-f4ab-434a-af8b-ebf29ed29198-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e72e83b8-8wq9j\" (UID: \"9d51ed2a-f4ab-434a-af8b-ebf29ed29198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72e83b8-8wq9j" Nov 01 20:45:46.162277 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:46.159297 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1184d79b-27xvv" Nov 01 20:45:46.174343 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:46.174310 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1184d79b-27xvv"] Nov 01 20:45:46.184332 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:46.183715 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zqzd\" (UniqueName: \"kubernetes.io/projected/9d51ed2a-f4ab-434a-af8b-ebf29ed29198-kube-api-access-4zqzd\") pod \"maas-api-revoke-sub-e72e83b8-8wq9j\" (UID: \"9d51ed2a-f4ab-434a-af8b-ebf29ed29198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72e83b8-8wq9j" Nov 01 20:45:46.226276 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:46.226236 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r5k8\" (UniqueName: \"kubernetes.io/projected/5e0e4ee4-329b-4600-89c5-82e6de5b0479-kube-api-access-9r5k8\") pod \"maas-api-revoke-sub-1184d79b-27xvv\" (UID: \"5e0e4ee4-329b-4600-89c5-82e6de5b0479\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1184d79b-27xvv" Nov 01 20:45:46.226446 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:46.226332 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e0e4ee4-329b-4600-89c5-82e6de5b0479-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1184d79b-27xvv\" (UID: \"5e0e4ee4-329b-4600-89c5-82e6de5b0479\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1184d79b-27xvv" Nov 01 20:45:46.226446 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:46.226419 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e0e4ee4-329b-4600-89c5-82e6de5b0479-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1184d79b-27xvv\" (UID: \"5e0e4ee4-329b-4600-89c5-82e6de5b0479\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1184d79b-27xvv" Nov 01 20:45:46.327958 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:46.327919 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r5k8\" (UniqueName: \"kubernetes.io/projected/5e0e4ee4-329b-4600-89c5-82e6de5b0479-kube-api-access-9r5k8\") pod \"maas-api-revoke-sub-1184d79b-27xvv\" (UID: \"5e0e4ee4-329b-4600-89c5-82e6de5b0479\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1184d79b-27xvv" Nov 01 20:45:46.328350 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:46.328326 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e0e4ee4-329b-4600-89c5-82e6de5b0479-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1184d79b-27xvv\" (UID: \"5e0e4ee4-329b-4600-89c5-82e6de5b0479\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1184d79b-27xvv" Nov 01 20:45:46.328468 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:46.328410 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e0e4ee4-329b-4600-89c5-82e6de5b0479-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1184d79b-27xvv\" (UID: \"5e0e4ee4-329b-4600-89c5-82e6de5b0479\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1184d79b-27xvv" Nov 01 20:45:46.329199 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:46.329172 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e0e4ee4-329b-4600-89c5-82e6de5b0479-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1184d79b-27xvv\" (UID: \"5e0e4ee4-329b-4600-89c5-82e6de5b0479\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1184d79b-27xvv" Nov 01 20:45:46.331531 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:46.331509 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e0e4ee4-329b-4600-89c5-82e6de5b0479-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1184d79b-27xvv\" (UID: \"5e0e4ee4-329b-4600-89c5-82e6de5b0479\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1184d79b-27xvv" Nov 01 20:45:46.386214 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:46.386180 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r5k8\" (UniqueName: \"kubernetes.io/projected/5e0e4ee4-329b-4600-89c5-82e6de5b0479-kube-api-access-9r5k8\") pod \"maas-api-revoke-sub-1184d79b-27xvv\" (UID: \"5e0e4ee4-329b-4600-89c5-82e6de5b0479\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1184d79b-27xvv" Nov 01 20:45:46.462638 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:46.462551 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72e83b8-8wq9j" Nov 01 20:45:46.512949 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:46.512666 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1184d79b-27xvv" Nov 01 20:45:46.566125 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:46.566069 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-438e9c5e-rt5dz"] Nov 01 20:45:46.872458 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:46.872367 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e72e83b8-8wq9j"] Nov 01 20:45:46.914506 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:45:46.914469 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d51ed2a_f4ab_434a_af8b_ebf29ed29198.slice/crio-3f8ab3c76baa754401af9fc562472f73b72412a496fa038534b078eef79773e3 WatchSource:0}: Error finding container 3f8ab3c76baa754401af9fc562472f73b72412a496fa038534b078eef79773e3: Status 404 returned error can't find the container with id 3f8ab3c76baa754401af9fc562472f73b72412a496fa038534b078eef79773e3 Nov 01 20:45:46.933520 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:46.933489 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1184d79b-27xvv"] Nov 01 20:45:46.934204 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:45:46.934154 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e0e4ee4_329b_4600_89c5_82e6de5b0479.slice/crio-37626788a12627c37279e84027d0a13e13dca6e57b40ed2aaacde9149409bc68 WatchSource:0}: Error finding container 37626788a12627c37279e84027d0a13e13dca6e57b40ed2aaacde9149409bc68: Status 404 returned error can't find the container with id 37626788a12627c37279e84027d0a13e13dca6e57b40ed2aaacde9149409bc68 Nov 01 20:45:47.496465 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:47.496427 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1184d79b-27xvv" event={"ID":"5e0e4ee4-329b-4600-89c5-82e6de5b0479","Type":"ContainerStarted","Data":"355fc072c5aa5cc2c2e94123d319d22f10b0cf82d1c8dfe7023320c1c6d973d9"} Nov 01 20:45:47.496917 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:47.496495 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1184d79b-27xvv" event={"ID":"5e0e4ee4-329b-4600-89c5-82e6de5b0479","Type":"ContainerStarted","Data":"37626788a12627c37279e84027d0a13e13dca6e57b40ed2aaacde9149409bc68"} Nov 01 20:45:47.498199 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:47.498162 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-438e9c5e-rt5dz" event={"ID":"ee3f4ee2-4fa0-4db0-8bc0-19a591f90f1e","Type":"ContainerStarted","Data":"a346337f718291adb82a33b9ef360d3eaa2314580ce99f180567b3ab72f6cd8c"} Nov 01 20:45:47.498346 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:47.498212 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-438e9c5e-rt5dz" event={"ID":"ee3f4ee2-4fa0-4db0-8bc0-19a591f90f1e","Type":"ContainerStarted","Data":"862b9b50bcaa725f7ce64f64d865c27cc478c2b329d4222a00533c840a8ee4ff"} Nov 01 20:45:47.499845 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:47.499821 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72e83b8-8wq9j" event={"ID":"9d51ed2a-f4ab-434a-af8b-ebf29ed29198","Type":"ContainerStarted","Data":"33606dc0bbd0a7596e351dce4be71457e29f1648345463715ec13bdde553bdab"} Nov 01 20:45:47.499959 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:47.499860 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72e83b8-8wq9j" event={"ID":"9d51ed2a-f4ab-434a-af8b-ebf29ed29198","Type":"ContainerStarted","Data":"3f8ab3c76baa754401af9fc562472f73b72412a496fa038534b078eef79773e3"} Nov 01 20:45:47.614644 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:47.614584 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1184d79b-27xvv" podStartSLOduration=2.614568892 podStartE2EDuration="2.614568892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:45:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:45:47.611268545 +0000 UTC m=+898.388908714" watchObservedRunningTime="2026-11-01 20:45:47.614568892 +0000 UTC m=+898.392209037" Nov 01 20:45:47.728264 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:47.728209 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-438e9c5e-rt5dz" podStartSLOduration=2.72819471 podStartE2EDuration="2.72819471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:45:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:45:47.725188925 +0000 UTC m=+898.502829081" watchObservedRunningTime="2026-11-01 20:45:47.72819471 +0000 UTC m=+898.505834846" Nov 01 20:45:47.841288 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:47.841241 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72e83b8-8wq9j" podStartSLOduration=2.841225894 podStartE2EDuration="2.841225894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:45:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:45:47.841028545 +0000 UTC m=+898.618668678" watchObservedRunningTime="2026-11-01 20:45:47.841225894 +0000 UTC m=+898.618866028" Nov 01 20:45:48.507484 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:48.507443 4484 generic.go:356] "Generic (PLEG): container finished" podID="5e0e4ee4-329b-4600-89c5-82e6de5b0479" containerID="355fc072c5aa5cc2c2e94123d319d22f10b0cf82d1c8dfe7023320c1c6d973d9" exitCode=0 Nov 01 20:45:48.507964 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:48.507532 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1184d79b-27xvv" event={"ID":"5e0e4ee4-329b-4600-89c5-82e6de5b0479","Type":"ContainerDied","Data":"355fc072c5aa5cc2c2e94123d319d22f10b0cf82d1c8dfe7023320c1c6d973d9"} Nov 01 20:45:48.509026 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:48.508989 4484 generic.go:356] "Generic (PLEG): container finished" podID="ee3f4ee2-4fa0-4db0-8bc0-19a591f90f1e" containerID="a346337f718291adb82a33b9ef360d3eaa2314580ce99f180567b3ab72f6cd8c" exitCode=0 Nov 01 20:45:48.509136 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:48.509085 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-438e9c5e-rt5dz" event={"ID":"ee3f4ee2-4fa0-4db0-8bc0-19a591f90f1e","Type":"ContainerDied","Data":"a346337f718291adb82a33b9ef360d3eaa2314580ce99f180567b3ab72f6cd8c"} Nov 01 20:45:48.510545 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:48.510522 4484 generic.go:356] "Generic (PLEG): container finished" podID="9d51ed2a-f4ab-434a-af8b-ebf29ed29198" containerID="33606dc0bbd0a7596e351dce4be71457e29f1648345463715ec13bdde553bdab" exitCode=0 Nov 01 20:45:48.510642 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:48.510597 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72e83b8-8wq9j" event={"ID":"9d51ed2a-f4ab-434a-af8b-ebf29ed29198","Type":"ContainerDied","Data":"33606dc0bbd0a7596e351dce4be71457e29f1648345463715ec13bdde553bdab"} Nov 01 20:45:49.677950 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:49.677921 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1184d79b-27xvv" Nov 01 20:45:49.731770 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:49.731740 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-438e9c5e-rt5dz" Nov 01 20:45:49.750199 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:49.750141 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72e83b8-8wq9j" Nov 01 20:45:49.771949 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:49.771918 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e0e4ee4-329b-4600-89c5-82e6de5b0479-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e0e4ee4-329b-4600-89c5-82e6de5b0479-maas-api-xks-ca\") pod \"5e0e4ee4-329b-4600-89c5-82e6de5b0479\" (UID: \"5e0e4ee4-329b-4600-89c5-82e6de5b0479\") " Nov 01 20:45:49.772102 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:49.771985 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee3f4ee2-4fa0-4db0-8bc0-19a591f90f1e-kube-api-access-lwslm\" (UniqueName: \"kubernetes.io/projected/ee3f4ee2-4fa0-4db0-8bc0-19a591f90f1e-kube-api-access-lwslm\") pod \"ee3f4ee2-4fa0-4db0-8bc0-19a591f90f1e\" (UID: \"ee3f4ee2-4fa0-4db0-8bc0-19a591f90f1e\") " Nov 01 20:45:49.772175 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:49.772156 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee3f4ee2-4fa0-4db0-8bc0-19a591f90f1e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee3f4ee2-4fa0-4db0-8bc0-19a591f90f1e-maas-api-openshift-ca\") pod \"ee3f4ee2-4fa0-4db0-8bc0-19a591f90f1e\" (UID: \"ee3f4ee2-4fa0-4db0-8bc0-19a591f90f1e\") " Nov 01 20:45:49.772248 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:49.772236 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e0e4ee4-329b-4600-89c5-82e6de5b0479-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e0e4ee4-329b-4600-89c5-82e6de5b0479-maas-api-openshift-ca\") pod \"5e0e4ee4-329b-4600-89c5-82e6de5b0479\" (UID: \"5e0e4ee4-329b-4600-89c5-82e6de5b0479\") " Nov 01 20:45:49.772304 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:49.772269 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee3f4ee2-4fa0-4db0-8bc0-19a591f90f1e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee3f4ee2-4fa0-4db0-8bc0-19a591f90f1e-maas-api-xks-ca\") pod \"ee3f4ee2-4fa0-4db0-8bc0-19a591f90f1e\" (UID: \"ee3f4ee2-4fa0-4db0-8bc0-19a591f90f1e\") " Nov 01 20:45:49.772357 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:49.772326 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e0e4ee4-329b-4600-89c5-82e6de5b0479-kube-api-access-9r5k8\" (UniqueName: \"kubernetes.io/projected/5e0e4ee4-329b-4600-89c5-82e6de5b0479-kube-api-access-9r5k8\") pod \"5e0e4ee4-329b-4600-89c5-82e6de5b0479\" (UID: \"5e0e4ee4-329b-4600-89c5-82e6de5b0479\") " Nov 01 20:45:49.772539 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:49.772510 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee3f4ee2-4fa0-4db0-8bc0-19a591f90f1e-maas-api-openshift-ca" pod "ee3f4ee2-4fa0-4db0-8bc0-19a591f90f1e" (UID: "ee3f4ee2-4fa0-4db0-8bc0-19a591f90f1e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:45:49.772671 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:49.772568 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e0e4ee4-329b-4600-89c5-82e6de5b0479-maas-api-openshift-ca" pod "5e0e4ee4-329b-4600-89c5-82e6de5b0479" (UID: "5e0e4ee4-329b-4600-89c5-82e6de5b0479"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:45:49.772753 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:49.772737 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee3f4ee2-4fa0-4db0-8bc0-19a591f90f1e-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:49.772814 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:49.772755 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e0e4ee4-329b-4600-89c5-82e6de5b0479-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:49.774310 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:49.774282 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee3f4ee2-4fa0-4db0-8bc0-19a591f90f1e-maas-api-xks-ca" pod "ee3f4ee2-4fa0-4db0-8bc0-19a591f90f1e" (UID: "ee3f4ee2-4fa0-4db0-8bc0-19a591f90f1e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:45:49.774310 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:49.774291 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e0e4ee4-329b-4600-89c5-82e6de5b0479-kube-api-access-9r5k8" pod "5e0e4ee4-329b-4600-89c5-82e6de5b0479" (UID: "5e0e4ee4-329b-4600-89c5-82e6de5b0479"). InnerVolumeSpecName "kube-api-access-9r5k8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:45:49.774465 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:49.774447 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e0e4ee4-329b-4600-89c5-82e6de5b0479-maas-api-xks-ca" pod "5e0e4ee4-329b-4600-89c5-82e6de5b0479" (UID: "5e0e4ee4-329b-4600-89c5-82e6de5b0479"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:45:49.774522 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:49.774487 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee3f4ee2-4fa0-4db0-8bc0-19a591f90f1e-kube-api-access-lwslm" pod "ee3f4ee2-4fa0-4db0-8bc0-19a591f90f1e" (UID: "ee3f4ee2-4fa0-4db0-8bc0-19a591f90f1e"). InnerVolumeSpecName "kube-api-access-lwslm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:45:49.873121 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:49.873088 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d51ed2a-f4ab-434a-af8b-ebf29ed29198-kube-api-access-4zqzd\" (UniqueName: \"kubernetes.io/projected/9d51ed2a-f4ab-434a-af8b-ebf29ed29198-kube-api-access-4zqzd\") pod \"9d51ed2a-f4ab-434a-af8b-ebf29ed29198\" (UID: \"9d51ed2a-f4ab-434a-af8b-ebf29ed29198\") " Nov 01 20:45:49.873310 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:49.873204 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d51ed2a-f4ab-434a-af8b-ebf29ed29198-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d51ed2a-f4ab-434a-af8b-ebf29ed29198-maas-api-xks-ca\") pod \"9d51ed2a-f4ab-434a-af8b-ebf29ed29198\" (UID: \"9d51ed2a-f4ab-434a-af8b-ebf29ed29198\") " Nov 01 20:45:49.873396 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:49.873372 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d51ed2a-f4ab-434a-af8b-ebf29ed29198-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d51ed2a-f4ab-434a-af8b-ebf29ed29198-maas-api-openshift-ca\") pod \"9d51ed2a-f4ab-434a-af8b-ebf29ed29198\" (UID: \"9d51ed2a-f4ab-434a-af8b-ebf29ed29198\") " Nov 01 20:45:49.873694 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:49.873666 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d51ed2a-f4ab-434a-af8b-ebf29ed29198-maas-api-openshift-ca" pod "9d51ed2a-f4ab-434a-af8b-ebf29ed29198" (UID: "9d51ed2a-f4ab-434a-af8b-ebf29ed29198"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:45:49.873859 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:49.873842 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee3f4ee2-4fa0-4db0-8bc0-19a591f90f1e-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:49.873930 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:49.873865 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r5k8\" (UniqueName: \"kubernetes.io/projected/5e0e4ee4-329b-4600-89c5-82e6de5b0479-kube-api-access-9r5k8\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:49.873930 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:49.873881 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e0e4ee4-329b-4600-89c5-82e6de5b0479-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:49.873930 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:49.873896 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwslm\" (UniqueName: \"kubernetes.io/projected/ee3f4ee2-4fa0-4db0-8bc0-19a591f90f1e-kube-api-access-lwslm\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:49.873930 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:49.873911 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d51ed2a-f4ab-434a-af8b-ebf29ed29198-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:49.875239 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:49.875218 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d51ed2a-f4ab-434a-af8b-ebf29ed29198-kube-api-access-4zqzd" pod "9d51ed2a-f4ab-434a-af8b-ebf29ed29198" (UID: "9d51ed2a-f4ab-434a-af8b-ebf29ed29198"). InnerVolumeSpecName "kube-api-access-4zqzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:45:49.875304 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:49.875218 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d51ed2a-f4ab-434a-af8b-ebf29ed29198-maas-api-xks-ca" pod "9d51ed2a-f4ab-434a-af8b-ebf29ed29198" (UID: "9d51ed2a-f4ab-434a-af8b-ebf29ed29198"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:45:49.974741 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:49.974708 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d51ed2a-f4ab-434a-af8b-ebf29ed29198-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:49.974741 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:49.974734 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zqzd\" (UniqueName: \"kubernetes.io/projected/9d51ed2a-f4ab-434a-af8b-ebf29ed29198-kube-api-access-4zqzd\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:50.158867 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:50.158775 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:50.523220 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:50.523180 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1184d79b-27xvv" event={"ID":"5e0e4ee4-329b-4600-89c5-82e6de5b0479","Type":"ContainerDied","Data":"37626788a12627c37279e84027d0a13e13dca6e57b40ed2aaacde9149409bc68"} Nov 01 20:45:50.523220 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:50.523207 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1184d79b-27xvv" Nov 01 20:45:50.523468 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:50.523260 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37626788a12627c37279e84027d0a13e13dca6e57b40ed2aaacde9149409bc68" Nov 01 20:45:50.524599 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:50.524575 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-438e9c5e-rt5dz" Nov 01 20:45:50.524747 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:50.524594 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-438e9c5e-rt5dz" event={"ID":"ee3f4ee2-4fa0-4db0-8bc0-19a591f90f1e","Type":"ContainerDied","Data":"862b9b50bcaa725f7ce64f64d865c27cc478c2b329d4222a00533c840a8ee4ff"} Nov 01 20:45:50.524747 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:50.524642 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="862b9b50bcaa725f7ce64f64d865c27cc478c2b329d4222a00533c840a8ee4ff" Nov 01 20:45:50.526029 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:50.525987 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72e83b8-8wq9j" Nov 01 20:45:50.526029 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:50.525987 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72e83b8-8wq9j" event={"ID":"9d51ed2a-f4ab-434a-af8b-ebf29ed29198","Type":"ContainerDied","Data":"3f8ab3c76baa754401af9fc562472f73b72412a496fa038534b078eef79773e3"} Nov 01 20:45:50.526233 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:50.526044 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f8ab3c76baa754401af9fc562472f73b72412a496fa038534b078eef79773e3" Nov 01 20:45:51.999082 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:51.999045 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b"] Nov 01 20:45:51.999606 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:51.999592 4484 cpu_manager.go:414] "containerMap: removing container" podUID="ee3f4ee2-4fa0-4db0-8bc0-19a591f90f1e" containerName="revoke-keys" Nov 01 20:45:51.999651 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:51.999606 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee3f4ee2-4fa0-4db0-8bc0-19a591f90f1e" containerName="revoke-keys" Nov 01 20:45:51.999651 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:51.999624 4484 cpu_manager.go:414] "containerMap: removing container" podUID="9d51ed2a-f4ab-434a-af8b-ebf29ed29198" containerName="revoke-keys" Nov 01 20:45:51.999651 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:51.999629 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d51ed2a-f4ab-434a-af8b-ebf29ed29198" containerName="revoke-keys" Nov 01 20:45:51.999651 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:51.999636 4484 cpu_manager.go:414] "containerMap: removing container" podUID="5e0e4ee4-329b-4600-89c5-82e6de5b0479" containerName="revoke-keys" Nov 01 20:45:51.999651 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:51.999642 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e0e4ee4-329b-4600-89c5-82e6de5b0479" containerName="revoke-keys" Nov 01 20:45:51.999796 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:51.999729 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee3f4ee2-4fa0-4db0-8bc0-19a591f90f1e" containerName="revoke-keys" Nov 01 20:45:51.999796 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:51.999740 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d51ed2a-f4ab-434a-af8b-ebf29ed29198" containerName="revoke-keys" Nov 01 20:45:51.999796 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:51.999748 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e0e4ee4-329b-4600-89c5-82e6de5b0479" containerName="revoke-keys" Nov 01 20:45:52.004318 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:52.004301 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b" Nov 01 20:45:52.063787 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:52.063756 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b"] Nov 01 20:45:52.113073 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:52.113040 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c75a1500-a843-43a3-98f7-a459015090c3-workload-socket\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b\" (UID: \"c75a1500-a843-43a3-98f7-a459015090c3\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b" Nov 01 20:45:52.113250 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:52.113088 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c75a1500-a843-43a3-98f7-a459015090c3-istiod-ca-cert\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b\" (UID: \"c75a1500-a843-43a3-98f7-a459015090c3\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b" Nov 01 20:45:52.113250 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:52.113175 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c75a1500-a843-43a3-98f7-a459015090c3-istio-podinfo\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b\" (UID: \"c75a1500-a843-43a3-98f7-a459015090c3\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b" Nov 01 20:45:52.113250 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:52.113207 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c75a1500-a843-43a3-98f7-a459015090c3-istio-token\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b\" (UID: \"c75a1500-a843-43a3-98f7-a459015090c3\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b" Nov 01 20:45:52.113353 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:52.113296 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c75a1500-a843-43a3-98f7-a459015090c3-workload-certs\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b\" (UID: \"c75a1500-a843-43a3-98f7-a459015090c3\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b" Nov 01 20:45:52.113387 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:52.113376 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c75a1500-a843-43a3-98f7-a459015090c3-istio-envoy\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b\" (UID: \"c75a1500-a843-43a3-98f7-a459015090c3\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b" Nov 01 20:45:52.113452 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:52.113433 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c75a1500-a843-43a3-98f7-a459015090c3-istio-data\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b\" (UID: \"c75a1500-a843-43a3-98f7-a459015090c3\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b" Nov 01 20:45:52.113522 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:52.113494 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c75a1500-a843-43a3-98f7-a459015090c3-credential-socket\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b\" (UID: \"c75a1500-a843-43a3-98f7-a459015090c3\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b" Nov 01 20:45:52.113635 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:52.113563 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fqjb\" (UniqueName: \"kubernetes.io/projected/c75a1500-a843-43a3-98f7-a459015090c3-kube-api-access-5fqjb\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b\" (UID: \"c75a1500-a843-43a3-98f7-a459015090c3\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b" Nov 01 20:45:52.214374 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:52.214330 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c75a1500-a843-43a3-98f7-a459015090c3-istio-envoy\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b\" (UID: \"c75a1500-a843-43a3-98f7-a459015090c3\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b" Nov 01 20:45:52.214586 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:52.214490 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c75a1500-a843-43a3-98f7-a459015090c3-istio-data\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b\" (UID: \"c75a1500-a843-43a3-98f7-a459015090c3\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b" Nov 01 20:45:52.214586 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:52.214520 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c75a1500-a843-43a3-98f7-a459015090c3-credential-socket\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b\" (UID: \"c75a1500-a843-43a3-98f7-a459015090c3\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b" Nov 01 20:45:52.214586 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:52.214538 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fqjb\" (UniqueName: \"kubernetes.io/projected/c75a1500-a843-43a3-98f7-a459015090c3-kube-api-access-5fqjb\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b\" (UID: \"c75a1500-a843-43a3-98f7-a459015090c3\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b" Nov 01 20:45:52.214586 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:52.214584 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c75a1500-a843-43a3-98f7-a459015090c3-workload-socket\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b\" (UID: \"c75a1500-a843-43a3-98f7-a459015090c3\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b" Nov 01 20:45:52.214809 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:52.214630 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c75a1500-a843-43a3-98f7-a459015090c3-istiod-ca-cert\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b\" (UID: \"c75a1500-a843-43a3-98f7-a459015090c3\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b" Nov 01 20:45:52.214809 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:52.214666 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c75a1500-a843-43a3-98f7-a459015090c3-istio-podinfo\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b\" (UID: \"c75a1500-a843-43a3-98f7-a459015090c3\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b" Nov 01 20:45:52.214809 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:52.214691 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c75a1500-a843-43a3-98f7-a459015090c3-istio-token\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b\" (UID: \"c75a1500-a843-43a3-98f7-a459015090c3\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b" Nov 01 20:45:52.214809 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:52.214753 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c75a1500-a843-43a3-98f7-a459015090c3-workload-certs\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b\" (UID: \"c75a1500-a843-43a3-98f7-a459015090c3\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b" Nov 01 20:45:52.215106 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:52.214926 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c75a1500-a843-43a3-98f7-a459015090c3-istio-data\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b\" (UID: \"c75a1500-a843-43a3-98f7-a459015090c3\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b" Nov 01 20:45:52.215106 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:52.214967 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c75a1500-a843-43a3-98f7-a459015090c3-credential-socket\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b\" (UID: \"c75a1500-a843-43a3-98f7-a459015090c3\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b" Nov 01 20:45:52.215106 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:52.215028 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c75a1500-a843-43a3-98f7-a459015090c3-workload-socket\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b\" (UID: \"c75a1500-a843-43a3-98f7-a459015090c3\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b" Nov 01 20:45:52.215257 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:52.215166 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c75a1500-a843-43a3-98f7-a459015090c3-workload-certs\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b\" (UID: \"c75a1500-a843-43a3-98f7-a459015090c3\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b" Nov 01 20:45:52.215481 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:52.215457 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c75a1500-a843-43a3-98f7-a459015090c3-istiod-ca-cert\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b\" (UID: \"c75a1500-a843-43a3-98f7-a459015090c3\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b" Nov 01 20:45:52.216718 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:52.216695 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c75a1500-a843-43a3-98f7-a459015090c3-istio-envoy\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b\" (UID: \"c75a1500-a843-43a3-98f7-a459015090c3\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b" Nov 01 20:45:52.217134 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:52.217118 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c75a1500-a843-43a3-98f7-a459015090c3-istio-podinfo\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b\" (UID: \"c75a1500-a843-43a3-98f7-a459015090c3\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b" Nov 01 20:45:52.272067 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:52.271956 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c75a1500-a843-43a3-98f7-a459015090c3-istio-token\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b\" (UID: \"c75a1500-a843-43a3-98f7-a459015090c3\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b" Nov 01 20:45:52.272205 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:52.272107 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fqjb\" (UniqueName: \"kubernetes.io/projected/c75a1500-a843-43a3-98f7-a459015090c3-kube-api-access-5fqjb\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b\" (UID: \"c75a1500-a843-43a3-98f7-a459015090c3\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b" Nov 01 20:45:52.317843 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:52.317808 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b" Nov 01 20:45:52.746458 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:45:52.746428 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc75a1500_a843_43a3_98f7_a459015090c3.slice/crio-10b510d860c91efb40f500952145912a02dd93b287f4b1bc99c5272ddc94358d WatchSource:0}: Error finding container 10b510d860c91efb40f500952145912a02dd93b287f4b1bc99c5272ddc94358d: Status 404 returned error can't find the container with id 10b510d860c91efb40f500952145912a02dd93b287f4b1bc99c5272ddc94358d Nov 01 20:45:52.746945 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:52.746917 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b"] Nov 01 20:45:52.749122 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:52.749090 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:45:52.749202 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:52.749157 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:45:52.749248 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:52.749201 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:45:53.553127 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:53.553089 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b" event={"ID":"c75a1500-a843-43a3-98f7-a459015090c3","Type":"ContainerStarted","Data":"950f959c5ced7f6cf412c2a0dd0f53561086d99027b5bf5eda837a2be3bf52ab"} Nov 01 20:45:53.553638 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:53.553173 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b" event={"ID":"c75a1500-a843-43a3-98f7-a459015090c3","Type":"ContainerStarted","Data":"10b510d860c91efb40f500952145912a02dd93b287f4b1bc99c5272ddc94358d"} Nov 01 20:45:54.328634 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:54.328601 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b" Nov 01 20:45:54.330367 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:54.330342 4484 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 01 20:45:54.330480 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:54.330405 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b" podUID="c75a1500-a843-43a3-98f7-a459015090c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 01 20:45:55.126022 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:55.125968 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:55.320143 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:55.320105 4484 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 01 20:45:55.320416 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:55.320388 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b" podUID="c75a1500-a843-43a3-98f7-a459015090c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 01 20:45:56.323020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:56.319424 4484 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 01 20:45:56.323020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:56.319505 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b" podUID="c75a1500-a843-43a3-98f7-a459015090c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 01 20:45:57.322506 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:57.322466 4484 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 01 20:45:57.322801 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:57.322773 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b" podUID="c75a1500-a843-43a3-98f7-a459015090c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 01 20:45:58.321502 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:58.321463 4484 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 01 20:45:58.322157 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:58.322120 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b" podUID="c75a1500-a843-43a3-98f7-a459015090c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 01 20:45:59.465472 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:59.465438 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b" Nov 01 20:45:59.466381 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:59.466352 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b" Nov 01 20:45:59.467174 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:59.467154 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b" Nov 01 20:45:59.642017 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:45:59.640167 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b" podStartSLOduration=8.640148795 podStartE2EDuration="8.640148795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:45:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:45:53.783594369 +0000 UTC m=+904.561234504" watchObservedRunningTime="2026-11-01 20:45:59.640148795 +0000 UTC m=+910.417788930" Nov 01 20:46:00.238213 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:00.238146 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:46:00.299145 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:00.298975 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-306bdb4a-v6w48"] Nov 01 20:46:00.307408 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:00.307377 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-306bdb4a-v6w48" Nov 01 20:46:00.350963 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:00.350931 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-306bdb4a-v6w48"] Nov 01 20:46:00.351138 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:00.351065 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8624507-afd3-4878-bbd7-dc386e2cd3ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-306bdb4a-v6w48\" (UID: \"f8624507-afd3-4878-bbd7-dc386e2cd3ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-306bdb4a-v6w48" Nov 01 20:46:00.351200 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:00.351183 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5tc4\" (UniqueName: \"kubernetes.io/projected/f8624507-afd3-4878-bbd7-dc386e2cd3ab-kube-api-access-h5tc4\") pod \"maas-api-revoke-sub-306bdb4a-v6w48\" (UID: \"f8624507-afd3-4878-bbd7-dc386e2cd3ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-306bdb4a-v6w48" Nov 01 20:46:00.351251 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:00.351239 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8624507-afd3-4878-bbd7-dc386e2cd3ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-306bdb4a-v6w48\" (UID: \"f8624507-afd3-4878-bbd7-dc386e2cd3ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-306bdb4a-v6w48" Nov 01 20:46:00.361317 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:00.361292 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:46:00.361525 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:00.361337 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:46:00.361525 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:00.361350 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:46:00.361525 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:00.361440 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 20:46:00.452798 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:00.452761 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8624507-afd3-4878-bbd7-dc386e2cd3ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-306bdb4a-v6w48\" (UID: \"f8624507-afd3-4878-bbd7-dc386e2cd3ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-306bdb4a-v6w48" Nov 01 20:46:00.452949 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:00.452886 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5tc4\" (UniqueName: \"kubernetes.io/projected/f8624507-afd3-4878-bbd7-dc386e2cd3ab-kube-api-access-h5tc4\") pod \"maas-api-revoke-sub-306bdb4a-v6w48\" (UID: \"f8624507-afd3-4878-bbd7-dc386e2cd3ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-306bdb4a-v6w48" Nov 01 20:46:00.452949 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:00.452936 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8624507-afd3-4878-bbd7-dc386e2cd3ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-306bdb4a-v6w48\" (UID: \"f8624507-afd3-4878-bbd7-dc386e2cd3ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-306bdb4a-v6w48" Nov 01 20:46:00.453675 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:00.453645 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8624507-afd3-4878-bbd7-dc386e2cd3ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-306bdb4a-v6w48\" (UID: \"f8624507-afd3-4878-bbd7-dc386e2cd3ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-306bdb4a-v6w48" Nov 01 20:46:00.455255 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:00.455234 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8624507-afd3-4878-bbd7-dc386e2cd3ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-306bdb4a-v6w48\" (UID: \"f8624507-afd3-4878-bbd7-dc386e2cd3ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-306bdb4a-v6w48" Nov 01 20:46:00.525651 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:00.525562 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5tc4\" (UniqueName: \"kubernetes.io/projected/f8624507-afd3-4878-bbd7-dc386e2cd3ab-kube-api-access-h5tc4\") pod \"maas-api-revoke-sub-306bdb4a-v6w48\" (UID: \"f8624507-afd3-4878-bbd7-dc386e2cd3ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-306bdb4a-v6w48" Nov 01 20:46:00.626286 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:00.626254 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-306bdb4a-v6w48" Nov 01 20:46:00.732962 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:00.732926 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b5e8b28-mdkfk"] Nov 01 20:46:00.738210 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:00.738186 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b5e8b28-mdkfk" Nov 01 20:46:00.757009 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:00.756963 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkjwm\" (UniqueName: \"kubernetes.io/projected/df220463-0233-42b1-a927-496af334cd6e-kube-api-access-kkjwm\") pod \"maas-api-revoke-sub-8b5e8b28-mdkfk\" (UID: \"df220463-0233-42b1-a927-496af334cd6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b5e8b28-mdkfk" Nov 01 20:46:00.757163 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:00.757081 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df220463-0233-42b1-a927-496af334cd6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b5e8b28-mdkfk\" (UID: \"df220463-0233-42b1-a927-496af334cd6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b5e8b28-mdkfk" Nov 01 20:46:00.757230 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:00.757178 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df220463-0233-42b1-a927-496af334cd6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b5e8b28-mdkfk\" (UID: \"df220463-0233-42b1-a927-496af334cd6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b5e8b28-mdkfk" Nov 01 20:46:00.799494 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:00.799376 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b5e8b28-mdkfk"] Nov 01 20:46:00.858128 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:00.858096 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df220463-0233-42b1-a927-496af334cd6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b5e8b28-mdkfk\" (UID: \"df220463-0233-42b1-a927-496af334cd6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b5e8b28-mdkfk" Nov 01 20:46:00.858495 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:00.858468 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkjwm\" (UniqueName: \"kubernetes.io/projected/df220463-0233-42b1-a927-496af334cd6e-kube-api-access-kkjwm\") pod \"maas-api-revoke-sub-8b5e8b28-mdkfk\" (UID: \"df220463-0233-42b1-a927-496af334cd6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b5e8b28-mdkfk" Nov 01 20:46:00.858630 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:00.858539 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df220463-0233-42b1-a927-496af334cd6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b5e8b28-mdkfk\" (UID: \"df220463-0233-42b1-a927-496af334cd6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b5e8b28-mdkfk" Nov 01 20:46:00.859258 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:00.859229 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df220463-0233-42b1-a927-496af334cd6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b5e8b28-mdkfk\" (UID: \"df220463-0233-42b1-a927-496af334cd6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b5e8b28-mdkfk" Nov 01 20:46:00.861298 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:00.861277 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df220463-0233-42b1-a927-496af334cd6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b5e8b28-mdkfk\" (UID: \"df220463-0233-42b1-a927-496af334cd6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b5e8b28-mdkfk" Nov 01 20:46:00.915934 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:00.915906 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkjwm\" (UniqueName: \"kubernetes.io/projected/df220463-0233-42b1-a927-496af334cd6e-kube-api-access-kkjwm\") pod \"maas-api-revoke-sub-8b5e8b28-mdkfk\" (UID: \"df220463-0233-42b1-a927-496af334cd6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b5e8b28-mdkfk" Nov 01 20:46:01.004547 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:01.004518 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-306bdb4a-v6w48"] Nov 01 20:46:01.004818 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:46:01.004797 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8624507_afd3_4878_bbd7_dc386e2cd3ab.slice/crio-6d0613af6a5b5f48585da4dbbf766cba6ae123f1edb321ffac49c4624beb1a87 WatchSource:0}: Error finding container 6d0613af6a5b5f48585da4dbbf766cba6ae123f1edb321ffac49c4624beb1a87: Status 404 returned error can't find the container with id 6d0613af6a5b5f48585da4dbbf766cba6ae123f1edb321ffac49c4624beb1a87 Nov 01 20:46:01.057117 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:01.057042 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b5e8b28-mdkfk" Nov 01 20:46:01.173374 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:01.173338 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dcd387e-spkmz"] Nov 01 20:46:01.178770 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:01.178746 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dcd387e-spkmz" Nov 01 20:46:01.233680 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:01.233647 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dcd387e-spkmz"] Nov 01 20:46:01.263415 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:01.263378 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8thc5\" (UniqueName: \"kubernetes.io/projected/33f110a1-18ca-49a7-b518-14f75737f0da-kube-api-access-8thc5\") pod \"maas-api-revoke-sub-2dcd387e-spkmz\" (UID: \"33f110a1-18ca-49a7-b518-14f75737f0da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dcd387e-spkmz" Nov 01 20:46:01.263596 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:01.263428 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33f110a1-18ca-49a7-b518-14f75737f0da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2dcd387e-spkmz\" (UID: \"33f110a1-18ca-49a7-b518-14f75737f0da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dcd387e-spkmz" Nov 01 20:46:01.263685 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:01.263667 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33f110a1-18ca-49a7-b518-14f75737f0da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2dcd387e-spkmz\" (UID: \"33f110a1-18ca-49a7-b518-14f75737f0da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dcd387e-spkmz" Nov 01 20:46:01.365105 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:01.365058 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33f110a1-18ca-49a7-b518-14f75737f0da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2dcd387e-spkmz\" (UID: \"33f110a1-18ca-49a7-b518-14f75737f0da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dcd387e-spkmz" Nov 01 20:46:01.365105 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:01.365122 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8thc5\" (UniqueName: \"kubernetes.io/projected/33f110a1-18ca-49a7-b518-14f75737f0da-kube-api-access-8thc5\") pod \"maas-api-revoke-sub-2dcd387e-spkmz\" (UID: \"33f110a1-18ca-49a7-b518-14f75737f0da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dcd387e-spkmz" Nov 01 20:46:01.365366 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:01.365226 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33f110a1-18ca-49a7-b518-14f75737f0da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2dcd387e-spkmz\" (UID: \"33f110a1-18ca-49a7-b518-14f75737f0da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dcd387e-spkmz" Nov 01 20:46:01.365718 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:01.365697 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33f110a1-18ca-49a7-b518-14f75737f0da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2dcd387e-spkmz\" (UID: \"33f110a1-18ca-49a7-b518-14f75737f0da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dcd387e-spkmz" Nov 01 20:46:01.367637 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:01.367614 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33f110a1-18ca-49a7-b518-14f75737f0da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2dcd387e-spkmz\" (UID: \"33f110a1-18ca-49a7-b518-14f75737f0da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dcd387e-spkmz" Nov 01 20:46:01.422764 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:01.422731 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8thc5\" (UniqueName: \"kubernetes.io/projected/33f110a1-18ca-49a7-b518-14f75737f0da-kube-api-access-8thc5\") pod \"maas-api-revoke-sub-2dcd387e-spkmz\" (UID: \"33f110a1-18ca-49a7-b518-14f75737f0da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dcd387e-spkmz" Nov 01 20:46:01.433554 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:01.433527 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b5e8b28-mdkfk"] Nov 01 20:46:01.434474 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:46:01.434447 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf220463_0233_42b1_a927_496af334cd6e.slice/crio-88c6add9001d2c7960c539b150fa7561484860531fa8be4bf258b3ef9ac763d9 WatchSource:0}: Error finding container 88c6add9001d2c7960c539b150fa7561484860531fa8be4bf258b3ef9ac763d9: Status 404 returned error can't find the container with id 88c6add9001d2c7960c539b150fa7561484860531fa8be4bf258b3ef9ac763d9 Nov 01 20:46:01.495576 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:01.495541 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dcd387e-spkmz" Nov 01 20:46:01.612762 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:01.612681 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-306bdb4a-v6w48" event={"ID":"f8624507-afd3-4878-bbd7-dc386e2cd3ab","Type":"ContainerStarted","Data":"c0c82dfbe9a056ee2ca0e95811cf9e84a9c9eaf1eacf5e61b927880b74dcff46"} Nov 01 20:46:01.612762 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:01.612746 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-306bdb4a-v6w48" event={"ID":"f8624507-afd3-4878-bbd7-dc386e2cd3ab","Type":"ContainerStarted","Data":"6d0613af6a5b5f48585da4dbbf766cba6ae123f1edb321ffac49c4624beb1a87"} Nov 01 20:46:01.614163 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:01.614137 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b5e8b28-mdkfk" event={"ID":"df220463-0233-42b1-a927-496af334cd6e","Type":"ContainerStarted","Data":"88c6add9001d2c7960c539b150fa7561484860531fa8be4bf258b3ef9ac763d9"} Nov 01 20:46:01.727510 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:01.727463 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-306bdb4a-v6w48" podStartSLOduration=1.727448329 podStartE2EDuration="1.727448329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:46:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:46:01.726919869 +0000 UTC m=+912.504560003" watchObservedRunningTime="2026-11-01 20:46:01.727448329 +0000 UTC m=+912.505088493" Nov 01 20:46:01.869981 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:01.869952 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dcd387e-spkmz"] Nov 01 20:46:01.870692 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:46:01.870657 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33f110a1_18ca_49a7_b518_14f75737f0da.slice/crio-12a5ee610df8b81db22353746576a164511208319a653a55af4c34676b4b2191 WatchSource:0}: Error finding container 12a5ee610df8b81db22353746576a164511208319a653a55af4c34676b4b2191: Status 404 returned error can't find the container with id 12a5ee610df8b81db22353746576a164511208319a653a55af4c34676b4b2191 Nov 01 20:46:02.074932 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:02.074895 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:46:02.630879 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:02.630833 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dcd387e-spkmz" event={"ID":"33f110a1-18ca-49a7-b518-14f75737f0da","Type":"ContainerStarted","Data":"0a6ae19c7585f2c47111f0bafb528a38cb47d422ab4f66b96f6eeb4b5634ead5"} Nov 01 20:46:02.631360 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:02.630939 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dcd387e-spkmz" event={"ID":"33f110a1-18ca-49a7-b518-14f75737f0da","Type":"ContainerStarted","Data":"12a5ee610df8b81db22353746576a164511208319a653a55af4c34676b4b2191"} Nov 01 20:46:02.632660 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:02.632630 4484 generic.go:356] "Generic (PLEG): container finished" podID="f8624507-afd3-4878-bbd7-dc386e2cd3ab" containerID="c0c82dfbe9a056ee2ca0e95811cf9e84a9c9eaf1eacf5e61b927880b74dcff46" exitCode=0 Nov 01 20:46:02.632808 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:02.632713 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-306bdb4a-v6w48" event={"ID":"f8624507-afd3-4878-bbd7-dc386e2cd3ab","Type":"ContainerDied","Data":"c0c82dfbe9a056ee2ca0e95811cf9e84a9c9eaf1eacf5e61b927880b74dcff46"} Nov 01 20:46:02.634478 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:02.634455 4484 generic.go:356] "Generic (PLEG): container finished" podID="df220463-0233-42b1-a927-496af334cd6e" containerID="aa56e507833f602332e90ff8bf5ef525aee037a2f1a459b708e8f9e769e2099a" exitCode=0 Nov 01 20:46:02.634596 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:02.634539 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b5e8b28-mdkfk" event={"ID":"df220463-0233-42b1-a927-496af334cd6e","Type":"ContainerDied","Data":"aa56e507833f602332e90ff8bf5ef525aee037a2f1a459b708e8f9e769e2099a"} Nov 01 20:46:02.747991 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:02.747936 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dcd387e-spkmz" podStartSLOduration=1.747921024 podStartE2EDuration="1.747921024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:46:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:46:02.745250454 +0000 UTC m=+913.522890580" watchObservedRunningTime="2026-11-01 20:46:02.747921024 +0000 UTC m=+913.525561159" Nov 01 20:46:03.642652 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:03.642605 4484 generic.go:356] "Generic (PLEG): container finished" podID="33f110a1-18ca-49a7-b518-14f75737f0da" containerID="0a6ae19c7585f2c47111f0bafb528a38cb47d422ab4f66b96f6eeb4b5634ead5" exitCode=0 Nov 01 20:46:03.642652 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:03.642643 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dcd387e-spkmz" event={"ID":"33f110a1-18ca-49a7-b518-14f75737f0da","Type":"ContainerDied","Data":"0a6ae19c7585f2c47111f0bafb528a38cb47d422ab4f66b96f6eeb4b5634ead5"} Nov 01 20:46:03.852641 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:03.852612 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b5e8b28-mdkfk" Nov 01 20:46:03.856778 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:03.856756 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-306bdb4a-v6w48" Nov 01 20:46:03.896270 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:03.896165 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df220463-0233-42b1-a927-496af334cd6e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df220463-0233-42b1-a927-496af334cd6e-maas-api-xks-ca\") pod \"df220463-0233-42b1-a927-496af334cd6e\" (UID: \"df220463-0233-42b1-a927-496af334cd6e\") " Nov 01 20:46:03.896270 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:03.896224 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8624507-afd3-4878-bbd7-dc386e2cd3ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8624507-afd3-4878-bbd7-dc386e2cd3ab-maas-api-xks-ca\") pod \"f8624507-afd3-4878-bbd7-dc386e2cd3ab\" (UID: \"f8624507-afd3-4878-bbd7-dc386e2cd3ab\") " Nov 01 20:46:03.896487 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:03.896281 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df220463-0233-42b1-a927-496af334cd6e-kube-api-access-kkjwm\" (UniqueName: \"kubernetes.io/projected/df220463-0233-42b1-a927-496af334cd6e-kube-api-access-kkjwm\") pod \"df220463-0233-42b1-a927-496af334cd6e\" (UID: \"df220463-0233-42b1-a927-496af334cd6e\") " Nov 01 20:46:03.896487 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:03.896368 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df220463-0233-42b1-a927-496af334cd6e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df220463-0233-42b1-a927-496af334cd6e-maas-api-openshift-ca\") pod \"df220463-0233-42b1-a927-496af334cd6e\" (UID: \"df220463-0233-42b1-a927-496af334cd6e\") " Nov 01 20:46:03.896487 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:03.896407 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8624507-afd3-4878-bbd7-dc386e2cd3ab-kube-api-access-h5tc4\" (UniqueName: \"kubernetes.io/projected/f8624507-afd3-4878-bbd7-dc386e2cd3ab-kube-api-access-h5tc4\") pod \"f8624507-afd3-4878-bbd7-dc386e2cd3ab\" (UID: \"f8624507-afd3-4878-bbd7-dc386e2cd3ab\") " Nov 01 20:46:03.896664 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:03.896501 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8624507-afd3-4878-bbd7-dc386e2cd3ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8624507-afd3-4878-bbd7-dc386e2cd3ab-maas-api-openshift-ca\") pod \"f8624507-afd3-4878-bbd7-dc386e2cd3ab\" (UID: \"f8624507-afd3-4878-bbd7-dc386e2cd3ab\") " Nov 01 20:46:03.897284 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:03.896907 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df220463-0233-42b1-a927-496af334cd6e-maas-api-openshift-ca" pod "df220463-0233-42b1-a927-496af334cd6e" (UID: "df220463-0233-42b1-a927-496af334cd6e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:46:03.897284 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:03.897231 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8624507-afd3-4878-bbd7-dc386e2cd3ab-maas-api-openshift-ca" pod "f8624507-afd3-4878-bbd7-dc386e2cd3ab" (UID: "f8624507-afd3-4878-bbd7-dc386e2cd3ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:46:03.901065 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:03.901028 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df220463-0233-42b1-a927-496af334cd6e-kube-api-access-kkjwm" pod "df220463-0233-42b1-a927-496af334cd6e" (UID: "df220463-0233-42b1-a927-496af334cd6e"). InnerVolumeSpecName "kube-api-access-kkjwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:46:03.901301 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:03.901185 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df220463-0233-42b1-a927-496af334cd6e-maas-api-xks-ca" pod "df220463-0233-42b1-a927-496af334cd6e" (UID: "df220463-0233-42b1-a927-496af334cd6e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:46:03.901301 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:03.901264 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8624507-afd3-4878-bbd7-dc386e2cd3ab-maas-api-xks-ca" pod "f8624507-afd3-4878-bbd7-dc386e2cd3ab" (UID: "f8624507-afd3-4878-bbd7-dc386e2cd3ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:46:03.902478 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:03.902451 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8624507-afd3-4878-bbd7-dc386e2cd3ab-kube-api-access-h5tc4" pod "f8624507-afd3-4878-bbd7-dc386e2cd3ab" (UID: "f8624507-afd3-4878-bbd7-dc386e2cd3ab"). InnerVolumeSpecName "kube-api-access-h5tc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:46:03.997989 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:03.997951 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8624507-afd3-4878-bbd7-dc386e2cd3ab-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:46:03.997989 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:03.997986 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkjwm\" (UniqueName: \"kubernetes.io/projected/df220463-0233-42b1-a927-496af334cd6e-kube-api-access-kkjwm\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:46:03.997989 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:03.998010 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df220463-0233-42b1-a927-496af334cd6e-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:46:03.998302 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:03.998020 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5tc4\" (UniqueName: \"kubernetes.io/projected/f8624507-afd3-4878-bbd7-dc386e2cd3ab-kube-api-access-h5tc4\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:46:03.998302 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:03.998029 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8624507-afd3-4878-bbd7-dc386e2cd3ab-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:46:03.998302 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:03.998038 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df220463-0233-42b1-a927-496af334cd6e-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:46:04.165886 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:04.165801 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c05c922a-cgffx"] Nov 01 20:46:04.166387 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:04.166372 4484 cpu_manager.go:414] "containerMap: removing container" podUID="f8624507-afd3-4878-bbd7-dc386e2cd3ab" containerName="revoke-keys" Nov 01 20:46:04.166435 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:04.166390 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8624507-afd3-4878-bbd7-dc386e2cd3ab" containerName="revoke-keys" Nov 01 20:46:04.166471 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:04.166439 4484 cpu_manager.go:414] "containerMap: removing container" podUID="df220463-0233-42b1-a927-496af334cd6e" containerName="revoke-keys" Nov 01 20:46:04.166471 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:04.166445 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df220463-0233-42b1-a927-496af334cd6e" containerName="revoke-keys" Nov 01 20:46:04.166532 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:04.166522 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="df220463-0233-42b1-a927-496af334cd6e" containerName="revoke-keys" Nov 01 20:46:04.166532 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:04.166531 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8624507-afd3-4878-bbd7-dc386e2cd3ab" containerName="revoke-keys" Nov 01 20:46:04.170261 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:04.170242 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c05c922a-cgffx" Nov 01 20:46:04.202731 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:04.202695 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46c9582c-b073-4fb8-9b12-705307d2f82a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c05c922a-cgffx\" (UID: \"46c9582c-b073-4fb8-9b12-705307d2f82a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c05c922a-cgffx" Nov 01 20:46:04.202932 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:04.202746 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5kx7\" (UniqueName: \"kubernetes.io/projected/46c9582c-b073-4fb8-9b12-705307d2f82a-kube-api-access-w5kx7\") pod \"maas-api-revoke-sub-c05c922a-cgffx\" (UID: \"46c9582c-b073-4fb8-9b12-705307d2f82a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c05c922a-cgffx" Nov 01 20:46:04.203045 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:04.202941 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46c9582c-b073-4fb8-9b12-705307d2f82a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c05c922a-cgffx\" (UID: \"46c9582c-b073-4fb8-9b12-705307d2f82a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c05c922a-cgffx" Nov 01 20:46:04.226104 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:04.226070 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c05c922a-cgffx"] Nov 01 20:46:04.303683 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:04.303649 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5kx7\" (UniqueName: \"kubernetes.io/projected/46c9582c-b073-4fb8-9b12-705307d2f82a-kube-api-access-w5kx7\") pod \"maas-api-revoke-sub-c05c922a-cgffx\" (UID: \"46c9582c-b073-4fb8-9b12-705307d2f82a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c05c922a-cgffx" Nov 01 20:46:04.303866 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:04.303742 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46c9582c-b073-4fb8-9b12-705307d2f82a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c05c922a-cgffx\" (UID: \"46c9582c-b073-4fb8-9b12-705307d2f82a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c05c922a-cgffx" Nov 01 20:46:04.303866 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:04.303827 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46c9582c-b073-4fb8-9b12-705307d2f82a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c05c922a-cgffx\" (UID: \"46c9582c-b073-4fb8-9b12-705307d2f82a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c05c922a-cgffx" Nov 01 20:46:04.304472 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:04.304449 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46c9582c-b073-4fb8-9b12-705307d2f82a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c05c922a-cgffx\" (UID: \"46c9582c-b073-4fb8-9b12-705307d2f82a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c05c922a-cgffx" Nov 01 20:46:04.306163 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:04.306146 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46c9582c-b073-4fb8-9b12-705307d2f82a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c05c922a-cgffx\" (UID: \"46c9582c-b073-4fb8-9b12-705307d2f82a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c05c922a-cgffx" Nov 01 20:46:04.361301 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:04.361269 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5kx7\" (UniqueName: \"kubernetes.io/projected/46c9582c-b073-4fb8-9b12-705307d2f82a-kube-api-access-w5kx7\") pod \"maas-api-revoke-sub-c05c922a-cgffx\" (UID: \"46c9582c-b073-4fb8-9b12-705307d2f82a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c05c922a-cgffx" Nov 01 20:46:04.483619 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:04.483535 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c05c922a-cgffx" Nov 01 20:46:04.649534 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:04.649503 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-306bdb4a-v6w48" Nov 01 20:46:04.649947 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:04.649544 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-306bdb4a-v6w48" event={"ID":"f8624507-afd3-4878-bbd7-dc386e2cd3ab","Type":"ContainerDied","Data":"6d0613af6a5b5f48585da4dbbf766cba6ae123f1edb321ffac49c4624beb1a87"} Nov 01 20:46:04.649947 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:04.649605 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d0613af6a5b5f48585da4dbbf766cba6ae123f1edb321ffac49c4624beb1a87" Nov 01 20:46:04.650972 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:04.650949 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b5e8b28-mdkfk" Nov 01 20:46:04.651131 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:04.650988 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b5e8b28-mdkfk" event={"ID":"df220463-0233-42b1-a927-496af334cd6e","Type":"ContainerDied","Data":"88c6add9001d2c7960c539b150fa7561484860531fa8be4bf258b3ef9ac763d9"} Nov 01 20:46:04.651131 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:04.651040 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88c6add9001d2c7960c539b150fa7561484860531fa8be4bf258b3ef9ac763d9" Nov 01 20:46:04.781476 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:04.781449 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dcd387e-spkmz" Nov 01 20:46:04.810547 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:04.810517 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33f110a1-18ca-49a7-b518-14f75737f0da-kube-api-access-8thc5\" (UniqueName: \"kubernetes.io/projected/33f110a1-18ca-49a7-b518-14f75737f0da-kube-api-access-8thc5\") pod \"33f110a1-18ca-49a7-b518-14f75737f0da\" (UID: \"33f110a1-18ca-49a7-b518-14f75737f0da\") " Nov 01 20:46:04.810771 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:04.810579 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33f110a1-18ca-49a7-b518-14f75737f0da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33f110a1-18ca-49a7-b518-14f75737f0da-maas-api-openshift-ca\") pod \"33f110a1-18ca-49a7-b518-14f75737f0da\" (UID: \"33f110a1-18ca-49a7-b518-14f75737f0da\") " Nov 01 20:46:04.810771 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:04.810660 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33f110a1-18ca-49a7-b518-14f75737f0da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33f110a1-18ca-49a7-b518-14f75737f0da-maas-api-xks-ca\") pod \"33f110a1-18ca-49a7-b518-14f75737f0da\" (UID: \"33f110a1-18ca-49a7-b518-14f75737f0da\") " Nov 01 20:46:04.810952 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:04.810929 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33f110a1-18ca-49a7-b518-14f75737f0da-maas-api-openshift-ca" pod "33f110a1-18ca-49a7-b518-14f75737f0da" (UID: "33f110a1-18ca-49a7-b518-14f75737f0da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:46:04.811258 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:04.811237 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33f110a1-18ca-49a7-b518-14f75737f0da-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:46:04.812666 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:04.812642 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33f110a1-18ca-49a7-b518-14f75737f0da-maas-api-xks-ca" pod "33f110a1-18ca-49a7-b518-14f75737f0da" (UID: "33f110a1-18ca-49a7-b518-14f75737f0da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:46:04.812782 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:04.812761 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33f110a1-18ca-49a7-b518-14f75737f0da-kube-api-access-8thc5" pod "33f110a1-18ca-49a7-b518-14f75737f0da" (UID: "33f110a1-18ca-49a7-b518-14f75737f0da"). InnerVolumeSpecName "kube-api-access-8thc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:46:04.875682 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:04.875645 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c05c922a-cgffx"] Nov 01 20:46:04.876070 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:46:04.876045 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46c9582c_b073_4fb8_9b12_705307d2f82a.slice/crio-cf2813cecf11e592d1892df432494f295dead12536b0253c3aab7a656033756b WatchSource:0}: Error finding container cf2813cecf11e592d1892df432494f295dead12536b0253c3aab7a656033756b: Status 404 returned error can't find the container with id cf2813cecf11e592d1892df432494f295dead12536b0253c3aab7a656033756b Nov 01 20:46:04.913646 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:04.913618 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33f110a1-18ca-49a7-b518-14f75737f0da-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:46:04.913646 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:04.913646 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8thc5\" (UniqueName: \"kubernetes.io/projected/33f110a1-18ca-49a7-b518-14f75737f0da-kube-api-access-8thc5\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:46:05.086311 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:05.086192 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:46:05.665728 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:05.665633 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dcd387e-spkmz" event={"ID":"33f110a1-18ca-49a7-b518-14f75737f0da","Type":"ContainerDied","Data":"12a5ee610df8b81db22353746576a164511208319a653a55af4c34676b4b2191"} Nov 01 20:46:05.665728 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:05.665714 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12a5ee610df8b81db22353746576a164511208319a653a55af4c34676b4b2191" Nov 01 20:46:05.665728 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:05.665650 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dcd387e-spkmz" Nov 01 20:46:05.667552 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:05.667523 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c05c922a-cgffx" event={"ID":"46c9582c-b073-4fb8-9b12-705307d2f82a","Type":"ContainerStarted","Data":"3186ba9aef3a4c6eaf742b00585c9f27a3764744dec49851021eacd3dcb9e2b7"} Nov 01 20:46:05.667679 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:05.667573 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c05c922a-cgffx" event={"ID":"46c9582c-b073-4fb8-9b12-705307d2f82a","Type":"ContainerStarted","Data":"cf2813cecf11e592d1892df432494f295dead12536b0253c3aab7a656033756b"} Nov 01 20:46:05.783946 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:05.783886 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c05c922a-cgffx" podStartSLOduration=1.7838720000000001 podStartE2EDuration="1.783872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:46:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:46:05.78175489 +0000 UTC m=+916.559395025" watchObservedRunningTime="2026-11-01 20:46:05.783872 +0000 UTC m=+916.561512151" Nov 01 20:46:06.722020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:06.719377 4484 generic.go:356] "Generic (PLEG): container finished" podID="46c9582c-b073-4fb8-9b12-705307d2f82a" containerID="3186ba9aef3a4c6eaf742b00585c9f27a3764744dec49851021eacd3dcb9e2b7" exitCode=0 Nov 01 20:46:06.722020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:06.719439 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c05c922a-cgffx" event={"ID":"46c9582c-b073-4fb8-9b12-705307d2f82a","Type":"ContainerDied","Data":"3186ba9aef3a4c6eaf742b00585c9f27a3764744dec49851021eacd3dcb9e2b7"} Nov 01 20:46:08.257020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:08.251176 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c05c922a-cgffx" Nov 01 20:46:08.391794 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:08.391199 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46c9582c-b073-4fb8-9b12-705307d2f82a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46c9582c-b073-4fb8-9b12-705307d2f82a-maas-api-openshift-ca\") pod \"46c9582c-b073-4fb8-9b12-705307d2f82a\" (UID: \"46c9582c-b073-4fb8-9b12-705307d2f82a\") " Nov 01 20:46:08.392077 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:08.392053 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46c9582c-b073-4fb8-9b12-705307d2f82a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46c9582c-b073-4fb8-9b12-705307d2f82a-maas-api-xks-ca\") pod \"46c9582c-b073-4fb8-9b12-705307d2f82a\" (UID: \"46c9582c-b073-4fb8-9b12-705307d2f82a\") " Nov 01 20:46:08.392294 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:08.392273 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46c9582c-b073-4fb8-9b12-705307d2f82a-kube-api-access-w5kx7\" (UniqueName: \"kubernetes.io/projected/46c9582c-b073-4fb8-9b12-705307d2f82a-kube-api-access-w5kx7\") pod \"46c9582c-b073-4fb8-9b12-705307d2f82a\" (UID: \"46c9582c-b073-4fb8-9b12-705307d2f82a\") " Nov 01 20:46:08.394051 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:08.394025 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46c9582c-b073-4fb8-9b12-705307d2f82a-maas-api-openshift-ca" pod "46c9582c-b073-4fb8-9b12-705307d2f82a" (UID: "46c9582c-b073-4fb8-9b12-705307d2f82a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:46:08.398593 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:08.398547 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46c9582c-b073-4fb8-9b12-705307d2f82a-kube-api-access-w5kx7" pod "46c9582c-b073-4fb8-9b12-705307d2f82a" (UID: "46c9582c-b073-4fb8-9b12-705307d2f82a"). InnerVolumeSpecName "kube-api-access-w5kx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:46:08.400383 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:08.400354 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46c9582c-b073-4fb8-9b12-705307d2f82a-maas-api-xks-ca" pod "46c9582c-b073-4fb8-9b12-705307d2f82a" (UID: "46c9582c-b073-4fb8-9b12-705307d2f82a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:46:08.493899 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:08.493856 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46c9582c-b073-4fb8-9b12-705307d2f82a-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:46:08.493899 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:08.493901 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46c9582c-b073-4fb8-9b12-705307d2f82a-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:46:08.494188 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:08.493916 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5kx7\" (UniqueName: \"kubernetes.io/projected/46c9582c-b073-4fb8-9b12-705307d2f82a-kube-api-access-w5kx7\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:46:08.736653 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:08.736576 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c05c922a-cgffx" Nov 01 20:46:08.736802 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:08.736574 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c05c922a-cgffx" event={"ID":"46c9582c-b073-4fb8-9b12-705307d2f82a","Type":"ContainerDied","Data":"cf2813cecf11e592d1892df432494f295dead12536b0253c3aab7a656033756b"} Nov 01 20:46:08.736802 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:08.736700 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf2813cecf11e592d1892df432494f295dead12536b0253c3aab7a656033756b" Nov 01 20:46:09.990344 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:09.989107 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:46:11.036399 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:11.036357 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bcd2422-wr76f"] Nov 01 20:46:11.036970 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:11.036956 4484 cpu_manager.go:414] "containerMap: removing container" podUID="46c9582c-b073-4fb8-9b12-705307d2f82a" containerName="revoke-keys" Nov 01 20:46:11.037037 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:11.036971 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46c9582c-b073-4fb8-9b12-705307d2f82a" containerName="revoke-keys" Nov 01 20:46:11.037037 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:11.036985 4484 cpu_manager.go:414] "containerMap: removing container" podUID="33f110a1-18ca-49a7-b518-14f75737f0da" containerName="revoke-keys" Nov 01 20:46:11.037037 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:11.036990 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33f110a1-18ca-49a7-b518-14f75737f0da" containerName="revoke-keys" Nov 01 20:46:11.037148 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:11.037134 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="33f110a1-18ca-49a7-b518-14f75737f0da" containerName="revoke-keys" Nov 01 20:46:11.037182 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:11.037154 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="46c9582c-b073-4fb8-9b12-705307d2f82a" containerName="revoke-keys" Nov 01 20:46:11.040750 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:11.040734 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bcd2422-wr76f" Nov 01 20:46:11.096711 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:11.096677 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bcd2422-wr76f"] Nov 01 20:46:11.123652 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:11.123619 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb324776-9df4-47c0-911c-473b3e74f259-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bcd2422-wr76f\" (UID: \"bb324776-9df4-47c0-911c-473b3e74f259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bcd2422-wr76f" Nov 01 20:46:11.123859 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:11.123671 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fw8n\" (UniqueName: \"kubernetes.io/projected/bb324776-9df4-47c0-911c-473b3e74f259-kube-api-access-2fw8n\") pod \"maas-api-revoke-sub-0bcd2422-wr76f\" (UID: \"bb324776-9df4-47c0-911c-473b3e74f259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bcd2422-wr76f" Nov 01 20:46:11.123859 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:11.123828 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb324776-9df4-47c0-911c-473b3e74f259-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bcd2422-wr76f\" (UID: \"bb324776-9df4-47c0-911c-473b3e74f259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bcd2422-wr76f" Nov 01 20:46:11.143555 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:11.143519 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:46:11.143555 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:11.143562 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:46:11.143765 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:11.143625 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 20:46:11.145106 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:11.145089 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:46:11.225544 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:11.225505 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb324776-9df4-47c0-911c-473b3e74f259-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bcd2422-wr76f\" (UID: \"bb324776-9df4-47c0-911c-473b3e74f259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bcd2422-wr76f" Nov 01 20:46:11.225733 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:11.225621 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb324776-9df4-47c0-911c-473b3e74f259-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bcd2422-wr76f\" (UID: \"bb324776-9df4-47c0-911c-473b3e74f259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bcd2422-wr76f" Nov 01 20:46:11.225733 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:11.225663 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fw8n\" (UniqueName: \"kubernetes.io/projected/bb324776-9df4-47c0-911c-473b3e74f259-kube-api-access-2fw8n\") pod \"maas-api-revoke-sub-0bcd2422-wr76f\" (UID: \"bb324776-9df4-47c0-911c-473b3e74f259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bcd2422-wr76f" Nov 01 20:46:11.226224 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:11.226205 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb324776-9df4-47c0-911c-473b3e74f259-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bcd2422-wr76f\" (UID: \"bb324776-9df4-47c0-911c-473b3e74f259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bcd2422-wr76f" Nov 01 20:46:11.228055 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:11.228033 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb324776-9df4-47c0-911c-473b3e74f259-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bcd2422-wr76f\" (UID: \"bb324776-9df4-47c0-911c-473b3e74f259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bcd2422-wr76f" Nov 01 20:46:11.283420 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:11.283389 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fw8n\" (UniqueName: \"kubernetes.io/projected/bb324776-9df4-47c0-911c-473b3e74f259-kube-api-access-2fw8n\") pod \"maas-api-revoke-sub-0bcd2422-wr76f\" (UID: \"bb324776-9df4-47c0-911c-473b3e74f259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bcd2422-wr76f" Nov 01 20:46:11.354238 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:11.354143 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bcd2422-wr76f" Nov 01 20:46:11.734561 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:11.734531 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bcd2422-wr76f"] Nov 01 20:46:11.735156 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:46:11.735124 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb324776_9df4_47c0_911c_473b3e74f259.slice/crio-e38893fd8f07825d5eb6ea185f73b6ab5544767e200ac6600a7d418f42d75395 WatchSource:0}: Error finding container e38893fd8f07825d5eb6ea185f73b6ab5544767e200ac6600a7d418f42d75395: Status 404 returned error can't find the container with id e38893fd8f07825d5eb6ea185f73b6ab5544767e200ac6600a7d418f42d75395 Nov 01 20:46:11.755148 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:11.755121 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bcd2422-wr76f" event={"ID":"bb324776-9df4-47c0-911c-473b3e74f259","Type":"ContainerStarted","Data":"e38893fd8f07825d5eb6ea185f73b6ab5544767e200ac6600a7d418f42d75395"} Nov 01 20:46:12.767461 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:12.767422 4484 generic.go:356] "Generic (PLEG): container finished" podID="bb324776-9df4-47c0-911c-473b3e74f259" containerID="71d1ec2ecd53dfb86bd46233a4add5b92659bec4f6462eb5952905fdfc767870" exitCode=0 Nov 01 20:46:12.767952 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:12.767483 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bcd2422-wr76f" event={"ID":"bb324776-9df4-47c0-911c-473b3e74f259","Type":"ContainerDied","Data":"71d1ec2ecd53dfb86bd46233a4add5b92659bec4f6462eb5952905fdfc767870"} Nov 01 20:46:13.932539 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:13.932514 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bcd2422-wr76f" Nov 01 20:46:13.958675 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:13.958646 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb324776-9df4-47c0-911c-473b3e74f259-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb324776-9df4-47c0-911c-473b3e74f259-maas-api-xks-ca\") pod \"bb324776-9df4-47c0-911c-473b3e74f259\" (UID: \"bb324776-9df4-47c0-911c-473b3e74f259\") " Nov 01 20:46:13.958831 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:13.958799 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb324776-9df4-47c0-911c-473b3e74f259-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb324776-9df4-47c0-911c-473b3e74f259-maas-api-openshift-ca\") pod \"bb324776-9df4-47c0-911c-473b3e74f259\" (UID: \"bb324776-9df4-47c0-911c-473b3e74f259\") " Nov 01 20:46:13.958893 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:13.958865 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb324776-9df4-47c0-911c-473b3e74f259-kube-api-access-2fw8n\" (UniqueName: \"kubernetes.io/projected/bb324776-9df4-47c0-911c-473b3e74f259-kube-api-access-2fw8n\") pod \"bb324776-9df4-47c0-911c-473b3e74f259\" (UID: \"bb324776-9df4-47c0-911c-473b3e74f259\") " Nov 01 20:46:13.959183 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:13.959158 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb324776-9df4-47c0-911c-473b3e74f259-maas-api-openshift-ca" pod "bb324776-9df4-47c0-911c-473b3e74f259" (UID: "bb324776-9df4-47c0-911c-473b3e74f259"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:46:13.959404 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:13.959386 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb324776-9df4-47c0-911c-473b3e74f259-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:46:13.960932 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:13.960909 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb324776-9df4-47c0-911c-473b3e74f259-maas-api-xks-ca" pod "bb324776-9df4-47c0-911c-473b3e74f259" (UID: "bb324776-9df4-47c0-911c-473b3e74f259"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:46:13.961290 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:13.961268 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb324776-9df4-47c0-911c-473b3e74f259-kube-api-access-2fw8n" pod "bb324776-9df4-47c0-911c-473b3e74f259" (UID: "bb324776-9df4-47c0-911c-473b3e74f259"). InnerVolumeSpecName "kube-api-access-2fw8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:46:14.060378 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:14.060304 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fw8n\" (UniqueName: \"kubernetes.io/projected/bb324776-9df4-47c0-911c-473b3e74f259-kube-api-access-2fw8n\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:46:14.060378 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:14.060341 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb324776-9df4-47c0-911c-473b3e74f259-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:46:14.721194 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:14.721161 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:46:14.790329 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:14.790296 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bcd2422-wr76f" Nov 01 20:46:14.790535 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:14.790325 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bcd2422-wr76f" event={"ID":"bb324776-9df4-47c0-911c-473b3e74f259","Type":"ContainerDied","Data":"e38893fd8f07825d5eb6ea185f73b6ab5544767e200ac6600a7d418f42d75395"} Nov 01 20:46:14.790535 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:14.790378 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e38893fd8f07825d5eb6ea185f73b6ab5544767e200ac6600a7d418f42d75395" Nov 01 20:46:15.765859 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:46:15.765822 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 20:46:16.981169 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:16.981135 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54db2327-5gml4"] Nov 01 20:46:16.981716 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:16.981702 4484 cpu_manager.go:414] "containerMap: removing container" podUID="bb324776-9df4-47c0-911c-473b3e74f259" containerName="revoke-keys" Nov 01 20:46:16.981773 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:16.981717 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb324776-9df4-47c0-911c-473b3e74f259" containerName="revoke-keys" Nov 01 20:46:16.981816 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:16.981810 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb324776-9df4-47c0-911c-473b3e74f259" containerName="revoke-keys" Nov 01 20:46:16.985464 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:16.985446 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54db2327-5gml4" Nov 01 20:46:17.041413 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:17.041375 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54db2327-5gml4"] Nov 01 20:46:17.096900 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:17.096334 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:46:17.096900 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:17.096350 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:46:17.096900 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:17.096405 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 20:46:17.096900 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:17.096728 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be70021e-7901-455b-985a-78fb5f4b66e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54db2327-5gml4\" (UID: \"be70021e-7901-455b-985a-78fb5f4b66e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54db2327-5gml4" Nov 01 20:46:17.096900 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:17.096827 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzfds\" (UniqueName: \"kubernetes.io/projected/be70021e-7901-455b-985a-78fb5f4b66e7-kube-api-access-vzfds\") pod \"maas-api-revoke-sub-54db2327-5gml4\" (UID: \"be70021e-7901-455b-985a-78fb5f4b66e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54db2327-5gml4" Nov 01 20:46:17.096900 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:17.096828 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:46:17.097240 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:17.096970 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be70021e-7901-455b-985a-78fb5f4b66e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54db2327-5gml4\" (UID: \"be70021e-7901-455b-985a-78fb5f4b66e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54db2327-5gml4" Nov 01 20:46:17.143620 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:17.143583 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:46:17.198544 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:17.198506 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be70021e-7901-455b-985a-78fb5f4b66e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54db2327-5gml4\" (UID: \"be70021e-7901-455b-985a-78fb5f4b66e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54db2327-5gml4" Nov 01 20:46:17.198726 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:17.198650 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzfds\" (UniqueName: \"kubernetes.io/projected/be70021e-7901-455b-985a-78fb5f4b66e7-kube-api-access-vzfds\") pod \"maas-api-revoke-sub-54db2327-5gml4\" (UID: \"be70021e-7901-455b-985a-78fb5f4b66e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54db2327-5gml4" Nov 01 20:46:17.198726 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:17.198704 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be70021e-7901-455b-985a-78fb5f4b66e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54db2327-5gml4\" (UID: \"be70021e-7901-455b-985a-78fb5f4b66e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54db2327-5gml4" Nov 01 20:46:17.199251 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:17.199229 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be70021e-7901-455b-985a-78fb5f4b66e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54db2327-5gml4\" (UID: \"be70021e-7901-455b-985a-78fb5f4b66e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54db2327-5gml4" Nov 01 20:46:17.201119 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:17.201096 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be70021e-7901-455b-985a-78fb5f4b66e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54db2327-5gml4\" (UID: \"be70021e-7901-455b-985a-78fb5f4b66e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54db2327-5gml4" Nov 01 20:46:17.257457 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:17.257416 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzfds\" (UniqueName: \"kubernetes.io/projected/be70021e-7901-455b-985a-78fb5f4b66e7-kube-api-access-vzfds\") pod \"maas-api-revoke-sub-54db2327-5gml4\" (UID: \"be70021e-7901-455b-985a-78fb5f4b66e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54db2327-5gml4" Nov 01 20:46:17.299058 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:17.298986 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:46:17.301987 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:17.301967 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54db2327-5gml4" Nov 01 20:46:17.442057 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:17.442017 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9279a206-p7mq2"] Nov 01 20:46:17.447247 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:17.447226 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9279a206-p7mq2" Nov 01 20:46:17.501622 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:17.501592 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk5ks\" (UniqueName: \"kubernetes.io/projected/8bc83511-e182-4a03-97d1-376fed073d0e-kube-api-access-jk5ks\") pod \"maas-api-revoke-sub-9279a206-p7mq2\" (UID: \"8bc83511-e182-4a03-97d1-376fed073d0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9279a206-p7mq2" Nov 01 20:46:17.501838 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:17.501661 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bc83511-e182-4a03-97d1-376fed073d0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9279a206-p7mq2\" (UID: \"8bc83511-e182-4a03-97d1-376fed073d0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9279a206-p7mq2" Nov 01 20:46:17.501838 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:17.501714 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bc83511-e182-4a03-97d1-376fed073d0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9279a206-p7mq2\" (UID: \"8bc83511-e182-4a03-97d1-376fed073d0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9279a206-p7mq2" Nov 01 20:46:17.501838 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:17.501785 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9279a206-p7mq2"] Nov 01 20:46:17.603093 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:17.602978 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk5ks\" (UniqueName: \"kubernetes.io/projected/8bc83511-e182-4a03-97d1-376fed073d0e-kube-api-access-jk5ks\") pod \"maas-api-revoke-sub-9279a206-p7mq2\" (UID: \"8bc83511-e182-4a03-97d1-376fed073d0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9279a206-p7mq2" Nov 01 20:46:17.603093 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:17.603066 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bc83511-e182-4a03-97d1-376fed073d0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9279a206-p7mq2\" (UID: \"8bc83511-e182-4a03-97d1-376fed073d0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9279a206-p7mq2" Nov 01 20:46:17.603301 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:17.603101 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bc83511-e182-4a03-97d1-376fed073d0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9279a206-p7mq2\" (UID: \"8bc83511-e182-4a03-97d1-376fed073d0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9279a206-p7mq2" Nov 01 20:46:17.603808 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:17.603780 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bc83511-e182-4a03-97d1-376fed073d0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9279a206-p7mq2\" (UID: \"8bc83511-e182-4a03-97d1-376fed073d0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9279a206-p7mq2" Nov 01 20:46:17.605346 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:17.605326 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bc83511-e182-4a03-97d1-376fed073d0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9279a206-p7mq2\" (UID: \"8bc83511-e182-4a03-97d1-376fed073d0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9279a206-p7mq2" Nov 01 20:46:17.660707 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:17.660671 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk5ks\" (UniqueName: \"kubernetes.io/projected/8bc83511-e182-4a03-97d1-376fed073d0e-kube-api-access-jk5ks\") pod \"maas-api-revoke-sub-9279a206-p7mq2\" (UID: \"8bc83511-e182-4a03-97d1-376fed073d0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9279a206-p7mq2" Nov 01 20:46:17.681138 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:17.681112 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54db2327-5gml4"] Nov 01 20:46:17.682042 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:46:17.681986 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe70021e_7901_455b_985a_78fb5f4b66e7.slice/crio-a4f2c84e6f94f3d58dcb882d98a8014fc43dbcbe702fe0d9e941bca2584c1161 WatchSource:0}: Error finding container a4f2c84e6f94f3d58dcb882d98a8014fc43dbcbe702fe0d9e941bca2584c1161: Status 404 returned error can't find the container with id a4f2c84e6f94f3d58dcb882d98a8014fc43dbcbe702fe0d9e941bca2584c1161 Nov 01 20:46:17.761134 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:17.761103 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9279a206-p7mq2" Nov 01 20:46:17.811435 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:17.811401 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54db2327-5gml4" event={"ID":"be70021e-7901-455b-985a-78fb5f4b66e7","Type":"ContainerStarted","Data":"a4f2c84e6f94f3d58dcb882d98a8014fc43dbcbe702fe0d9e941bca2584c1161"} Nov 01 20:46:18.135494 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:18.135466 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9279a206-p7mq2"] Nov 01 20:46:18.136291 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:46:18.136264 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bc83511_e182_4a03_97d1_376fed073d0e.slice/crio-897d23cb6788ef10ee52d97d33122baecbc9a958cdffb79aa90b41de45cca8de WatchSource:0}: Error finding container 897d23cb6788ef10ee52d97d33122baecbc9a958cdffb79aa90b41de45cca8de: Status 404 returned error can't find the container with id 897d23cb6788ef10ee52d97d33122baecbc9a958cdffb79aa90b41de45cca8de Nov 01 20:46:18.819751 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:18.819720 4484 generic.go:356] "Generic (PLEG): container finished" podID="be70021e-7901-455b-985a-78fb5f4b66e7" containerID="695f8ce87a7586ec8185af77047bd547bbf8bcac3241c1fcd7a9669091eebf95" exitCode=0 Nov 01 20:46:18.819933 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:18.819804 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54db2327-5gml4" event={"ID":"be70021e-7901-455b-985a-78fb5f4b66e7","Type":"ContainerDied","Data":"695f8ce87a7586ec8185af77047bd547bbf8bcac3241c1fcd7a9669091eebf95"} Nov 01 20:46:18.821462 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:18.821438 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9279a206-p7mq2" event={"ID":"8bc83511-e182-4a03-97d1-376fed073d0e","Type":"ContainerStarted","Data":"6f9c877c16607bd0a26419238c37ec08a892a0ac81635cb94b3bf527112d52ed"} Nov 01 20:46:18.821573 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:18.821488 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9279a206-p7mq2" event={"ID":"8bc83511-e182-4a03-97d1-376fed073d0e","Type":"ContainerStarted","Data":"897d23cb6788ef10ee52d97d33122baecbc9a958cdffb79aa90b41de45cca8de"} Nov 01 20:46:19.055031 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:19.054223 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9279a206-p7mq2" podStartSLOduration=2.054203723 podStartE2EDuration="2.054203723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:46:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:46:19.050115971 +0000 UTC m=+929.827756107" watchObservedRunningTime="2026-11-01 20:46:19.054203723 +0000 UTC m=+929.831843863" Nov 01 20:46:19.586015 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:19.579671 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:46:19.829149 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:19.829113 4484 generic.go:356] "Generic (PLEG): container finished" podID="8bc83511-e182-4a03-97d1-376fed073d0e" containerID="6f9c877c16607bd0a26419238c37ec08a892a0ac81635cb94b3bf527112d52ed" exitCode=0 Nov 01 20:46:19.829323 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:19.829193 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9279a206-p7mq2" event={"ID":"8bc83511-e182-4a03-97d1-376fed073d0e","Type":"ContainerDied","Data":"6f9c877c16607bd0a26419238c37ec08a892a0ac81635cb94b3bf527112d52ed"} Nov 01 20:46:19.992264 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:19.992230 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54db2327-5gml4" Nov 01 20:46:20.030541 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:20.030509 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be70021e-7901-455b-985a-78fb5f4b66e7-kube-api-access-vzfds\" (UniqueName: \"kubernetes.io/projected/be70021e-7901-455b-985a-78fb5f4b66e7-kube-api-access-vzfds\") pod \"be70021e-7901-455b-985a-78fb5f4b66e7\" (UID: \"be70021e-7901-455b-985a-78fb5f4b66e7\") " Nov 01 20:46:20.030716 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:20.030565 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be70021e-7901-455b-985a-78fb5f4b66e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be70021e-7901-455b-985a-78fb5f4b66e7-maas-api-xks-ca\") pod \"be70021e-7901-455b-985a-78fb5f4b66e7\" (UID: \"be70021e-7901-455b-985a-78fb5f4b66e7\") " Nov 01 20:46:20.030716 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:20.030667 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be70021e-7901-455b-985a-78fb5f4b66e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be70021e-7901-455b-985a-78fb5f4b66e7-maas-api-openshift-ca\") pod \"be70021e-7901-455b-985a-78fb5f4b66e7\" (UID: \"be70021e-7901-455b-985a-78fb5f4b66e7\") " Nov 01 20:46:20.031029 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:20.030989 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be70021e-7901-455b-985a-78fb5f4b66e7-maas-api-openshift-ca" pod "be70021e-7901-455b-985a-78fb5f4b66e7" (UID: "be70021e-7901-455b-985a-78fb5f4b66e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:46:20.032603 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:20.032581 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be70021e-7901-455b-985a-78fb5f4b66e7-maas-api-xks-ca" pod "be70021e-7901-455b-985a-78fb5f4b66e7" (UID: "be70021e-7901-455b-985a-78fb5f4b66e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:46:20.032688 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:20.032670 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be70021e-7901-455b-985a-78fb5f4b66e7-kube-api-access-vzfds" pod "be70021e-7901-455b-985a-78fb5f4b66e7" (UID: "be70021e-7901-455b-985a-78fb5f4b66e7"). InnerVolumeSpecName "kube-api-access-vzfds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:46:20.132484 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:20.132450 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzfds\" (UniqueName: \"kubernetes.io/projected/be70021e-7901-455b-985a-78fb5f4b66e7-kube-api-access-vzfds\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:46:20.132484 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:20.132484 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be70021e-7901-455b-985a-78fb5f4b66e7-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:46:20.132727 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:20.132498 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be70021e-7901-455b-985a-78fb5f4b66e7-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:46:20.845102 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:20.845067 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54db2327-5gml4" Nov 01 20:46:20.845102 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:20.845066 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54db2327-5gml4" event={"ID":"be70021e-7901-455b-985a-78fb5f4b66e7","Type":"ContainerDied","Data":"a4f2c84e6f94f3d58dcb882d98a8014fc43dbcbe702fe0d9e941bca2584c1161"} Nov 01 20:46:20.845590 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:20.845136 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4f2c84e6f94f3d58dcb882d98a8014fc43dbcbe702fe0d9e941bca2584c1161" Nov 01 20:46:21.006501 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:21.006472 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9279a206-p7mq2" Nov 01 20:46:21.147447 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:21.147359 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bc83511-e182-4a03-97d1-376fed073d0e-kube-api-access-jk5ks\" (UniqueName: \"kubernetes.io/projected/8bc83511-e182-4a03-97d1-376fed073d0e-kube-api-access-jk5ks\") pod \"8bc83511-e182-4a03-97d1-376fed073d0e\" (UID: \"8bc83511-e182-4a03-97d1-376fed073d0e\") " Nov 01 20:46:21.147907 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:21.147877 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bc83511-e182-4a03-97d1-376fed073d0e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bc83511-e182-4a03-97d1-376fed073d0e-maas-api-openshift-ca\") pod \"8bc83511-e182-4a03-97d1-376fed073d0e\" (UID: \"8bc83511-e182-4a03-97d1-376fed073d0e\") " Nov 01 20:46:21.148140 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:21.148122 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bc83511-e182-4a03-97d1-376fed073d0e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bc83511-e182-4a03-97d1-376fed073d0e-maas-api-xks-ca\") pod \"8bc83511-e182-4a03-97d1-376fed073d0e\" (UID: \"8bc83511-e182-4a03-97d1-376fed073d0e\") " Nov 01 20:46:21.150165 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:21.150140 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bc83511-e182-4a03-97d1-376fed073d0e-maas-api-openshift-ca" pod "8bc83511-e182-4a03-97d1-376fed073d0e" (UID: "8bc83511-e182-4a03-97d1-376fed073d0e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:46:21.156224 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:21.156190 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bc83511-e182-4a03-97d1-376fed073d0e-maas-api-xks-ca" pod "8bc83511-e182-4a03-97d1-376fed073d0e" (UID: "8bc83511-e182-4a03-97d1-376fed073d0e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:46:21.156462 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:21.156439 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bc83511-e182-4a03-97d1-376fed073d0e-kube-api-access-jk5ks" pod "8bc83511-e182-4a03-97d1-376fed073d0e" (UID: "8bc83511-e182-4a03-97d1-376fed073d0e"). InnerVolumeSpecName "kube-api-access-jk5ks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:46:21.250536 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:21.250494 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bc83511-e182-4a03-97d1-376fed073d0e-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:46:21.250536 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:21.250529 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bc83511-e182-4a03-97d1-376fed073d0e-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:46:21.250536 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:21.250538 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jk5ks\" (UniqueName: \"kubernetes.io/projected/8bc83511-e182-4a03-97d1-376fed073d0e-kube-api-access-jk5ks\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:46:21.805681 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:21.805650 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9eae9a85-c6hsl"] Nov 01 20:46:21.806232 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:21.806215 4484 cpu_manager.go:414] "containerMap: removing container" podUID="8bc83511-e182-4a03-97d1-376fed073d0e" containerName="revoke-keys" Nov 01 20:46:21.806289 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:21.806235 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bc83511-e182-4a03-97d1-376fed073d0e" containerName="revoke-keys" Nov 01 20:46:21.806289 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:21.806278 4484 cpu_manager.go:414] "containerMap: removing container" podUID="be70021e-7901-455b-985a-78fb5f4b66e7" containerName="revoke-keys" Nov 01 20:46:21.806289 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:21.806283 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be70021e-7901-455b-985a-78fb5f4b66e7" containerName="revoke-keys" Nov 01 20:46:21.806387 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:21.806362 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="be70021e-7901-455b-985a-78fb5f4b66e7" containerName="revoke-keys" Nov 01 20:46:21.806387 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:21.806375 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bc83511-e182-4a03-97d1-376fed073d0e" containerName="revoke-keys" Nov 01 20:46:21.809903 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:21.809884 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eae9a85-c6hsl" Nov 01 20:46:21.854384 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:21.854359 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9279a206-p7mq2" Nov 01 20:46:21.854384 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:21.854361 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9279a206-p7mq2" event={"ID":"8bc83511-e182-4a03-97d1-376fed073d0e","Type":"ContainerDied","Data":"897d23cb6788ef10ee52d97d33122baecbc9a958cdffb79aa90b41de45cca8de"} Nov 01 20:46:21.854836 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:21.854415 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="897d23cb6788ef10ee52d97d33122baecbc9a958cdffb79aa90b41de45cca8de" Nov 01 20:46:21.872118 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:21.872087 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9eae9a85-c6hsl"] Nov 01 20:46:21.960222 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:21.960187 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cac12c9e-f52b-4786-ab06-ca7daea3acb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9eae9a85-c6hsl\" (UID: \"cac12c9e-f52b-4786-ab06-ca7daea3acb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eae9a85-c6hsl" Nov 01 20:46:21.960420 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:21.960245 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gcmn\" (UniqueName: \"kubernetes.io/projected/cac12c9e-f52b-4786-ab06-ca7daea3acb9-kube-api-access-4gcmn\") pod \"maas-api-revoke-sub-9eae9a85-c6hsl\" (UID: \"cac12c9e-f52b-4786-ab06-ca7daea3acb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eae9a85-c6hsl" Nov 01 20:46:21.960420 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:21.960300 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cac12c9e-f52b-4786-ab06-ca7daea3acb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9eae9a85-c6hsl\" (UID: \"cac12c9e-f52b-4786-ab06-ca7daea3acb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eae9a85-c6hsl" Nov 01 20:46:22.062082 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:22.061975 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cac12c9e-f52b-4786-ab06-ca7daea3acb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9eae9a85-c6hsl\" (UID: \"cac12c9e-f52b-4786-ab06-ca7daea3acb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eae9a85-c6hsl" Nov 01 20:46:22.062082 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:22.062066 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gcmn\" (UniqueName: \"kubernetes.io/projected/cac12c9e-f52b-4786-ab06-ca7daea3acb9-kube-api-access-4gcmn\") pod \"maas-api-revoke-sub-9eae9a85-c6hsl\" (UID: \"cac12c9e-f52b-4786-ab06-ca7daea3acb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eae9a85-c6hsl" Nov 01 20:46:22.062330 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:22.062108 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cac12c9e-f52b-4786-ab06-ca7daea3acb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9eae9a85-c6hsl\" (UID: \"cac12c9e-f52b-4786-ab06-ca7daea3acb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eae9a85-c6hsl" Nov 01 20:46:22.062725 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:22.062705 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cac12c9e-f52b-4786-ab06-ca7daea3acb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9eae9a85-c6hsl\" (UID: \"cac12c9e-f52b-4786-ab06-ca7daea3acb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eae9a85-c6hsl" Nov 01 20:46:22.064469 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:22.064450 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cac12c9e-f52b-4786-ab06-ca7daea3acb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9eae9a85-c6hsl\" (UID: \"cac12c9e-f52b-4786-ab06-ca7daea3acb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eae9a85-c6hsl" Nov 01 20:46:22.120335 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:22.120294 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gcmn\" (UniqueName: \"kubernetes.io/projected/cac12c9e-f52b-4786-ab06-ca7daea3acb9-kube-api-access-4gcmn\") pod \"maas-api-revoke-sub-9eae9a85-c6hsl\" (UID: \"cac12c9e-f52b-4786-ab06-ca7daea3acb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eae9a85-c6hsl" Nov 01 20:46:22.124913 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:22.124888 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eae9a85-c6hsl" Nov 01 20:46:22.496864 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:22.496830 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk"] Nov 01 20:46:22.502300 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:22.502279 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk" Nov 01 20:46:22.503300 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:46:22.503275 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcac12c9e_f52b_4786_ab06_ca7daea3acb9.slice/crio-6a8919e9415ddd67a99d7b43d8dc0a359bad7398248c3e2c605604c202a3c4f5 WatchSource:0}: Error finding container 6a8919e9415ddd67a99d7b43d8dc0a359bad7398248c3e2c605604c202a3c4f5: Status 404 returned error can't find the container with id 6a8919e9415ddd67a99d7b43d8dc0a359bad7398248c3e2c605604c202a3c4f5 Nov 01 20:46:22.504530 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:22.504491 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9eae9a85-c6hsl"] Nov 01 20:46:22.555452 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:22.555430 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-dd038b-openshift-default-dockercfg-vkx72\"" Nov 01 20:46:22.569266 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:22.569240 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17927c6e-7e23-4717-be79-d2569ebae4ca-istio-envoy\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk\" (UID: \"17927c6e-7e23-4717-be79-d2569ebae4ca\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk" Nov 01 20:46:22.569422 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:22.569405 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17927c6e-7e23-4717-be79-d2569ebae4ca-workload-certs\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk\" (UID: \"17927c6e-7e23-4717-be79-d2569ebae4ca\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk" Nov 01 20:46:22.569464 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:22.569430 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17927c6e-7e23-4717-be79-d2569ebae4ca-istiod-ca-cert\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk\" (UID: \"17927c6e-7e23-4717-be79-d2569ebae4ca\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk" Nov 01 20:46:22.569464 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:22.569447 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17927c6e-7e23-4717-be79-d2569ebae4ca-istio-token\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk\" (UID: \"17927c6e-7e23-4717-be79-d2569ebae4ca\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk" Nov 01 20:46:22.569555 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:22.569467 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17927c6e-7e23-4717-be79-d2569ebae4ca-workload-socket\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk\" (UID: \"17927c6e-7e23-4717-be79-d2569ebae4ca\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk" Nov 01 20:46:22.569603 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:22.569584 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17927c6e-7e23-4717-be79-d2569ebae4ca-istio-data\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk\" (UID: \"17927c6e-7e23-4717-be79-d2569ebae4ca\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk" Nov 01 20:46:22.569639 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:22.569616 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17927c6e-7e23-4717-be79-d2569ebae4ca-istio-podinfo\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk\" (UID: \"17927c6e-7e23-4717-be79-d2569ebae4ca\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk" Nov 01 20:46:22.569691 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:22.569641 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17927c6e-7e23-4717-be79-d2569ebae4ca-credential-socket\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk\" (UID: \"17927c6e-7e23-4717-be79-d2569ebae4ca\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk" Nov 01 20:46:22.569738 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:22.569711 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mff9h\" (UniqueName: \"kubernetes.io/projected/17927c6e-7e23-4717-be79-d2569ebae4ca-kube-api-access-mff9h\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk\" (UID: \"17927c6e-7e23-4717-be79-d2569ebae4ca\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk" Nov 01 20:46:22.602921 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:22.602894 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk"] Nov 01 20:46:22.670409 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:22.670370 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17927c6e-7e23-4717-be79-d2569ebae4ca-istio-envoy\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk\" (UID: \"17927c6e-7e23-4717-be79-d2569ebae4ca\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk" Nov 01 20:46:22.670601 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:22.670536 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17927c6e-7e23-4717-be79-d2569ebae4ca-workload-certs\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk\" (UID: \"17927c6e-7e23-4717-be79-d2569ebae4ca\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk" Nov 01 20:46:22.670601 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:22.670574 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17927c6e-7e23-4717-be79-d2569ebae4ca-istiod-ca-cert\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk\" (UID: \"17927c6e-7e23-4717-be79-d2569ebae4ca\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk" Nov 01 20:46:22.670760 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:22.670603 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17927c6e-7e23-4717-be79-d2569ebae4ca-istio-token\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk\" (UID: \"17927c6e-7e23-4717-be79-d2569ebae4ca\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk" Nov 01 20:46:22.670760 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:22.670640 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17927c6e-7e23-4717-be79-d2569ebae4ca-workload-socket\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk\" (UID: \"17927c6e-7e23-4717-be79-d2569ebae4ca\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk" Nov 01 20:46:22.670760 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:22.670669 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17927c6e-7e23-4717-be79-d2569ebae4ca-istio-data\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk\" (UID: \"17927c6e-7e23-4717-be79-d2569ebae4ca\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk" Nov 01 20:46:22.670760 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:22.670714 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17927c6e-7e23-4717-be79-d2569ebae4ca-istio-podinfo\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk\" (UID: \"17927c6e-7e23-4717-be79-d2569ebae4ca\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk" Nov 01 20:46:22.670979 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:22.670781 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17927c6e-7e23-4717-be79-d2569ebae4ca-credential-socket\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk\" (UID: \"17927c6e-7e23-4717-be79-d2569ebae4ca\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk" Nov 01 20:46:22.670979 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:22.670842 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mff9h\" (UniqueName: \"kubernetes.io/projected/17927c6e-7e23-4717-be79-d2569ebae4ca-kube-api-access-mff9h\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk\" (UID: \"17927c6e-7e23-4717-be79-d2569ebae4ca\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk" Nov 01 20:46:22.670979 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:22.670966 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17927c6e-7e23-4717-be79-d2569ebae4ca-workload-certs\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk\" (UID: \"17927c6e-7e23-4717-be79-d2569ebae4ca\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk" Nov 01 20:46:22.671418 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:22.671370 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17927c6e-7e23-4717-be79-d2569ebae4ca-istiod-ca-cert\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk\" (UID: \"17927c6e-7e23-4717-be79-d2569ebae4ca\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk" Nov 01 20:46:22.671565 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:22.671423 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17927c6e-7e23-4717-be79-d2569ebae4ca-credential-socket\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk\" (UID: \"17927c6e-7e23-4717-be79-d2569ebae4ca\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk" Nov 01 20:46:22.671676 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:22.671650 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17927c6e-7e23-4717-be79-d2569ebae4ca-workload-socket\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk\" (UID: \"17927c6e-7e23-4717-be79-d2569ebae4ca\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk" Nov 01 20:46:22.671772 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:22.671748 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17927c6e-7e23-4717-be79-d2569ebae4ca-istio-data\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk\" (UID: \"17927c6e-7e23-4717-be79-d2569ebae4ca\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk" Nov 01 20:46:22.673035 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:22.673014 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17927c6e-7e23-4717-be79-d2569ebae4ca-istio-envoy\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk\" (UID: \"17927c6e-7e23-4717-be79-d2569ebae4ca\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk" Nov 01 20:46:22.673628 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:22.673611 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17927c6e-7e23-4717-be79-d2569ebae4ca-istio-podinfo\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk\" (UID: \"17927c6e-7e23-4717-be79-d2569ebae4ca\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk" Nov 01 20:46:22.728342 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:22.728313 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17927c6e-7e23-4717-be79-d2569ebae4ca-istio-token\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk\" (UID: \"17927c6e-7e23-4717-be79-d2569ebae4ca\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk" Nov 01 20:46:22.728480 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:22.728327 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mff9h\" (UniqueName: \"kubernetes.io/projected/17927c6e-7e23-4717-be79-d2569ebae4ca-kube-api-access-mff9h\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk\" (UID: \"17927c6e-7e23-4717-be79-d2569ebae4ca\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk" Nov 01 20:46:22.830162 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:22.830133 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk" Nov 01 20:46:22.850162 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:22.850125 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r"] Nov 01 20:46:22.854487 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:22.854464 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r" Nov 01 20:46:22.864062 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:22.864025 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eae9a85-c6hsl" event={"ID":"cac12c9e-f52b-4786-ab06-ca7daea3acb9","Type":"ContainerStarted","Data":"cf539a500adad9d6d35b61fa352fdbff05798df35d63d3d71fb118432c6e5ef4"} Nov 01 20:46:22.864190 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:22.864081 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eae9a85-c6hsl" event={"ID":"cac12c9e-f52b-4786-ab06-ca7daea3acb9","Type":"ContainerStarted","Data":"6a8919e9415ddd67a99d7b43d8dc0a359bad7398248c3e2c605604c202a3c4f5"} Nov 01 20:46:22.917648 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:22.917618 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r"] Nov 01 20:46:22.973717 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:22.973681 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-istio-podinfo\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r\" (UID: \"74fb0cfd-c53c-49da-a2c8-02cdd62a79b7\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r" Nov 01 20:46:22.973871 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:22.973753 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-istio-token\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r\" (UID: \"74fb0cfd-c53c-49da-a2c8-02cdd62a79b7\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r" Nov 01 20:46:22.973941 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:22.973862 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-istiod-ca-cert\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r\" (UID: \"74fb0cfd-c53c-49da-a2c8-02cdd62a79b7\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r" Nov 01 20:46:22.973982 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:22.973957 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-istio-data\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r\" (UID: \"74fb0cfd-c53c-49da-a2c8-02cdd62a79b7\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r" Nov 01 20:46:22.974177 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:22.974162 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-workload-certs\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r\" (UID: \"74fb0cfd-c53c-49da-a2c8-02cdd62a79b7\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r" Nov 01 20:46:22.974239 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:22.974196 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-642kd\" (UniqueName: \"kubernetes.io/projected/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-kube-api-access-642kd\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r\" (UID: \"74fb0cfd-c53c-49da-a2c8-02cdd62a79b7\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r" Nov 01 20:46:22.974239 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:22.974220 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-istio-envoy\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r\" (UID: \"74fb0cfd-c53c-49da-a2c8-02cdd62a79b7\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r" Nov 01 20:46:22.974348 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:22.974323 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-workload-socket\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r\" (UID: \"74fb0cfd-c53c-49da-a2c8-02cdd62a79b7\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r" Nov 01 20:46:22.974431 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:22.974412 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-credential-socket\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r\" (UID: \"74fb0cfd-c53c-49da-a2c8-02cdd62a79b7\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r" Nov 01 20:46:23.075803 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:23.075704 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-istiod-ca-cert\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r\" (UID: \"74fb0cfd-c53c-49da-a2c8-02cdd62a79b7\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r" Nov 01 20:46:23.075961 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:23.075811 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-istio-data\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r\" (UID: \"74fb0cfd-c53c-49da-a2c8-02cdd62a79b7\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r" Nov 01 20:46:23.075961 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:23.075875 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-workload-certs\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r\" (UID: \"74fb0cfd-c53c-49da-a2c8-02cdd62a79b7\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r" Nov 01 20:46:23.076083 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:23.076059 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-642kd\" (UniqueName: \"kubernetes.io/projected/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-kube-api-access-642kd\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r\" (UID: \"74fb0cfd-c53c-49da-a2c8-02cdd62a79b7\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r" Nov 01 20:46:23.076138 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:23.076112 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-istio-envoy\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r\" (UID: \"74fb0cfd-c53c-49da-a2c8-02cdd62a79b7\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r" Nov 01 20:46:23.076190 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:23.076157 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-workload-socket\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r\" (UID: \"74fb0cfd-c53c-49da-a2c8-02cdd62a79b7\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r" Nov 01 20:46:23.076250 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:23.076200 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-credential-socket\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r\" (UID: \"74fb0cfd-c53c-49da-a2c8-02cdd62a79b7\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r" Nov 01 20:46:23.076250 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:23.076208 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-istio-data\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r\" (UID: \"74fb0cfd-c53c-49da-a2c8-02cdd62a79b7\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r" Nov 01 20:46:23.076355 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:23.076286 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-istio-podinfo\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r\" (UID: \"74fb0cfd-c53c-49da-a2c8-02cdd62a79b7\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r" Nov 01 20:46:23.076355 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:23.076315 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-istio-token\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r\" (UID: \"74fb0cfd-c53c-49da-a2c8-02cdd62a79b7\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r" Nov 01 20:46:23.076460 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:23.076417 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-workload-certs\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r\" (UID: \"74fb0cfd-c53c-49da-a2c8-02cdd62a79b7\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r" Nov 01 20:46:23.076581 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:23.076559 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-credential-socket\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r\" (UID: \"74fb0cfd-c53c-49da-a2c8-02cdd62a79b7\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r" Nov 01 20:46:23.076648 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:23.076579 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-istiod-ca-cert\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r\" (UID: \"74fb0cfd-c53c-49da-a2c8-02cdd62a79b7\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r" Nov 01 20:46:23.076719 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:23.076696 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-workload-socket\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r\" (UID: \"74fb0cfd-c53c-49da-a2c8-02cdd62a79b7\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r" Nov 01 20:46:23.078445 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:23.078421 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-istio-envoy\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r\" (UID: \"74fb0cfd-c53c-49da-a2c8-02cdd62a79b7\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r" Nov 01 20:46:23.078833 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:23.078804 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-istio-podinfo\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r\" (UID: \"74fb0cfd-c53c-49da-a2c8-02cdd62a79b7\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r" Nov 01 20:46:23.087293 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:23.087252 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eae9a85-c6hsl" podStartSLOduration=2.087240701 podStartE2EDuration="2.087240701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:46:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:46:23.086573114 +0000 UTC m=+933.864213249" watchObservedRunningTime="2026-11-01 20:46:23.087240701 +0000 UTC m=+933.864880836" Nov 01 20:46:23.134106 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:23.134067 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-istio-token\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r\" (UID: \"74fb0cfd-c53c-49da-a2c8-02cdd62a79b7\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r" Nov 01 20:46:23.134243 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:23.134152 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-642kd\" (UniqueName: \"kubernetes.io/projected/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-kube-api-access-642kd\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r\" (UID: \"74fb0cfd-c53c-49da-a2c8-02cdd62a79b7\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r" Nov 01 20:46:23.170782 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:23.170748 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r" Nov 01 20:46:23.210620 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:23.210588 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk"] Nov 01 20:46:23.211144 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:46:23.211114 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17927c6e_7e23_4717_be79_d2569ebae4ca.slice/crio-a6c519ea34576b502c913fa8b80e97cb35936a99544215e3b22757edd0e78326 WatchSource:0}: Error finding container a6c519ea34576b502c913fa8b80e97cb35936a99544215e3b22757edd0e78326: Status 404 returned error can't find the container with id a6c519ea34576b502c913fa8b80e97cb35936a99544215e3b22757edd0e78326 Nov 01 20:46:23.214169 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:23.214133 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:46:23.214259 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:23.214201 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:46:23.214259 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:23.214230 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:46:23.561911 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:23.559106 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r"] Nov 01 20:46:23.563698 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:46:23.562496 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74fb0cfd_c53c_49da_a2c8_02cdd62a79b7.slice/crio-22776dc59cdbcda85af1703be5bd2b3386b7345ec3ecc584ad6e4a16f16a24bc WatchSource:0}: Error finding container 22776dc59cdbcda85af1703be5bd2b3386b7345ec3ecc584ad6e4a16f16a24bc: Status 404 returned error can't find the container with id 22776dc59cdbcda85af1703be5bd2b3386b7345ec3ecc584ad6e4a16f16a24bc Nov 01 20:46:23.566491 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:23.566455 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:46:23.566604 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:23.566525 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:46:23.566604 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:23.566553 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:46:23.871955 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:23.871925 4484 generic.go:356] "Generic (PLEG): container finished" podID="cac12c9e-f52b-4786-ab06-ca7daea3acb9" containerID="cf539a500adad9d6d35b61fa352fdbff05798df35d63d3d71fb118432c6e5ef4" exitCode=0 Nov 01 20:46:23.872408 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:23.872029 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eae9a85-c6hsl" event={"ID":"cac12c9e-f52b-4786-ab06-ca7daea3acb9","Type":"ContainerDied","Data":"cf539a500adad9d6d35b61fa352fdbff05798df35d63d3d71fb118432c6e5ef4"} Nov 01 20:46:23.873789 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:23.873760 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r" event={"ID":"74fb0cfd-c53c-49da-a2c8-02cdd62a79b7","Type":"ContainerStarted","Data":"2fa0e01f056ed2a498ec729af832d820d805c5377adf30d0f65b509f7a2a2614"} Nov 01 20:46:23.873916 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:23.873808 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r" event={"ID":"74fb0cfd-c53c-49da-a2c8-02cdd62a79b7","Type":"ContainerStarted","Data":"22776dc59cdbcda85af1703be5bd2b3386b7345ec3ecc584ad6e4a16f16a24bc"} Nov 01 20:46:23.875540 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:23.875512 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk" event={"ID":"17927c6e-7e23-4717-be79-d2569ebae4ca","Type":"ContainerStarted","Data":"a45831dcad5f6438e76a52c65838f3d14b088583f334964e75b98f643d744308"} Nov 01 20:46:23.875651 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:23.875556 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk" event={"ID":"17927c6e-7e23-4717-be79-d2569ebae4ca","Type":"ContainerStarted","Data":"a6c519ea34576b502c913fa8b80e97cb35936a99544215e3b22757edd0e78326"} Nov 01 20:46:24.123075 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:24.112775 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk" podStartSLOduration=2.11275595 podStartE2EDuration="2.11275595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:46:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:46:24.104616103 +0000 UTC m=+934.882256239" watchObservedRunningTime="2026-11-01 20:46:24.11275595 +0000 UTC m=+934.890396085" Nov 01 20:46:24.171067 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:24.171033 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r" Nov 01 20:46:24.172890 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:24.172861 4484 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 20:46:24.173035 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:24.172909 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r" podUID="74fb0cfd-c53c-49da-a2c8-02cdd62a79b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 20:46:24.228021 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:24.227702 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r" podStartSLOduration=2.227680234 podStartE2EDuration="2.227680234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:46:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:46:24.225516903 +0000 UTC m=+935.003157061" watchObservedRunningTime="2026-11-01 20:46:24.227680234 +0000 UTC m=+935.005320370" Nov 01 20:46:24.307033 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:24.306592 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:46:24.841213 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:24.841160 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk" Nov 01 20:46:24.858228 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:24.858173 4484 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 01 20:46:24.858557 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:24.858517 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk" podUID="17927c6e-7e23-4717-be79-d2569ebae4ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 01 20:46:25.179378 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:25.178802 4484 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 20:46:25.179378 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:25.178863 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r" podUID="74fb0cfd-c53c-49da-a2c8-02cdd62a79b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 20:46:25.543018 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:25.536810 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eae9a85-c6hsl" Nov 01 20:46:25.613028 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:25.612419 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cac12c9e-f52b-4786-ab06-ca7daea3acb9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cac12c9e-f52b-4786-ab06-ca7daea3acb9-maas-api-openshift-ca\") pod \"cac12c9e-f52b-4786-ab06-ca7daea3acb9\" (UID: \"cac12c9e-f52b-4786-ab06-ca7daea3acb9\") " Nov 01 20:46:25.613028 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:25.612535 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cac12c9e-f52b-4786-ab06-ca7daea3acb9-kube-api-access-4gcmn\" (UniqueName: \"kubernetes.io/projected/cac12c9e-f52b-4786-ab06-ca7daea3acb9-kube-api-access-4gcmn\") pod \"cac12c9e-f52b-4786-ab06-ca7daea3acb9\" (UID: \"cac12c9e-f52b-4786-ab06-ca7daea3acb9\") " Nov 01 20:46:25.613028 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:25.612585 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cac12c9e-f52b-4786-ab06-ca7daea3acb9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cac12c9e-f52b-4786-ab06-ca7daea3acb9-maas-api-xks-ca\") pod \"cac12c9e-f52b-4786-ab06-ca7daea3acb9\" (UID: \"cac12c9e-f52b-4786-ab06-ca7daea3acb9\") " Nov 01 20:46:25.641021 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:25.614264 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cac12c9e-f52b-4786-ab06-ca7daea3acb9-maas-api-openshift-ca" pod "cac12c9e-f52b-4786-ab06-ca7daea3acb9" (UID: "cac12c9e-f52b-4786-ab06-ca7daea3acb9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:46:25.641021 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:25.619903 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cac12c9e-f52b-4786-ab06-ca7daea3acb9-kube-api-access-4gcmn" pod "cac12c9e-f52b-4786-ab06-ca7daea3acb9" (UID: "cac12c9e-f52b-4786-ab06-ca7daea3acb9"). InnerVolumeSpecName "kube-api-access-4gcmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:46:25.641021 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:25.620712 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cac12c9e-f52b-4786-ab06-ca7daea3acb9-maas-api-xks-ca" pod "cac12c9e-f52b-4786-ab06-ca7daea3acb9" (UID: "cac12c9e-f52b-4786-ab06-ca7daea3acb9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:46:26.217020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:26.212455 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cac12c9e-f52b-4786-ab06-ca7daea3acb9-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:46:26.244132 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:26.244078 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gcmn\" (UniqueName: \"kubernetes.io/projected/cac12c9e-f52b-4786-ab06-ca7daea3acb9-kube-api-access-4gcmn\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:46:26.320018 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:26.317142 4484 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 01 20:46:26.320018 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:26.317253 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk" podUID="17927c6e-7e23-4717-be79-d2569ebae4ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 01 20:46:26.320018 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:26.318412 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cac12c9e-f52b-4786-ab06-ca7daea3acb9-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:46:26.338689 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:26.338640 4484 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 20:46:26.338978 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:26.338932 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r" podUID="74fb0cfd-c53c-49da-a2c8-02cdd62a79b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 20:46:26.386031 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:26.382734 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" podUID="17a9ac24-19cb-48c4-99b5-d74ce8aac0a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:46:26.526236 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:26.526201 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eae9a85-c6hsl" Nov 01 20:46:26.532207 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:26.532158 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eae9a85-c6hsl" event={"ID":"cac12c9e-f52b-4786-ab06-ca7daea3acb9","Type":"ContainerDied","Data":"6a8919e9415ddd67a99d7b43d8dc0a359bad7398248c3e2c605604c202a3c4f5"} Nov 01 20:46:26.532476 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:26.532446 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a8919e9415ddd67a99d7b43d8dc0a359bad7398248c3e2c605604c202a3c4f5" Nov 01 20:46:26.831532 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:26.831435 4484 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 01 20:46:26.831805 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:26.831777 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk" podUID="17927c6e-7e23-4717-be79-d2569ebae4ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 01 20:46:26.942110 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:26.941227 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" podUID="311f9699-6585-4a2c-8c20-e6b696b2f857" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:46:27.171591 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:27.171497 4484 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 20:46:27.171869 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:27.171829 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r" podUID="74fb0cfd-c53c-49da-a2c8-02cdd62a79b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 20:46:27.341766 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:27.341091 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" podUID="fe8307db-264d-4b8a-b097-f180bc255a9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 01 20:46:27.383526 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:27.383163 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" podUID="17a9ac24-19cb-48c4-99b5-d74ce8aac0a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:46:27.406264 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:27.405547 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" Nov 01 20:46:27.427251 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:27.427094 4484 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" podUID="311f9699-6585-4a2c-8c20-e6b696b2f857" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:46:27.783737 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:27.783691 4484 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:46:27.785023 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:27.784971 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9" podUID="3d20fc4c-5175-4154-8669-2494376150ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:46:27.848021 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:27.842292 4484 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 01 20:46:27.863329 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:27.863284 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk" podUID="17927c6e-7e23-4717-be79-d2569ebae4ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 01 20:46:27.997028 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:27.996978 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" Nov 01 20:46:28.172493 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:28.172407 4484 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 20:46:28.175322 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:28.175267 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r" podUID="74fb0cfd-c53c-49da-a2c8-02cdd62a79b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 20:46:28.662022 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:28.658171 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10fcaf59-jcdrh"] Nov 01 20:46:28.662022 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:28.660254 4484 cpu_manager.go:414] "containerMap: removing container" podUID="cac12c9e-f52b-4786-ab06-ca7daea3acb9" containerName="revoke-keys" Nov 01 20:46:28.662022 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:28.660277 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cac12c9e-f52b-4786-ab06-ca7daea3acb9" containerName="revoke-keys" Nov 01 20:46:28.662022 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:28.660427 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="cac12c9e-f52b-4786-ab06-ca7daea3acb9" containerName="revoke-keys" Nov 01 20:46:28.665677 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:28.665652 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fcaf59-jcdrh" Nov 01 20:46:28.673789 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:28.673722 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62dwp\" (UniqueName: \"kubernetes.io/projected/ebd12d3e-e201-4170-b5f0-18ed8d300682-kube-api-access-62dwp\") pod \"maas-api-revoke-sub-10fcaf59-jcdrh\" (UID: \"ebd12d3e-e201-4170-b5f0-18ed8d300682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fcaf59-jcdrh" Nov 01 20:46:28.673917 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:28.673867 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebd12d3e-e201-4170-b5f0-18ed8d300682-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10fcaf59-jcdrh\" (UID: \"ebd12d3e-e201-4170-b5f0-18ed8d300682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fcaf59-jcdrh" Nov 01 20:46:28.673917 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:28.673909 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebd12d3e-e201-4170-b5f0-18ed8d300682-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10fcaf59-jcdrh\" (UID: \"ebd12d3e-e201-4170-b5f0-18ed8d300682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fcaf59-jcdrh" Nov 01 20:46:28.699172 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:28.699144 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10fcaf59-jcdrh"] Nov 01 20:46:28.758325 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:28.758281 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:46:28.758530 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:28.758380 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:46:28.758603 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:28.758551 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:46:28.758669 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:28.758645 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 20:46:28.775164 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:28.775125 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62dwp\" (UniqueName: \"kubernetes.io/projected/ebd12d3e-e201-4170-b5f0-18ed8d300682-kube-api-access-62dwp\") pod \"maas-api-revoke-sub-10fcaf59-jcdrh\" (UID: \"ebd12d3e-e201-4170-b5f0-18ed8d300682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fcaf59-jcdrh" Nov 01 20:46:28.775331 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:28.775226 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebd12d3e-e201-4170-b5f0-18ed8d300682-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10fcaf59-jcdrh\" (UID: \"ebd12d3e-e201-4170-b5f0-18ed8d300682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fcaf59-jcdrh" Nov 01 20:46:28.775331 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:28.775267 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebd12d3e-e201-4170-b5f0-18ed8d300682-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10fcaf59-jcdrh\" (UID: \"ebd12d3e-e201-4170-b5f0-18ed8d300682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fcaf59-jcdrh" Nov 01 20:46:28.776238 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:28.776215 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebd12d3e-e201-4170-b5f0-18ed8d300682-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10fcaf59-jcdrh\" (UID: \"ebd12d3e-e201-4170-b5f0-18ed8d300682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fcaf59-jcdrh" Nov 01 20:46:28.779318 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:28.779297 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebd12d3e-e201-4170-b5f0-18ed8d300682-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10fcaf59-jcdrh\" (UID: \"ebd12d3e-e201-4170-b5f0-18ed8d300682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fcaf59-jcdrh" Nov 01 20:46:28.835062 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:28.834406 4484 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 01 20:46:28.835062 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:28.834469 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk" podUID="17927c6e-7e23-4717-be79-d2569ebae4ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 01 20:46:28.844804 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:28.844731 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62dwp\" (UniqueName: \"kubernetes.io/projected/ebd12d3e-e201-4170-b5f0-18ed8d300682-kube-api-access-62dwp\") pod \"maas-api-revoke-sub-10fcaf59-jcdrh\" (UID: \"ebd12d3e-e201-4170-b5f0-18ed8d300682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fcaf59-jcdrh" Nov 01 20:46:28.980254 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:28.980154 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fcaf59-jcdrh" Nov 01 20:46:29.208062 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:29.198624 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:46:29.276073 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:29.274766 4484 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 20:46:29.276073 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:29.274825 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r" podUID="74fb0cfd-c53c-49da-a2c8-02cdd62a79b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 20:46:29.468539 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:29.468383 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10fcaf59-jcdrh"] Nov 01 20:46:29.748078 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:29.745354 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fcaf59-jcdrh" event={"ID":"ebd12d3e-e201-4170-b5f0-18ed8d300682","Type":"ContainerStarted","Data":"fe5d7d4d184d4a5b3f573ac41b055a9d0517aab1c9d87f4754c693a9303373fd"} Nov 01 20:46:29.843020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:29.842425 4484 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 01 20:46:29.843020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:29.842488 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk" podUID="17927c6e-7e23-4717-be79-d2569ebae4ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 01 20:46:30.172184 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:30.172095 4484 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 20:46:30.172184 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:30.172154 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r" podUID="74fb0cfd-c53c-49da-a2c8-02cdd62a79b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 20:46:30.754469 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:30.754430 4484 generic.go:356] "Generic (PLEG): container finished" podID="ebd12d3e-e201-4170-b5f0-18ed8d300682" containerID="e85f09c1a2b24dd350d7c3f710bc7d2d4e58a9068a6918438c51dc72b0a4e239" exitCode=0 Nov 01 20:46:30.754853 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:30.754515 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fcaf59-jcdrh" event={"ID":"ebd12d3e-e201-4170-b5f0-18ed8d300682","Type":"ContainerDied","Data":"e85f09c1a2b24dd350d7c3f710bc7d2d4e58a9068a6918438c51dc72b0a4e239"} Nov 01 20:46:30.832016 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:30.831574 4484 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 01 20:46:30.832016 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:30.831653 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk" podUID="17927c6e-7e23-4717-be79-d2569ebae4ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 01 20:46:31.171918 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:31.171876 4484 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 20:46:31.172132 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:31.171958 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r" podUID="74fb0cfd-c53c-49da-a2c8-02cdd62a79b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 20:46:31.959788 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:31.959759 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fcaf59-jcdrh" Nov 01 20:46:31.971929 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:31.971896 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebd12d3e-e201-4170-b5f0-18ed8d300682-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebd12d3e-e201-4170-b5f0-18ed8d300682-maas-api-xks-ca\") pod \"ebd12d3e-e201-4170-b5f0-18ed8d300682\" (UID: \"ebd12d3e-e201-4170-b5f0-18ed8d300682\") " Nov 01 20:46:31.972088 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:31.972041 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebd12d3e-e201-4170-b5f0-18ed8d300682-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebd12d3e-e201-4170-b5f0-18ed8d300682-maas-api-openshift-ca\") pod \"ebd12d3e-e201-4170-b5f0-18ed8d300682\" (UID: \"ebd12d3e-e201-4170-b5f0-18ed8d300682\") " Nov 01 20:46:31.972289 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:31.972265 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebd12d3e-e201-4170-b5f0-18ed8d300682-kube-api-access-62dwp\" (UniqueName: \"kubernetes.io/projected/ebd12d3e-e201-4170-b5f0-18ed8d300682-kube-api-access-62dwp\") pod \"ebd12d3e-e201-4170-b5f0-18ed8d300682\" (UID: \"ebd12d3e-e201-4170-b5f0-18ed8d300682\") " Nov 01 20:46:31.972476 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:31.972448 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebd12d3e-e201-4170-b5f0-18ed8d300682-maas-api-openshift-ca" pod "ebd12d3e-e201-4170-b5f0-18ed8d300682" (UID: "ebd12d3e-e201-4170-b5f0-18ed8d300682"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:46:31.972729 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:31.972708 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebd12d3e-e201-4170-b5f0-18ed8d300682-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:46:31.974154 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:31.974130 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebd12d3e-e201-4170-b5f0-18ed8d300682-maas-api-xks-ca" pod "ebd12d3e-e201-4170-b5f0-18ed8d300682" (UID: "ebd12d3e-e201-4170-b5f0-18ed8d300682"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:46:31.974240 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:31.974230 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebd12d3e-e201-4170-b5f0-18ed8d300682-kube-api-access-62dwp" pod "ebd12d3e-e201-4170-b5f0-18ed8d300682" (UID: "ebd12d3e-e201-4170-b5f0-18ed8d300682"). InnerVolumeSpecName "kube-api-access-62dwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:46:32.074597 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:32.074513 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62dwp\" (UniqueName: \"kubernetes.io/projected/ebd12d3e-e201-4170-b5f0-18ed8d300682-kube-api-access-62dwp\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:46:32.074597 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:32.074553 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebd12d3e-e201-4170-b5f0-18ed8d300682-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:46:32.397514 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:32.397425 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk" Nov 01 20:46:32.398513 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:32.398183 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk" Nov 01 20:46:32.399175 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:32.399156 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk" Nov 01 20:46:32.772571 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:32.772546 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fcaf59-jcdrh" Nov 01 20:46:32.772749 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:32.772584 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fcaf59-jcdrh" event={"ID":"ebd12d3e-e201-4170-b5f0-18ed8d300682","Type":"ContainerDied","Data":"fe5d7d4d184d4a5b3f573ac41b055a9d0517aab1c9d87f4754c693a9303373fd"} Nov 01 20:46:32.772749 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:32.772629 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe5d7d4d184d4a5b3f573ac41b055a9d0517aab1c9d87f4754c693a9303373fd" Nov 01 20:46:32.863891 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:32.863860 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r" Nov 01 20:46:32.864683 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:32.864653 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r" Nov 01 20:46:32.865317 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:32.865288 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r" Nov 01 20:46:34.042883 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:34.042835 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:46:38.184156 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:38.184107 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:46:38.342811 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:38.342771 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:46:38.862781 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:38.862674 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:46:39.416886 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:39.416850 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ce9c445-gwxch"] Nov 01 20:46:39.417450 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:39.417435 4484 cpu_manager.go:414] "containerMap: removing container" podUID="ebd12d3e-e201-4170-b5f0-18ed8d300682" containerName="revoke-keys" Nov 01 20:46:39.417499 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:39.417450 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebd12d3e-e201-4170-b5f0-18ed8d300682" containerName="revoke-keys" Nov 01 20:46:39.417561 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:39.417551 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebd12d3e-e201-4170-b5f0-18ed8d300682" containerName="revoke-keys" Nov 01 20:46:39.420840 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:39.420824 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce9c445-gwxch" Nov 01 20:46:39.470820 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:39.470783 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bvl6\" (UniqueName: \"kubernetes.io/projected/3ed3c297-00e9-48ed-992e-84df9d758a06-kube-api-access-5bvl6\") pod \"maas-api-revoke-sub-1ce9c445-gwxch\" (UID: \"3ed3c297-00e9-48ed-992e-84df9d758a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce9c445-gwxch" Nov 01 20:46:39.471849 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:39.471142 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ed3c297-00e9-48ed-992e-84df9d758a06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ce9c445-gwxch\" (UID: \"3ed3c297-00e9-48ed-992e-84df9d758a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce9c445-gwxch" Nov 01 20:46:39.471849 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:39.471332 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ed3c297-00e9-48ed-992e-84df9d758a06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ce9c445-gwxch\" (UID: \"3ed3c297-00e9-48ed-992e-84df9d758a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce9c445-gwxch" Nov 01 20:46:39.477391 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:39.477363 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ce9c445-gwxch"] Nov 01 20:46:39.523504 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:39.523468 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:46:39.523504 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:39.523491 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:46:39.523504 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:39.523504 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 20:46:39.524970 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:39.524951 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:46:39.573144 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:39.573106 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ed3c297-00e9-48ed-992e-84df9d758a06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ce9c445-gwxch\" (UID: \"3ed3c297-00e9-48ed-992e-84df9d758a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce9c445-gwxch" Nov 01 20:46:39.573322 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:39.573186 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ed3c297-00e9-48ed-992e-84df9d758a06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ce9c445-gwxch\" (UID: \"3ed3c297-00e9-48ed-992e-84df9d758a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce9c445-gwxch" Nov 01 20:46:39.573322 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:39.573241 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bvl6\" (UniqueName: \"kubernetes.io/projected/3ed3c297-00e9-48ed-992e-84df9d758a06-kube-api-access-5bvl6\") pod \"maas-api-revoke-sub-1ce9c445-gwxch\" (UID: \"3ed3c297-00e9-48ed-992e-84df9d758a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce9c445-gwxch" Nov 01 20:46:39.573925 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:39.573889 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ed3c297-00e9-48ed-992e-84df9d758a06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ce9c445-gwxch\" (UID: \"3ed3c297-00e9-48ed-992e-84df9d758a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce9c445-gwxch" Nov 01 20:46:39.575693 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:39.575672 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ed3c297-00e9-48ed-992e-84df9d758a06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ce9c445-gwxch\" (UID: \"3ed3c297-00e9-48ed-992e-84df9d758a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce9c445-gwxch" Nov 01 20:46:39.630745 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:39.630717 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bvl6\" (UniqueName: \"kubernetes.io/projected/3ed3c297-00e9-48ed-992e-84df9d758a06-kube-api-access-5bvl6\") pod \"maas-api-revoke-sub-1ce9c445-gwxch\" (UID: \"3ed3c297-00e9-48ed-992e-84df9d758a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce9c445-gwxch" Nov 01 20:46:39.734201 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:39.734116 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce9c445-gwxch" Nov 01 20:46:40.114166 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:46:40.114133 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ed3c297_00e9_48ed_992e_84df9d758a06.slice/crio-30349934872cb8e75f7c05c551a5ce8db8c841d0ded1659c15770796c48d52fc WatchSource:0}: Error finding container 30349934872cb8e75f7c05c551a5ce8db8c841d0ded1659c15770796c48d52fc: Status 404 returned error can't find the container with id 30349934872cb8e75f7c05c551a5ce8db8c841d0ded1659c15770796c48d52fc Nov 01 20:46:40.114784 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:40.114756 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ce9c445-gwxch"] Nov 01 20:46:40.842320 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:40.842282 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce9c445-gwxch" event={"ID":"3ed3c297-00e9-48ed-992e-84df9d758a06","Type":"ContainerStarted","Data":"a242e4e528b1023c50568857d19fb85db06919e0b1c187e1f0d626ea9c4d1308"} Nov 01 20:46:40.842750 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:40.842343 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce9c445-gwxch" event={"ID":"3ed3c297-00e9-48ed-992e-84df9d758a06","Type":"ContainerStarted","Data":"30349934872cb8e75f7c05c551a5ce8db8c841d0ded1659c15770796c48d52fc"} Nov 01 20:46:41.876137 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:41.876100 4484 generic.go:356] "Generic (PLEG): container finished" podID="3ed3c297-00e9-48ed-992e-84df9d758a06" containerID="a242e4e528b1023c50568857d19fb85db06919e0b1c187e1f0d626ea9c4d1308" exitCode=0 Nov 01 20:46:41.876699 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:41.876519 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce9c445-gwxch" event={"ID":"3ed3c297-00e9-48ed-992e-84df9d758a06","Type":"ContainerDied","Data":"a242e4e528b1023c50568857d19fb85db06919e0b1c187e1f0d626ea9c4d1308"} Nov 01 20:46:43.035497 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:43.035467 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce9c445-gwxch" Nov 01 20:46:43.160157 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:43.160076 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ed3c297-00e9-48ed-992e-84df9d758a06-kube-api-access-5bvl6\" (UniqueName: \"kubernetes.io/projected/3ed3c297-00e9-48ed-992e-84df9d758a06-kube-api-access-5bvl6\") pod \"3ed3c297-00e9-48ed-992e-84df9d758a06\" (UID: \"3ed3c297-00e9-48ed-992e-84df9d758a06\") " Nov 01 20:46:43.160157 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:43.160117 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ed3c297-00e9-48ed-992e-84df9d758a06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ed3c297-00e9-48ed-992e-84df9d758a06-maas-api-openshift-ca\") pod \"3ed3c297-00e9-48ed-992e-84df9d758a06\" (UID: \"3ed3c297-00e9-48ed-992e-84df9d758a06\") " Nov 01 20:46:43.160376 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:43.160348 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ed3c297-00e9-48ed-992e-84df9d758a06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ed3c297-00e9-48ed-992e-84df9d758a06-maas-api-xks-ca\") pod \"3ed3c297-00e9-48ed-992e-84df9d758a06\" (UID: \"3ed3c297-00e9-48ed-992e-84df9d758a06\") " Nov 01 20:46:43.160616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:43.160582 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ed3c297-00e9-48ed-992e-84df9d758a06-maas-api-openshift-ca" pod "3ed3c297-00e9-48ed-992e-84df9d758a06" (UID: "3ed3c297-00e9-48ed-992e-84df9d758a06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:46:43.160845 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:43.160817 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ed3c297-00e9-48ed-992e-84df9d758a06-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:46:43.162498 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:43.162465 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ed3c297-00e9-48ed-992e-84df9d758a06-kube-api-access-5bvl6" pod "3ed3c297-00e9-48ed-992e-84df9d758a06" (UID: "3ed3c297-00e9-48ed-992e-84df9d758a06"). InnerVolumeSpecName "kube-api-access-5bvl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:46:43.162816 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:43.162792 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ed3c297-00e9-48ed-992e-84df9d758a06-maas-api-xks-ca" pod "3ed3c297-00e9-48ed-992e-84df9d758a06" (UID: "3ed3c297-00e9-48ed-992e-84df9d758a06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:46:43.261538 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:43.261503 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ed3c297-00e9-48ed-992e-84df9d758a06-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:46:43.261538 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:43.261532 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bvl6\" (UniqueName: \"kubernetes.io/projected/3ed3c297-00e9-48ed-992e-84df9d758a06-kube-api-access-5bvl6\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:46:43.691059 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:43.690927 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:46:43.891470 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:43.891439 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce9c445-gwxch" Nov 01 20:46:43.891651 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:43.891474 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce9c445-gwxch" event={"ID":"3ed3c297-00e9-48ed-992e-84df9d758a06","Type":"ContainerDied","Data":"30349934872cb8e75f7c05c551a5ce8db8c841d0ded1659c15770796c48d52fc"} Nov 01 20:46:43.891651 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:43.891517 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30349934872cb8e75f7c05c551a5ce8db8c841d0ded1659c15770796c48d52fc" Nov 01 20:46:45.767682 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:46:45.767645 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 20:46:48.443515 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:48.443471 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:46:52.145073 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:52.145044 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr"] Nov 01 20:46:52.145682 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:52.145667 4484 cpu_manager.go:414] "containerMap: removing container" podUID="3ed3c297-00e9-48ed-992e-84df9d758a06" containerName="revoke-keys" Nov 01 20:46:52.145734 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:52.145684 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ed3c297-00e9-48ed-992e-84df9d758a06" containerName="revoke-keys" Nov 01 20:46:52.145801 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:52.145792 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ed3c297-00e9-48ed-992e-84df9d758a06" containerName="revoke-keys" Nov 01 20:46:52.150570 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:52.150548 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr" Nov 01 20:46:52.218927 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:52.218889 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43443369-5e6a-4b4b-ac9d-acbb320656c1-istio-envoy\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr\" (UID: \"43443369-5e6a-4b4b-ac9d-acbb320656c1\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr" Nov 01 20:46:52.219139 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:52.218942 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43443369-5e6a-4b4b-ac9d-acbb320656c1-istio-data\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr\" (UID: \"43443369-5e6a-4b4b-ac9d-acbb320656c1\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr" Nov 01 20:46:52.219139 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:52.219094 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43443369-5e6a-4b4b-ac9d-acbb320656c1-istiod-ca-cert\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr\" (UID: \"43443369-5e6a-4b4b-ac9d-acbb320656c1\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr" Nov 01 20:46:52.219247 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:52.219203 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43443369-5e6a-4b4b-ac9d-acbb320656c1-istio-podinfo\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr\" (UID: \"43443369-5e6a-4b4b-ac9d-acbb320656c1\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr" Nov 01 20:46:52.219296 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:52.219243 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43443369-5e6a-4b4b-ac9d-acbb320656c1-istio-token\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr\" (UID: \"43443369-5e6a-4b4b-ac9d-acbb320656c1\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr" Nov 01 20:46:52.219296 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:52.219271 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43443369-5e6a-4b4b-ac9d-acbb320656c1-credential-socket\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr\" (UID: \"43443369-5e6a-4b4b-ac9d-acbb320656c1\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr" Nov 01 20:46:52.219423 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:52.219399 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43443369-5e6a-4b4b-ac9d-acbb320656c1-workload-socket\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr\" (UID: \"43443369-5e6a-4b4b-ac9d-acbb320656c1\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr" Nov 01 20:46:52.219479 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:52.219437 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9llj\" (UniqueName: \"kubernetes.io/projected/43443369-5e6a-4b4b-ac9d-acbb320656c1-kube-api-access-t9llj\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr\" (UID: \"43443369-5e6a-4b4b-ac9d-acbb320656c1\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr" Nov 01 20:46:52.219479 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:52.219459 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43443369-5e6a-4b4b-ac9d-acbb320656c1-workload-certs\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr\" (UID: \"43443369-5e6a-4b4b-ac9d-acbb320656c1\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr" Nov 01 20:46:52.256136 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:52.256093 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr"] Nov 01 20:46:52.320624 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:52.320573 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43443369-5e6a-4b4b-ac9d-acbb320656c1-istio-podinfo\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr\" (UID: \"43443369-5e6a-4b4b-ac9d-acbb320656c1\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr" Nov 01 20:46:52.320813 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:52.320664 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43443369-5e6a-4b4b-ac9d-acbb320656c1-istio-token\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr\" (UID: \"43443369-5e6a-4b4b-ac9d-acbb320656c1\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr" Nov 01 20:46:52.320813 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:52.320707 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43443369-5e6a-4b4b-ac9d-acbb320656c1-credential-socket\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr\" (UID: \"43443369-5e6a-4b4b-ac9d-acbb320656c1\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr" Nov 01 20:46:52.320813 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:52.320786 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43443369-5e6a-4b4b-ac9d-acbb320656c1-workload-socket\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr\" (UID: \"43443369-5e6a-4b4b-ac9d-acbb320656c1\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr" Nov 01 20:46:52.320983 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:52.320820 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9llj\" (UniqueName: \"kubernetes.io/projected/43443369-5e6a-4b4b-ac9d-acbb320656c1-kube-api-access-t9llj\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr\" (UID: \"43443369-5e6a-4b4b-ac9d-acbb320656c1\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr" Nov 01 20:46:52.320983 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:52.320858 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43443369-5e6a-4b4b-ac9d-acbb320656c1-workload-certs\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr\" (UID: \"43443369-5e6a-4b4b-ac9d-acbb320656c1\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr" Nov 01 20:46:52.320983 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:52.320922 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43443369-5e6a-4b4b-ac9d-acbb320656c1-istio-envoy\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr\" (UID: \"43443369-5e6a-4b4b-ac9d-acbb320656c1\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr" Nov 01 20:46:52.320983 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:52.320963 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43443369-5e6a-4b4b-ac9d-acbb320656c1-istio-data\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr\" (UID: \"43443369-5e6a-4b4b-ac9d-acbb320656c1\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr" Nov 01 20:46:52.321233 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:52.321020 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43443369-5e6a-4b4b-ac9d-acbb320656c1-istiod-ca-cert\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr\" (UID: \"43443369-5e6a-4b4b-ac9d-acbb320656c1\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr" Nov 01 20:46:52.321608 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:52.321583 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43443369-5e6a-4b4b-ac9d-acbb320656c1-workload-certs\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr\" (UID: \"43443369-5e6a-4b4b-ac9d-acbb320656c1\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr" Nov 01 20:46:52.321715 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:52.321583 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43443369-5e6a-4b4b-ac9d-acbb320656c1-istio-data\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr\" (UID: \"43443369-5e6a-4b4b-ac9d-acbb320656c1\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr" Nov 01 20:46:52.321858 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:52.321833 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43443369-5e6a-4b4b-ac9d-acbb320656c1-istiod-ca-cert\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr\" (UID: \"43443369-5e6a-4b4b-ac9d-acbb320656c1\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr" Nov 01 20:46:52.321934 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:52.321913 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43443369-5e6a-4b4b-ac9d-acbb320656c1-workload-socket\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr\" (UID: \"43443369-5e6a-4b4b-ac9d-acbb320656c1\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr" Nov 01 20:46:52.322193 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:52.322172 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43443369-5e6a-4b4b-ac9d-acbb320656c1-credential-socket\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr\" (UID: \"43443369-5e6a-4b4b-ac9d-acbb320656c1\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr" Nov 01 20:46:52.323887 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:52.323867 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43443369-5e6a-4b4b-ac9d-acbb320656c1-istio-envoy\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr\" (UID: \"43443369-5e6a-4b4b-ac9d-acbb320656c1\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr" Nov 01 20:46:52.323979 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:52.323934 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43443369-5e6a-4b4b-ac9d-acbb320656c1-istio-podinfo\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr\" (UID: \"43443369-5e6a-4b4b-ac9d-acbb320656c1\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr" Nov 01 20:46:52.379408 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:52.379371 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43443369-5e6a-4b4b-ac9d-acbb320656c1-istio-token\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr\" (UID: \"43443369-5e6a-4b4b-ac9d-acbb320656c1\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr" Nov 01 20:46:52.379592 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:52.379527 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9llj\" (UniqueName: \"kubernetes.io/projected/43443369-5e6a-4b4b-ac9d-acbb320656c1-kube-api-access-t9llj\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr\" (UID: \"43443369-5e6a-4b4b-ac9d-acbb320656c1\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr" Nov 01 20:46:52.464318 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:52.464234 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr" Nov 01 20:46:52.511561 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:52.511527 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:46:52.618877 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:52.611056 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vz6nd"] Nov 01 20:46:52.618877 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:52.611317 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vz6nd" podUID="59bfe493-3735-4e0f-bfca-03a345fcbeea" containerName="istio-proxy" containerID="cri-o://1fb90a461fa142b7e8c2bd7efc618ee56daa0fb7b2b7e069f2faeefb62e0c11c" gracePeriod=30 Nov 01 20:46:52.906879 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:52.906852 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr"] Nov 01 20:46:52.907178 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:46:52.907155 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43443369_5e6a_4b4b_ac9d_acbb320656c1.slice/crio-041fd9f4eae5c2f129a03a8c6f21f790d83dae14c6ed47051c081bdf78d013c5 WatchSource:0}: Error finding container 041fd9f4eae5c2f129a03a8c6f21f790d83dae14c6ed47051c081bdf78d013c5: Status 404 returned error can't find the container with id 041fd9f4eae5c2f129a03a8c6f21f790d83dae14c6ed47051c081bdf78d013c5 Nov 01 20:46:52.914171 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:52.914139 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:46:52.914250 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:52.914205 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:46:52.914250 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:52.914232 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:46:52.952253 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:52.952218 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr" event={"ID":"43443369-5e6a-4b4b-ac9d-acbb320656c1","Type":"ContainerStarted","Data":"041fd9f4eae5c2f129a03a8c6f21f790d83dae14c6ed47051c081bdf78d013c5"} Nov 01 20:46:53.335258 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:53.335224 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:46:53.959911 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:53.959870 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr" event={"ID":"43443369-5e6a-4b4b-ac9d-acbb320656c1","Type":"ContainerStarted","Data":"2937525e226fa5935d72fc9212c722f405c0da4cdd57776b4a699fd7255b5013"} Nov 01 20:46:54.067864 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:54.067829 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb"] Nov 01 20:46:54.072026 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:54.071990 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb" Nov 01 20:46:54.102156 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:54.102091 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr" podStartSLOduration=2.102076676 podStartE2EDuration="2.102076676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:46:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:46:54.077658969 +0000 UTC m=+964.855299104" watchObservedRunningTime="2026-11-01 20:46:54.102076676 +0000 UTC m=+964.879716810" Nov 01 20:46:54.136647 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:54.136606 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-1a1e9e-openshift-default-dockercfg-xvvfj\"" Nov 01 20:46:54.137010 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:54.136975 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb"] Nov 01 20:46:54.155240 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:54.155200 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4ca8d2-0b86-4eba-8920-d9633474d6d2-credential-socket\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb\" (UID: \"da4ca8d2-0b86-4eba-8920-d9633474d6d2\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb" Nov 01 20:46:54.155401 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:54.155336 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4ca8d2-0b86-4eba-8920-d9633474d6d2-workload-socket\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb\" (UID: \"da4ca8d2-0b86-4eba-8920-d9633474d6d2\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb" Nov 01 20:46:54.155492 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:54.155399 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da4ca8d2-0b86-4eba-8920-d9633474d6d2-istio-envoy\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb\" (UID: \"da4ca8d2-0b86-4eba-8920-d9633474d6d2\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb" Nov 01 20:46:54.155570 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:54.155495 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da4ca8d2-0b86-4eba-8920-d9633474d6d2-workload-certs\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb\" (UID: \"da4ca8d2-0b86-4eba-8920-d9633474d6d2\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb" Nov 01 20:46:54.155570 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:54.155531 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da4ca8d2-0b86-4eba-8920-d9633474d6d2-istiod-ca-cert\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb\" (UID: \"da4ca8d2-0b86-4eba-8920-d9633474d6d2\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb" Nov 01 20:46:54.155688 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:54.155610 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da4ca8d2-0b86-4eba-8920-d9633474d6d2-istio-podinfo\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb\" (UID: \"da4ca8d2-0b86-4eba-8920-d9633474d6d2\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb" Nov 01 20:46:54.155786 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:54.155687 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da4ca8d2-0b86-4eba-8920-d9633474d6d2-istio-token\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb\" (UID: \"da4ca8d2-0b86-4eba-8920-d9633474d6d2\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb" Nov 01 20:46:54.155786 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:54.155740 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqwxz\" (UniqueName: \"kubernetes.io/projected/da4ca8d2-0b86-4eba-8920-d9633474d6d2-kube-api-access-dqwxz\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb\" (UID: \"da4ca8d2-0b86-4eba-8920-d9633474d6d2\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb" Nov 01 20:46:54.155786 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:54.155778 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da4ca8d2-0b86-4eba-8920-d9633474d6d2-istio-data\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb\" (UID: \"da4ca8d2-0b86-4eba-8920-d9633474d6d2\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb" Nov 01 20:46:54.261375 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:54.261318 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da4ca8d2-0b86-4eba-8920-d9633474d6d2-istio-podinfo\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb\" (UID: \"da4ca8d2-0b86-4eba-8920-d9633474d6d2\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb" Nov 01 20:46:54.267824 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:54.267796 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da4ca8d2-0b86-4eba-8920-d9633474d6d2-istio-token\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb\" (UID: \"da4ca8d2-0b86-4eba-8920-d9633474d6d2\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb" Nov 01 20:46:54.268049 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:54.268029 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqwxz\" (UniqueName: \"kubernetes.io/projected/da4ca8d2-0b86-4eba-8920-d9633474d6d2-kube-api-access-dqwxz\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb\" (UID: \"da4ca8d2-0b86-4eba-8920-d9633474d6d2\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb" Nov 01 20:46:54.268198 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:54.268183 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da4ca8d2-0b86-4eba-8920-d9633474d6d2-istio-data\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb\" (UID: \"da4ca8d2-0b86-4eba-8920-d9633474d6d2\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb" Nov 01 20:46:54.268368 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:54.268353 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4ca8d2-0b86-4eba-8920-d9633474d6d2-credential-socket\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb\" (UID: \"da4ca8d2-0b86-4eba-8920-d9633474d6d2\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb" Nov 01 20:46:54.268613 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:54.268572 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4ca8d2-0b86-4eba-8920-d9633474d6d2-workload-socket\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb\" (UID: \"da4ca8d2-0b86-4eba-8920-d9633474d6d2\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb" Nov 01 20:46:54.268792 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:54.268777 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da4ca8d2-0b86-4eba-8920-d9633474d6d2-istio-envoy\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb\" (UID: \"da4ca8d2-0b86-4eba-8920-d9633474d6d2\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb" Nov 01 20:46:54.269018 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:54.268974 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da4ca8d2-0b86-4eba-8920-d9633474d6d2-workload-certs\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb\" (UID: \"da4ca8d2-0b86-4eba-8920-d9633474d6d2\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb" Nov 01 20:46:54.269148 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:54.269130 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da4ca8d2-0b86-4eba-8920-d9633474d6d2-istiod-ca-cert\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb\" (UID: \"da4ca8d2-0b86-4eba-8920-d9633474d6d2\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb" Nov 01 20:46:54.270210 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:54.270186 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da4ca8d2-0b86-4eba-8920-d9633474d6d2-istiod-ca-cert\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb\" (UID: \"da4ca8d2-0b86-4eba-8920-d9633474d6d2\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb" Nov 01 20:46:54.272399 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:54.271525 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4ca8d2-0b86-4eba-8920-d9633474d6d2-workload-socket\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb\" (UID: \"da4ca8d2-0b86-4eba-8920-d9633474d6d2\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb" Nov 01 20:46:54.272597 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:54.271776 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da4ca8d2-0b86-4eba-8920-d9633474d6d2-istio-data\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb\" (UID: \"da4ca8d2-0b86-4eba-8920-d9633474d6d2\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb" Nov 01 20:46:54.272700 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:54.272069 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da4ca8d2-0b86-4eba-8920-d9633474d6d2-workload-certs\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb\" (UID: \"da4ca8d2-0b86-4eba-8920-d9633474d6d2\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb" Nov 01 20:46:54.272799 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:54.272335 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4ca8d2-0b86-4eba-8920-d9633474d6d2-credential-socket\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb\" (UID: \"da4ca8d2-0b86-4eba-8920-d9633474d6d2\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb" Nov 01 20:46:54.309032 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:54.298950 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da4ca8d2-0b86-4eba-8920-d9633474d6d2-istio-envoy\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb\" (UID: \"da4ca8d2-0b86-4eba-8920-d9633474d6d2\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb" Nov 01 20:46:54.309032 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:54.299442 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da4ca8d2-0b86-4eba-8920-d9633474d6d2-istio-podinfo\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb\" (UID: \"da4ca8d2-0b86-4eba-8920-d9633474d6d2\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb" Nov 01 20:46:54.336849 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:54.336606 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqwxz\" (UniqueName: \"kubernetes.io/projected/da4ca8d2-0b86-4eba-8920-d9633474d6d2-kube-api-access-dqwxz\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb\" (UID: \"da4ca8d2-0b86-4eba-8920-d9633474d6d2\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb" Nov 01 20:46:54.337348 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:54.337210 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da4ca8d2-0b86-4eba-8920-d9633474d6d2-istio-token\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb\" (UID: \"da4ca8d2-0b86-4eba-8920-d9633474d6d2\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb" Nov 01 20:46:54.387428 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:54.387390 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb" Nov 01 20:46:54.464820 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:54.464784 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr" Nov 01 20:46:54.466660 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:54.466634 4484 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 01 20:46:54.466796 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:54.466714 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr" podUID="43443369-5e6a-4b4b-ac9d-acbb320656c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 01 20:46:54.849098 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:54.849070 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb"] Nov 01 20:46:54.849296 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:46:54.849273 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda4ca8d2_0b86_4eba_8920_d9633474d6d2.slice/crio-291edce2e389684abbce9438b178f721ac37a310f38c78610ba9a1d6164dd0d4 WatchSource:0}: Error finding container 291edce2e389684abbce9438b178f721ac37a310f38c78610ba9a1d6164dd0d4: Status 404 returned error can't find the container with id 291edce2e389684abbce9438b178f721ac37a310f38c78610ba9a1d6164dd0d4 Nov 01 20:46:54.852115 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:54.852080 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:46:54.852211 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:54.852144 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:46:54.852211 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:54.852170 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:46:54.970498 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:54.970465 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb" event={"ID":"da4ca8d2-0b86-4eba-8920-d9633474d6d2","Type":"ContainerStarted","Data":"1cdf7483f836ce3134a9ad04106b8bff6818378c4298844fab81ad0dfa577a41"} Nov 01 20:46:54.970654 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:54.970509 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb" event={"ID":"da4ca8d2-0b86-4eba-8920-d9633474d6d2","Type":"ContainerStarted","Data":"291edce2e389684abbce9438b178f721ac37a310f38c78610ba9a1d6164dd0d4"} Nov 01 20:46:55.000550 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:55.000517 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bafb9fe-8vwzd"] Nov 01 20:46:55.031018 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:55.023323 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bafb9fe-8vwzd" Nov 01 20:46:55.066013 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:55.065951 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bafb9fe-8vwzd"] Nov 01 20:46:55.077209 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:55.077176 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:46:55.077389 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:55.077327 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:46:55.077389 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:55.077387 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:46:55.077507 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:55.077499 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 20:46:55.142128 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:55.141803 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb" podStartSLOduration=1.141779979 podStartE2EDuration="1.141779979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:46:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:46:55.101387448 +0000 UTC m=+965.879027584" watchObservedRunningTime="2026-11-01 20:46:55.141779979 +0000 UTC m=+965.919420115" Nov 01 20:46:55.187033 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:55.186957 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwbzk\" (UniqueName: \"kubernetes.io/projected/8257bace-423a-4c35-bea7-0d046eef7144-kube-api-access-rwbzk\") pod \"maas-api-revoke-sub-7bafb9fe-8vwzd\" (UID: \"8257bace-423a-4c35-bea7-0d046eef7144\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bafb9fe-8vwzd" Nov 01 20:46:55.187201 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:55.187174 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8257bace-423a-4c35-bea7-0d046eef7144-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7bafb9fe-8vwzd\" (UID: \"8257bace-423a-4c35-bea7-0d046eef7144\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bafb9fe-8vwzd" Nov 01 20:46:55.187254 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:55.187233 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8257bace-423a-4c35-bea7-0d046eef7144-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7bafb9fe-8vwzd\" (UID: \"8257bace-423a-4c35-bea7-0d046eef7144\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bafb9fe-8vwzd" Nov 01 20:46:55.288402 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:55.288366 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8257bace-423a-4c35-bea7-0d046eef7144-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7bafb9fe-8vwzd\" (UID: \"8257bace-423a-4c35-bea7-0d046eef7144\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bafb9fe-8vwzd" Nov 01 20:46:55.288583 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:55.288427 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8257bace-423a-4c35-bea7-0d046eef7144-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7bafb9fe-8vwzd\" (UID: \"8257bace-423a-4c35-bea7-0d046eef7144\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bafb9fe-8vwzd" Nov 01 20:46:55.288583 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:55.288496 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwbzk\" (UniqueName: \"kubernetes.io/projected/8257bace-423a-4c35-bea7-0d046eef7144-kube-api-access-rwbzk\") pod \"maas-api-revoke-sub-7bafb9fe-8vwzd\" (UID: \"8257bace-423a-4c35-bea7-0d046eef7144\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bafb9fe-8vwzd" Nov 01 20:46:55.289173 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:55.289151 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8257bace-423a-4c35-bea7-0d046eef7144-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7bafb9fe-8vwzd\" (UID: \"8257bace-423a-4c35-bea7-0d046eef7144\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bafb9fe-8vwzd" Nov 01 20:46:55.290797 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:55.290773 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8257bace-423a-4c35-bea7-0d046eef7144-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7bafb9fe-8vwzd\" (UID: \"8257bace-423a-4c35-bea7-0d046eef7144\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bafb9fe-8vwzd" Nov 01 20:46:55.346084 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:55.346052 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwbzk\" (UniqueName: \"kubernetes.io/projected/8257bace-423a-4c35-bea7-0d046eef7144-kube-api-access-rwbzk\") pod \"maas-api-revoke-sub-7bafb9fe-8vwzd\" (UID: \"8257bace-423a-4c35-bea7-0d046eef7144\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bafb9fe-8vwzd" Nov 01 20:46:55.388663 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:55.388630 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb" Nov 01 20:46:55.390677 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:55.390652 4484 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 01 20:46:55.390821 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:55.390706 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb" podUID="da4ca8d2-0b86-4eba-8920-d9633474d6d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 01 20:46:55.465394 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:55.465309 4484 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 01 20:46:55.465394 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:55.465363 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr" podUID="43443369-5e6a-4b4b-ac9d-acbb320656c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 01 20:46:55.653065 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:55.653034 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bafb9fe-8vwzd" Nov 01 20:46:56.082243 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:56.082163 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bafb9fe-8vwzd"] Nov 01 20:46:56.185307 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:56.184537 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:46:56.388009 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:56.387919 4484 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 01 20:46:56.388420 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:56.388390 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb" podUID="da4ca8d2-0b86-4eba-8920-d9633474d6d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 01 20:46:56.471018 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:56.467374 4484 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 01 20:46:56.471018 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:56.467433 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr" podUID="43443369-5e6a-4b4b-ac9d-acbb320656c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 01 20:46:56.982712 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:56.982621 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bafb9fe-8vwzd" event={"ID":"8257bace-423a-4c35-bea7-0d046eef7144","Type":"ContainerStarted","Data":"654837a9a3c05ed5f4a2a473193955a561b5c829a8bb07c49bca455acd95747a"} Nov 01 20:46:56.982712 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:56.982678 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bafb9fe-8vwzd" event={"ID":"8257bace-423a-4c35-bea7-0d046eef7144","Type":"ContainerStarted","Data":"a6e2029c4f3c7d5e79966e9d9fee880db12017e40efcf499ea8fed9fcc3ea1c1"} Nov 01 20:46:57.114652 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:57.114576 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bafb9fe-8vwzd" podStartSLOduration=3.114555401 podStartE2EDuration="3.114555401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:46:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:46:57.113581667 +0000 UTC m=+967.891221795" watchObservedRunningTime="2026-11-01 20:46:57.114555401 +0000 UTC m=+967.892195539" Nov 01 20:46:57.402215 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:57.397836 4484 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 01 20:46:57.402215 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:57.397905 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb" podUID="da4ca8d2-0b86-4eba-8920-d9633474d6d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 01 20:46:57.465164 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:57.465126 4484 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 01 20:46:57.465328 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:57.465186 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr" podUID="43443369-5e6a-4b4b-ac9d-acbb320656c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 01 20:46:57.996015 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:57.995969 4484 generic.go:356] "Generic (PLEG): container finished" podID="59bfe493-3735-4e0f-bfca-03a345fcbeea" containerID="1fb90a461fa142b7e8c2bd7efc618ee56daa0fb7b2b7e069f2faeefb62e0c11c" exitCode=0 Nov 01 20:46:57.996161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:57.996075 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vz6nd" event={"ID":"59bfe493-3735-4e0f-bfca-03a345fcbeea","Type":"ContainerDied","Data":"1fb90a461fa142b7e8c2bd7efc618ee56daa0fb7b2b7e069f2faeefb62e0c11c"} Nov 01 20:46:57.996161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:57.996113 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vz6nd" event={"ID":"59bfe493-3735-4e0f-bfca-03a345fcbeea","Type":"ContainerDied","Data":"6686861e437c5ca59e699cf18c43239fcb159c0d656da1833b2229527efcd5d0"} Nov 01 20:46:57.996161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:57.996128 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6686861e437c5ca59e699cf18c43239fcb159c0d656da1833b2229527efcd5d0" Nov 01 20:46:57.998248 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:57.998221 4484 generic.go:356] "Generic (PLEG): container finished" podID="8257bace-423a-4c35-bea7-0d046eef7144" containerID="654837a9a3c05ed5f4a2a473193955a561b5c829a8bb07c49bca455acd95747a" exitCode=0 Nov 01 20:46:57.998415 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:57.998401 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bafb9fe-8vwzd" event={"ID":"8257bace-423a-4c35-bea7-0d046eef7144","Type":"ContainerDied","Data":"654837a9a3c05ed5f4a2a473193955a561b5c829a8bb07c49bca455acd95747a"} Nov 01 20:46:57.998647 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:57.998631 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vz6nd" Nov 01 20:46:58.078612 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:58.078573 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59bfe493-3735-4e0f-bfca-03a345fcbeea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59bfe493-3735-4e0f-bfca-03a345fcbeea-istio-data\") pod \"59bfe493-3735-4e0f-bfca-03a345fcbeea\" (UID: \"59bfe493-3735-4e0f-bfca-03a345fcbeea\") " Nov 01 20:46:58.078788 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:58.078660 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59bfe493-3735-4e0f-bfca-03a345fcbeea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59bfe493-3735-4e0f-bfca-03a345fcbeea-workload-certs\") pod \"59bfe493-3735-4e0f-bfca-03a345fcbeea\" (UID: \"59bfe493-3735-4e0f-bfca-03a345fcbeea\") " Nov 01 20:46:58.078788 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:58.078684 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59bfe493-3735-4e0f-bfca-03a345fcbeea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59bfe493-3735-4e0f-bfca-03a345fcbeea-istio-podinfo\") pod \"59bfe493-3735-4e0f-bfca-03a345fcbeea\" (UID: \"59bfe493-3735-4e0f-bfca-03a345fcbeea\") " Nov 01 20:46:58.078788 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:58.078739 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59bfe493-3735-4e0f-bfca-03a345fcbeea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59bfe493-3735-4e0f-bfca-03a345fcbeea-istiod-ca-cert\") pod \"59bfe493-3735-4e0f-bfca-03a345fcbeea\" (UID: \"59bfe493-3735-4e0f-bfca-03a345fcbeea\") " Nov 01 20:46:58.078893 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:58.078791 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59bfe493-3735-4e0f-bfca-03a345fcbeea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59bfe493-3735-4e0f-bfca-03a345fcbeea-istio-envoy\") pod \"59bfe493-3735-4e0f-bfca-03a345fcbeea\" (UID: \"59bfe493-3735-4e0f-bfca-03a345fcbeea\") " Nov 01 20:46:58.078893 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:58.078826 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59bfe493-3735-4e0f-bfca-03a345fcbeea-istio-token\" (UniqueName: \"kubernetes.io/projected/59bfe493-3735-4e0f-bfca-03a345fcbeea-istio-token\") pod \"59bfe493-3735-4e0f-bfca-03a345fcbeea\" (UID: \"59bfe493-3735-4e0f-bfca-03a345fcbeea\") " Nov 01 20:46:58.078893 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:58.078878 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59bfe493-3735-4e0f-bfca-03a345fcbeea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59bfe493-3735-4e0f-bfca-03a345fcbeea-workload-socket\") pod \"59bfe493-3735-4e0f-bfca-03a345fcbeea\" (UID: \"59bfe493-3735-4e0f-bfca-03a345fcbeea\") " Nov 01 20:46:58.078977 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:58.078899 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59bfe493-3735-4e0f-bfca-03a345fcbeea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59bfe493-3735-4e0f-bfca-03a345fcbeea-credential-socket\") pod \"59bfe493-3735-4e0f-bfca-03a345fcbeea\" (UID: \"59bfe493-3735-4e0f-bfca-03a345fcbeea\") " Nov 01 20:46:58.078977 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:58.078919 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59bfe493-3735-4e0f-bfca-03a345fcbeea-kube-api-access-7hbql\" (UniqueName: \"kubernetes.io/projected/59bfe493-3735-4e0f-bfca-03a345fcbeea-kube-api-access-7hbql\") pod \"59bfe493-3735-4e0f-bfca-03a345fcbeea\" (UID: \"59bfe493-3735-4e0f-bfca-03a345fcbeea\") " Nov 01 20:46:58.079355 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:58.079331 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59bfe493-3735-4e0f-bfca-03a345fcbeea-istiod-ca-cert" pod "59bfe493-3735-4e0f-bfca-03a345fcbeea" (UID: "59bfe493-3735-4e0f-bfca-03a345fcbeea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:46:58.079539 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:58.079524 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59bfe493-3735-4e0f-bfca-03a345fcbeea-istio-data" pod "59bfe493-3735-4e0f-bfca-03a345fcbeea" (UID: "59bfe493-3735-4e0f-bfca-03a345fcbeea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:46:58.079657 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:58.079645 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59bfe493-3735-4e0f-bfca-03a345fcbeea-workload-certs" pod "59bfe493-3735-4e0f-bfca-03a345fcbeea" (UID: "59bfe493-3735-4e0f-bfca-03a345fcbeea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:46:58.080138 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:58.080064 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59bfe493-3735-4e0f-bfca-03a345fcbeea-credential-socket" pod "59bfe493-3735-4e0f-bfca-03a345fcbeea" (UID: "59bfe493-3735-4e0f-bfca-03a345fcbeea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:46:58.080283 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:58.080167 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59bfe493-3735-4e0f-bfca-03a345fcbeea-workload-socket" pod "59bfe493-3735-4e0f-bfca-03a345fcbeea" (UID: "59bfe493-3735-4e0f-bfca-03a345fcbeea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:46:58.081380 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:58.081356 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59bfe493-3735-4e0f-bfca-03a345fcbeea-kube-api-access-7hbql" pod "59bfe493-3735-4e0f-bfca-03a345fcbeea" (UID: "59bfe493-3735-4e0f-bfca-03a345fcbeea"). InnerVolumeSpecName "kube-api-access-7hbql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:46:58.082233 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:58.082198 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59bfe493-3735-4e0f-bfca-03a345fcbeea-istio-envoy" pod "59bfe493-3735-4e0f-bfca-03a345fcbeea" (UID: "59bfe493-3735-4e0f-bfca-03a345fcbeea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:46:58.082759 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:58.082739 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59bfe493-3735-4e0f-bfca-03a345fcbeea-istio-podinfo" pod "59bfe493-3735-4e0f-bfca-03a345fcbeea" (UID: "59bfe493-3735-4e0f-bfca-03a345fcbeea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:46:58.082871 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:58.082767 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59bfe493-3735-4e0f-bfca-03a345fcbeea-istio-token" pod "59bfe493-3735-4e0f-bfca-03a345fcbeea" (UID: "59bfe493-3735-4e0f-bfca-03a345fcbeea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:46:58.180918 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:58.180886 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59bfe493-3735-4e0f-bfca-03a345fcbeea-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:46:58.180918 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:58.180912 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59bfe493-3735-4e0f-bfca-03a345fcbeea-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:46:58.180918 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:58.180922 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59bfe493-3735-4e0f-bfca-03a345fcbeea-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:46:58.181157 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:58.180931 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59bfe493-3735-4e0f-bfca-03a345fcbeea-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:46:58.181157 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:58.180941 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59bfe493-3735-4e0f-bfca-03a345fcbeea-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:46:58.181157 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:58.180952 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hbql\" (UniqueName: \"kubernetes.io/projected/59bfe493-3735-4e0f-bfca-03a345fcbeea-kube-api-access-7hbql\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:46:58.181157 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:58.180960 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59bfe493-3735-4e0f-bfca-03a345fcbeea-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:46:58.181157 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:58.180969 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59bfe493-3735-4e0f-bfca-03a345fcbeea-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:46:58.181157 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:58.180976 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59bfe493-3735-4e0f-bfca-03a345fcbeea-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:46:58.388543 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:58.388509 4484 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 01 20:46:58.388739 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:58.388562 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb" podUID="da4ca8d2-0b86-4eba-8920-d9633474d6d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 01 20:46:58.418026 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:58.417949 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:46:58.469474 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:58.469440 4484 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 01 20:46:58.469728 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:58.469701 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr" podUID="43443369-5e6a-4b4b-ac9d-acbb320656c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 01 20:46:59.039055 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:59.038953 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vz6nd" Nov 01 20:46:59.238913 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:59.238884 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vz6nd"] Nov 01 20:46:59.243701 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:59.243678 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bafb9fe-8vwzd" Nov 01 20:46:59.291526 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:59.291417 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vz6nd"] Nov 01 20:46:59.375576 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:59.375538 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59bfe493-3735-4e0f-bfca-03a345fcbeea" path="/var/lib/kubelet/pods/59bfe493-3735-4e0f-bfca-03a345fcbeea/volumes" Nov 01 20:46:59.388779 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:59.388742 4484 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 01 20:46:59.389058 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:59.389028 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb" podUID="da4ca8d2-0b86-4eba-8920-d9633474d6d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 01 20:46:59.396382 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:59.396354 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8257bace-423a-4c35-bea7-0d046eef7144-kube-api-access-rwbzk\" (UniqueName: \"kubernetes.io/projected/8257bace-423a-4c35-bea7-0d046eef7144-kube-api-access-rwbzk\") pod \"8257bace-423a-4c35-bea7-0d046eef7144\" (UID: \"8257bace-423a-4c35-bea7-0d046eef7144\") " Nov 01 20:46:59.396503 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:59.396471 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8257bace-423a-4c35-bea7-0d046eef7144-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8257bace-423a-4c35-bea7-0d046eef7144-maas-api-xks-ca\") pod \"8257bace-423a-4c35-bea7-0d046eef7144\" (UID: \"8257bace-423a-4c35-bea7-0d046eef7144\") " Nov 01 20:46:59.396593 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:59.396576 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8257bace-423a-4c35-bea7-0d046eef7144-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8257bace-423a-4c35-bea7-0d046eef7144-maas-api-openshift-ca\") pod \"8257bace-423a-4c35-bea7-0d046eef7144\" (UID: \"8257bace-423a-4c35-bea7-0d046eef7144\") " Nov 01 20:46:59.397404 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:59.397354 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8257bace-423a-4c35-bea7-0d046eef7144-maas-api-openshift-ca" pod "8257bace-423a-4c35-bea7-0d046eef7144" (UID: "8257bace-423a-4c35-bea7-0d046eef7144"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:46:59.441022 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:59.440125 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8257bace-423a-4c35-bea7-0d046eef7144-maas-api-xks-ca" pod "8257bace-423a-4c35-bea7-0d046eef7144" (UID: "8257bace-423a-4c35-bea7-0d046eef7144"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:46:59.441469 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:59.441439 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8257bace-423a-4c35-bea7-0d046eef7144-kube-api-access-rwbzk" pod "8257bace-423a-4c35-bea7-0d046eef7144" (UID: "8257bace-423a-4c35-bea7-0d046eef7144"). InnerVolumeSpecName "kube-api-access-rwbzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:46:59.465277 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:59.465246 4484 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 01 20:46:59.465429 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:59.465304 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr" podUID="43443369-5e6a-4b4b-ac9d-acbb320656c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 01 20:46:59.499147 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:59.499106 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwbzk\" (UniqueName: \"kubernetes.io/projected/8257bace-423a-4c35-bea7-0d046eef7144-kube-api-access-rwbzk\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:46:59.499147 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:59.499146 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8257bace-423a-4c35-bea7-0d046eef7144-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:46:59.499380 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:46:59.499161 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8257bace-423a-4c35-bea7-0d046eef7144-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:47:00.047116 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:00.047084 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bafb9fe-8vwzd" Nov 01 20:47:00.047116 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:00.047100 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bafb9fe-8vwzd" event={"ID":"8257bace-423a-4c35-bea7-0d046eef7144","Type":"ContainerDied","Data":"a6e2029c4f3c7d5e79966e9d9fee880db12017e40efcf499ea8fed9fcc3ea1c1"} Nov 01 20:47:00.047338 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:00.047158 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6e2029c4f3c7d5e79966e9d9fee880db12017e40efcf499ea8fed9fcc3ea1c1" Nov 01 20:47:00.387823 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:00.387746 4484 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 01 20:47:00.387823 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:00.387794 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb" podUID="da4ca8d2-0b86-4eba-8920-d9633474d6d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 01 20:47:00.469938 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:00.469912 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr" Nov 01 20:47:00.470518 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:00.470489 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr" Nov 01 20:47:00.471145 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:00.471126 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr" Nov 01 20:47:01.388054 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:01.388022 4484 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 01 20:47:01.388232 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:01.388071 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb" podUID="da4ca8d2-0b86-4eba-8920-d9633474d6d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 01 20:47:02.393070 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:02.393038 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb" Nov 01 20:47:02.395522 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:02.393457 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb" Nov 01 20:47:02.395522 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:02.394258 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb" Nov 01 20:47:03.226722 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:03.226690 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:47:08.046337 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:08.046297 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:47:10.978489 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:10.978450 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4"] Nov 01 20:47:10.979068 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:10.979053 4484 cpu_manager.go:414] "containerMap: removing container" podUID="8257bace-423a-4c35-bea7-0d046eef7144" containerName="revoke-keys" Nov 01 20:47:10.979121 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:10.979069 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8257bace-423a-4c35-bea7-0d046eef7144" containerName="revoke-keys" Nov 01 20:47:10.979121 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:10.979118 4484 cpu_manager.go:414] "containerMap: removing container" podUID="59bfe493-3735-4e0f-bfca-03a345fcbeea" containerName="istio-proxy" Nov 01 20:47:10.979185 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:10.979123 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59bfe493-3735-4e0f-bfca-03a345fcbeea" containerName="istio-proxy" Nov 01 20:47:10.979218 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:10.979209 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="59bfe493-3735-4e0f-bfca-03a345fcbeea" containerName="istio-proxy" Nov 01 20:47:10.979255 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:10.979218 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="8257bace-423a-4c35-bea7-0d046eef7144" containerName="revoke-keys" Nov 01 20:47:10.982669 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:10.982652 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" Nov 01 20:47:11.040613 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:11.040578 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4"] Nov 01 20:47:11.080389 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:11.080346 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70b4dacf-b3ac-46ec-99e8-a44f05587e7c-home\") pod \"distinct-w1-db0157-kserve-646d4c46b9-mqss4\" (UID: \"70b4dacf-b3ac-46ec-99e8-a44f05587e7c\") " pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" Nov 01 20:47:11.080568 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:11.080415 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sldq4\" (UniqueName: \"kubernetes.io/projected/70b4dacf-b3ac-46ec-99e8-a44f05587e7c-kube-api-access-sldq4\") pod \"distinct-w1-db0157-kserve-646d4c46b9-mqss4\" (UID: \"70b4dacf-b3ac-46ec-99e8-a44f05587e7c\") " pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" Nov 01 20:47:11.080568 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:11.080531 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70b4dacf-b3ac-46ec-99e8-a44f05587e7c-model-cache\") pod \"distinct-w1-db0157-kserve-646d4c46b9-mqss4\" (UID: \"70b4dacf-b3ac-46ec-99e8-a44f05587e7c\") " pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" Nov 01 20:47:11.080658 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:11.080570 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70b4dacf-b3ac-46ec-99e8-a44f05587e7c-tmp-dir\") pod \"distinct-w1-db0157-kserve-646d4c46b9-mqss4\" (UID: \"70b4dacf-b3ac-46ec-99e8-a44f05587e7c\") " pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" Nov 01 20:47:11.080658 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:11.080621 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70b4dacf-b3ac-46ec-99e8-a44f05587e7c-dshm\") pod \"distinct-w1-db0157-kserve-646d4c46b9-mqss4\" (UID: \"70b4dacf-b3ac-46ec-99e8-a44f05587e7c\") " pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" Nov 01 20:47:11.080754 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:11.080728 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70b4dacf-b3ac-46ec-99e8-a44f05587e7c-tls-certs\") pod \"distinct-w1-db0157-kserve-646d4c46b9-mqss4\" (UID: \"70b4dacf-b3ac-46ec-99e8-a44f05587e7c\") " pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" Nov 01 20:47:11.087284 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:11.087247 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-db0157\"/\"kube-root-ca.crt\"" Nov 01 20:47:11.087284 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:11.087248 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-db0157\"/\"distinct-w1-db0157-kserve-self-signed-certs\"" Nov 01 20:47:11.087504 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:11.087395 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-db0157\"/\"openshift-service-ca.crt\"" Nov 01 20:47:11.091510 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:11.091485 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:47:11.168659 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:11.168398 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-db0157\"/\"default-dockercfg-dssl5\"" Nov 01 20:47:11.183918 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:11.182397 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70b4dacf-b3ac-46ec-99e8-a44f05587e7c-model-cache\") pod \"distinct-w1-db0157-kserve-646d4c46b9-mqss4\" (UID: \"70b4dacf-b3ac-46ec-99e8-a44f05587e7c\") " pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" Nov 01 20:47:11.183918 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:11.182462 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70b4dacf-b3ac-46ec-99e8-a44f05587e7c-tmp-dir\") pod \"distinct-w1-db0157-kserve-646d4c46b9-mqss4\" (UID: \"70b4dacf-b3ac-46ec-99e8-a44f05587e7c\") " pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" Nov 01 20:47:11.183918 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:11.182529 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70b4dacf-b3ac-46ec-99e8-a44f05587e7c-dshm\") pod \"distinct-w1-db0157-kserve-646d4c46b9-mqss4\" (UID: \"70b4dacf-b3ac-46ec-99e8-a44f05587e7c\") " pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" Nov 01 20:47:11.183918 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:11.182581 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70b4dacf-b3ac-46ec-99e8-a44f05587e7c-tls-certs\") pod \"distinct-w1-db0157-kserve-646d4c46b9-mqss4\" (UID: \"70b4dacf-b3ac-46ec-99e8-a44f05587e7c\") " pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" Nov 01 20:47:11.183918 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:11.182681 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70b4dacf-b3ac-46ec-99e8-a44f05587e7c-home\") pod \"distinct-w1-db0157-kserve-646d4c46b9-mqss4\" (UID: \"70b4dacf-b3ac-46ec-99e8-a44f05587e7c\") " pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" Nov 01 20:47:11.183918 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:11.182751 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sldq4\" (UniqueName: \"kubernetes.io/projected/70b4dacf-b3ac-46ec-99e8-a44f05587e7c-kube-api-access-sldq4\") pod \"distinct-w1-db0157-kserve-646d4c46b9-mqss4\" (UID: \"70b4dacf-b3ac-46ec-99e8-a44f05587e7c\") " pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" Nov 01 20:47:11.183918 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:11.183470 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70b4dacf-b3ac-46ec-99e8-a44f05587e7c-model-cache\") pod \"distinct-w1-db0157-kserve-646d4c46b9-mqss4\" (UID: \"70b4dacf-b3ac-46ec-99e8-a44f05587e7c\") " pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" Nov 01 20:47:11.183918 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:11.183665 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70b4dacf-b3ac-46ec-99e8-a44f05587e7c-tmp-dir\") pod \"distinct-w1-db0157-kserve-646d4c46b9-mqss4\" (UID: \"70b4dacf-b3ac-46ec-99e8-a44f05587e7c\") " pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" Nov 01 20:47:11.184927 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:11.184894 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70b4dacf-b3ac-46ec-99e8-a44f05587e7c-home\") pod \"distinct-w1-db0157-kserve-646d4c46b9-mqss4\" (UID: \"70b4dacf-b3ac-46ec-99e8-a44f05587e7c\") " pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" Nov 01 20:47:11.186469 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:11.186425 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70b4dacf-b3ac-46ec-99e8-a44f05587e7c-dshm\") pod \"distinct-w1-db0157-kserve-646d4c46b9-mqss4\" (UID: \"70b4dacf-b3ac-46ec-99e8-a44f05587e7c\") " pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" Nov 01 20:47:11.188632 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:11.188607 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70b4dacf-b3ac-46ec-99e8-a44f05587e7c-tls-certs\") pod \"distinct-w1-db0157-kserve-646d4c46b9-mqss4\" (UID: \"70b4dacf-b3ac-46ec-99e8-a44f05587e7c\") " pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" Nov 01 20:47:11.241136 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:11.241101 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sldq4\" (UniqueName: \"kubernetes.io/projected/70b4dacf-b3ac-46ec-99e8-a44f05587e7c-kube-api-access-sldq4\") pod \"distinct-w1-db0157-kserve-646d4c46b9-mqss4\" (UID: \"70b4dacf-b3ac-46ec-99e8-a44f05587e7c\") " pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" Nov 01 20:47:11.295328 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:11.295292 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" Nov 01 20:47:11.764649 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:47:11.764621 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70b4dacf_b3ac_46ec_99e8_a44f05587e7c.slice/crio-8e31c566aed249e91d44cac7035f4d0a3658ca586e3e71c8a94aba1e67d2f03e WatchSource:0}: Error finding container 8e31c566aed249e91d44cac7035f4d0a3658ca586e3e71c8a94aba1e67d2f03e: Status 404 returned error can't find the container with id 8e31c566aed249e91d44cac7035f4d0a3658ca586e3e71c8a94aba1e67d2f03e Nov 01 20:47:11.765406 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:11.765386 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4"] Nov 01 20:47:12.201727 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:12.201621 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" event={"ID":"70b4dacf-b3ac-46ec-99e8-a44f05587e7c","Type":"ContainerStarted","Data":"391b7775010b6326b8a4242c3a87f859eb8589e190e9470c6efc490e6d0491fc"} Nov 01 20:47:12.201727 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:12.201692 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" event={"ID":"70b4dacf-b3ac-46ec-99e8-a44f05587e7c","Type":"ContainerStarted","Data":"8e31c566aed249e91d44cac7035f4d0a3658ca586e3e71c8a94aba1e67d2f03e"} Nov 01 20:47:12.295943 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:12.295910 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" Nov 01 20:47:12.320569 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:12.320511 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" podStartSLOduration=2.320497592 podStartE2EDuration="2.320497592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:47:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:47:12.317962282 +0000 UTC m=+983.095602414" watchObservedRunningTime="2026-11-01 20:47:12.320497592 +0000 UTC m=+983.098137726" Nov 01 20:47:12.882082 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:12.882046 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:47:15.767046 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:47:15.766984 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 20:47:17.647780 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:17.647753 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:47:21.295528 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:21.295488 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" Nov 01 20:47:21.308323 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:21.308296 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" Nov 01 20:47:22.290787 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:22.290760 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" Nov 01 20:47:22.444459 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:22.444420 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:47:22.815797 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:22.815746 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zndkb"] Nov 01 20:47:22.816055 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:22.816031 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zndkb" podUID="81291c2c-8fd4-482b-8ed1-c11e9cbb2aad" containerName="istio-proxy" containerID="cri-o://24d9dfd667fdec153506eb3eb352166ae386e60b9c873ae3e950038b91c3426b" gracePeriod=30 Nov 01 20:47:27.333527 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:27.333489 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:47:28.084696 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:28.084629 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zndkb" Nov 01 20:47:28.220427 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:28.220383 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-istio-podinfo\") pod \"81291c2c-8fd4-482b-8ed1-c11e9cbb2aad\" (UID: \"81291c2c-8fd4-482b-8ed1-c11e9cbb2aad\") " Nov 01 20:47:28.220600 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:28.220462 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-istio-data\") pod \"81291c2c-8fd4-482b-8ed1-c11e9cbb2aad\" (UID: \"81291c2c-8fd4-482b-8ed1-c11e9cbb2aad\") " Nov 01 20:47:28.220600 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:28.220581 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-istio-token\" (UniqueName: \"kubernetes.io/projected/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-istio-token\") pod \"81291c2c-8fd4-482b-8ed1-c11e9cbb2aad\" (UID: \"81291c2c-8fd4-482b-8ed1-c11e9cbb2aad\") " Nov 01 20:47:28.220739 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:28.220616 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-istio-envoy\") pod \"81291c2c-8fd4-482b-8ed1-c11e9cbb2aad\" (UID: \"81291c2c-8fd4-482b-8ed1-c11e9cbb2aad\") " Nov 01 20:47:28.220739 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:28.220650 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-istiod-ca-cert\") pod \"81291c2c-8fd4-482b-8ed1-c11e9cbb2aad\" (UID: \"81291c2c-8fd4-482b-8ed1-c11e9cbb2aad\") " Nov 01 20:47:28.220739 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:28.220667 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-workload-socket\") pod \"81291c2c-8fd4-482b-8ed1-c11e9cbb2aad\" (UID: \"81291c2c-8fd4-482b-8ed1-c11e9cbb2aad\") " Nov 01 20:47:28.220739 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:28.220721 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-kube-api-access-cncg6\" (UniqueName: \"kubernetes.io/projected/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-kube-api-access-cncg6\") pod \"81291c2c-8fd4-482b-8ed1-c11e9cbb2aad\" (UID: \"81291c2c-8fd4-482b-8ed1-c11e9cbb2aad\") " Nov 01 20:47:28.220950 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:28.220766 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-workload-certs\") pod \"81291c2c-8fd4-482b-8ed1-c11e9cbb2aad\" (UID: \"81291c2c-8fd4-482b-8ed1-c11e9cbb2aad\") " Nov 01 20:47:28.220950 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:28.220793 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-credential-socket\") pod \"81291c2c-8fd4-482b-8ed1-c11e9cbb2aad\" (UID: \"81291c2c-8fd4-482b-8ed1-c11e9cbb2aad\") " Nov 01 20:47:28.220950 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:28.220819 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-istio-data" pod "81291c2c-8fd4-482b-8ed1-c11e9cbb2aad" (UID: "81291c2c-8fd4-482b-8ed1-c11e9cbb2aad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:47:28.221150 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:28.221063 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-workload-socket" pod "81291c2c-8fd4-482b-8ed1-c11e9cbb2aad" (UID: "81291c2c-8fd4-482b-8ed1-c11e9cbb2aad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:47:28.221150 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:28.221079 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-workload-certs" pod "81291c2c-8fd4-482b-8ed1-c11e9cbb2aad" (UID: "81291c2c-8fd4-482b-8ed1-c11e9cbb2aad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:47:28.221341 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:28.221290 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-credential-socket" pod "81291c2c-8fd4-482b-8ed1-c11e9cbb2aad" (UID: "81291c2c-8fd4-482b-8ed1-c11e9cbb2aad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:47:28.221408 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:28.221380 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:47:28.221408 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:28.221402 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:47:28.221499 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:28.221417 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:47:28.221499 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:28.221431 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:47:28.221499 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:28.221454 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-istiod-ca-cert" pod "81291c2c-8fd4-482b-8ed1-c11e9cbb2aad" (UID: "81291c2c-8fd4-482b-8ed1-c11e9cbb2aad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:47:28.222913 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:28.222876 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-istio-envoy" pod "81291c2c-8fd4-482b-8ed1-c11e9cbb2aad" (UID: "81291c2c-8fd4-482b-8ed1-c11e9cbb2aad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:47:28.223165 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:28.223105 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-kube-api-access-cncg6" pod "81291c2c-8fd4-482b-8ed1-c11e9cbb2aad" (UID: "81291c2c-8fd4-482b-8ed1-c11e9cbb2aad"). InnerVolumeSpecName "kube-api-access-cncg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:47:28.223290 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:28.223179 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-istio-podinfo" pod "81291c2c-8fd4-482b-8ed1-c11e9cbb2aad" (UID: "81291c2c-8fd4-482b-8ed1-c11e9cbb2aad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:47:28.223484 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:28.223465 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-istio-token" pod "81291c2c-8fd4-482b-8ed1-c11e9cbb2aad" (UID: "81291c2c-8fd4-482b-8ed1-c11e9cbb2aad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:47:28.322822 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:28.322784 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:47:28.322822 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:28.322820 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:47:28.322822 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:28.322829 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:47:28.323096 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:28.322838 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:47:28.323096 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:28.322847 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cncg6\" (UniqueName: \"kubernetes.io/projected/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad-kube-api-access-cncg6\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:47:28.323461 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:28.323437 4484 generic.go:356] "Generic (PLEG): container finished" podID="81291c2c-8fd4-482b-8ed1-c11e9cbb2aad" containerID="24d9dfd667fdec153506eb3eb352166ae386e60b9c873ae3e950038b91c3426b" exitCode=0 Nov 01 20:47:28.323529 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:28.323498 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zndkb" Nov 01 20:47:28.323583 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:28.323521 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zndkb" event={"ID":"81291c2c-8fd4-482b-8ed1-c11e9cbb2aad","Type":"ContainerDied","Data":"24d9dfd667fdec153506eb3eb352166ae386e60b9c873ae3e950038b91c3426b"} Nov 01 20:47:28.323636 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:28.323602 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zndkb" event={"ID":"81291c2c-8fd4-482b-8ed1-c11e9cbb2aad","Type":"ContainerDied","Data":"2e18a19880dc0b0773ae6ac42ee0184c4e567e0336a498f8490f24f4b40234b6"} Nov 01 20:47:28.323636 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:28.323625 4484 scope.go:122] "RemoveContainer" containerID="24d9dfd667fdec153506eb3eb352166ae386e60b9c873ae3e950038b91c3426b" Nov 01 20:47:28.334791 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:28.334732 4484 scope.go:122] "RemoveContainer" containerID="24d9dfd667fdec153506eb3eb352166ae386e60b9c873ae3e950038b91c3426b" Nov 01 20:47:28.335123 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:47:28.335039 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24d9dfd667fdec153506eb3eb352166ae386e60b9c873ae3e950038b91c3426b\": container with ID starting with 24d9dfd667fdec153506eb3eb352166ae386e60b9c873ae3e950038b91c3426b not found: ID does not exist" containerID="24d9dfd667fdec153506eb3eb352166ae386e60b9c873ae3e950038b91c3426b" Nov 01 20:47:28.335123 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:28.335063 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24d9dfd667fdec153506eb3eb352166ae386e60b9c873ae3e950038b91c3426b"} err="failed to get container status \"24d9dfd667fdec153506eb3eb352166ae386e60b9c873ae3e950038b91c3426b\": rpc error: code = NotFound desc = could not find container \"24d9dfd667fdec153506eb3eb352166ae386e60b9c873ae3e950038b91c3426b\": container with ID starting with 24d9dfd667fdec153506eb3eb352166ae386e60b9c873ae3e950038b91c3426b not found: ID does not exist" Nov 01 20:47:28.493458 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:28.493428 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zndkb"] Nov 01 20:47:28.496928 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:28.496907 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zndkb"] Nov 01 20:47:29.140567 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:29.140529 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q"] Nov 01 20:47:29.141100 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:29.141087 4484 cpu_manager.go:414] "containerMap: removing container" podUID="81291c2c-8fd4-482b-8ed1-c11e9cbb2aad" containerName="istio-proxy" Nov 01 20:47:29.141144 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:29.141102 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81291c2c-8fd4-482b-8ed1-c11e9cbb2aad" containerName="istio-proxy" Nov 01 20:47:29.141227 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:29.141217 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="81291c2c-8fd4-482b-8ed1-c11e9cbb2aad" containerName="istio-proxy" Nov 01 20:47:29.145943 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:29.145925 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" Nov 01 20:47:29.201904 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:29.201871 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q"] Nov 01 20:47:29.233805 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:29.233764 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rjtz\" (UniqueName: \"kubernetes.io/projected/9f19be29-2c84-4ae4-9674-acca4905f4e0-kube-api-access-9rjtz\") pod \"distinct-w4-1a1e9e-kserve-754444b64c-z8h6q\" (UID: \"9f19be29-2c84-4ae4-9674-acca4905f4e0\") " pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" Nov 01 20:47:29.234010 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:29.233825 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f19be29-2c84-4ae4-9674-acca4905f4e0-model-cache\") pod \"distinct-w4-1a1e9e-kserve-754444b64c-z8h6q\" (UID: \"9f19be29-2c84-4ae4-9674-acca4905f4e0\") " pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" Nov 01 20:47:29.234010 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:29.233926 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f19be29-2c84-4ae4-9674-acca4905f4e0-home\") pod \"distinct-w4-1a1e9e-kserve-754444b64c-z8h6q\" (UID: \"9f19be29-2c84-4ae4-9674-acca4905f4e0\") " pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" Nov 01 20:47:29.234010 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:29.233980 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f19be29-2c84-4ae4-9674-acca4905f4e0-tmp-dir\") pod \"distinct-w4-1a1e9e-kserve-754444b64c-z8h6q\" (UID: \"9f19be29-2c84-4ae4-9674-acca4905f4e0\") " pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" Nov 01 20:47:29.234232 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:29.234115 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f19be29-2c84-4ae4-9674-acca4905f4e0-tls-certs\") pod \"distinct-w4-1a1e9e-kserve-754444b64c-z8h6q\" (UID: \"9f19be29-2c84-4ae4-9674-acca4905f4e0\") " pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" Nov 01 20:47:29.234232 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:29.234174 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f19be29-2c84-4ae4-9674-acca4905f4e0-dshm\") pod \"distinct-w4-1a1e9e-kserve-754444b64c-z8h6q\" (UID: \"9f19be29-2c84-4ae4-9674-acca4905f4e0\") " pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" Nov 01 20:47:29.248126 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:29.248090 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-1a1e9e\"/\"distinct-w4-1a1e9e-kserve-self-signed-certs\"" Nov 01 20:47:29.250961 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:29.250937 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:47:29.335673 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:29.335644 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f19be29-2c84-4ae4-9674-acca4905f4e0-dshm\") pod \"distinct-w4-1a1e9e-kserve-754444b64c-z8h6q\" (UID: \"9f19be29-2c84-4ae4-9674-acca4905f4e0\") " pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" Nov 01 20:47:29.336145 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:29.335706 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rjtz\" (UniqueName: \"kubernetes.io/projected/9f19be29-2c84-4ae4-9674-acca4905f4e0-kube-api-access-9rjtz\") pod \"distinct-w4-1a1e9e-kserve-754444b64c-z8h6q\" (UID: \"9f19be29-2c84-4ae4-9674-acca4905f4e0\") " pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" Nov 01 20:47:29.336145 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:29.335730 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f19be29-2c84-4ae4-9674-acca4905f4e0-model-cache\") pod \"distinct-w4-1a1e9e-kserve-754444b64c-z8h6q\" (UID: \"9f19be29-2c84-4ae4-9674-acca4905f4e0\") " pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" Nov 01 20:47:29.336145 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:29.335759 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f19be29-2c84-4ae4-9674-acca4905f4e0-home\") pod \"distinct-w4-1a1e9e-kserve-754444b64c-z8h6q\" (UID: \"9f19be29-2c84-4ae4-9674-acca4905f4e0\") " pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" Nov 01 20:47:29.336145 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:29.335815 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f19be29-2c84-4ae4-9674-acca4905f4e0-tmp-dir\") pod \"distinct-w4-1a1e9e-kserve-754444b64c-z8h6q\" (UID: \"9f19be29-2c84-4ae4-9674-acca4905f4e0\") " pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" Nov 01 20:47:29.336145 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:29.335890 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f19be29-2c84-4ae4-9674-acca4905f4e0-tls-certs\") pod \"distinct-w4-1a1e9e-kserve-754444b64c-z8h6q\" (UID: \"9f19be29-2c84-4ae4-9674-acca4905f4e0\") " pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" Nov 01 20:47:29.336733 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:29.336679 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f19be29-2c84-4ae4-9674-acca4905f4e0-model-cache\") pod \"distinct-w4-1a1e9e-kserve-754444b64c-z8h6q\" (UID: \"9f19be29-2c84-4ae4-9674-acca4905f4e0\") " pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" Nov 01 20:47:29.336733 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:29.336700 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f19be29-2c84-4ae4-9674-acca4905f4e0-tmp-dir\") pod \"distinct-w4-1a1e9e-kserve-754444b64c-z8h6q\" (UID: \"9f19be29-2c84-4ae4-9674-acca4905f4e0\") " pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" Nov 01 20:47:29.336880 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:29.336806 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f19be29-2c84-4ae4-9674-acca4905f4e0-home\") pod \"distinct-w4-1a1e9e-kserve-754444b64c-z8h6q\" (UID: \"9f19be29-2c84-4ae4-9674-acca4905f4e0\") " pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" Nov 01 20:47:29.338487 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:29.338468 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f19be29-2c84-4ae4-9674-acca4905f4e0-dshm\") pod \"distinct-w4-1a1e9e-kserve-754444b64c-z8h6q\" (UID: \"9f19be29-2c84-4ae4-9674-acca4905f4e0\") " pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" Nov 01 20:47:29.339097 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:29.339079 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f19be29-2c84-4ae4-9674-acca4905f4e0-tls-certs\") pod \"distinct-w4-1a1e9e-kserve-754444b64c-z8h6q\" (UID: \"9f19be29-2c84-4ae4-9674-acca4905f4e0\") " pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" Nov 01 20:47:29.372438 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:29.372406 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81291c2c-8fd4-482b-8ed1-c11e9cbb2aad" path="/var/lib/kubelet/pods/81291c2c-8fd4-482b-8ed1-c11e9cbb2aad/volumes" Nov 01 20:47:29.394938 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:29.394866 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rjtz\" (UniqueName: \"kubernetes.io/projected/9f19be29-2c84-4ae4-9674-acca4905f4e0-kube-api-access-9rjtz\") pod \"distinct-w4-1a1e9e-kserve-754444b64c-z8h6q\" (UID: \"9f19be29-2c84-4ae4-9674-acca4905f4e0\") " pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" Nov 01 20:47:29.458527 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:29.458491 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" Nov 01 20:47:29.883082 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:47:29.883044 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f19be29_2c84_4ae4_9674_acca4905f4e0.slice/crio-c1e8d907efbd3712c3bafcf5bc419865375e826060e678f8ebf7ccad8e7ab00e WatchSource:0}: Error finding container c1e8d907efbd3712c3bafcf5bc419865375e826060e678f8ebf7ccad8e7ab00e: Status 404 returned error can't find the container with id c1e8d907efbd3712c3bafcf5bc419865375e826060e678f8ebf7ccad8e7ab00e Nov 01 20:47:29.883962 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:29.883941 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q"] Nov 01 20:47:30.349068 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:30.349031 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" event={"ID":"9f19be29-2c84-4ae4-9674-acca4905f4e0","Type":"ContainerStarted","Data":"018a8111200584e1a07c782d1e392d72ba8219e4b5fbb40d840cec6b2b77f167"} Nov 01 20:47:30.349440 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:30.349081 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" event={"ID":"9f19be29-2c84-4ae4-9674-acca4905f4e0","Type":"ContainerStarted","Data":"c1e8d907efbd3712c3bafcf5bc419865375e826060e678f8ebf7ccad8e7ab00e"} Nov 01 20:47:30.458910 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:30.458879 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" Nov 01 20:47:30.467258 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:30.467209 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" podStartSLOduration=1.4671960880000001 podStartE2EDuration="1.467196088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:47:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:47:30.466185101 +0000 UTC m=+1001.243825236" watchObservedRunningTime="2026-11-01 20:47:30.467196088 +0000 UTC m=+1001.244836262" Nov 01 20:47:32.135224 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:32.135193 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:47:33.752101 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:33.752069 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:47:36.957673 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:36.957644 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:47:38.534444 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:38.534405 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:47:39.459607 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:39.459565 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" Nov 01 20:47:39.473765 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:39.473736 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" Nov 01 20:47:40.427227 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:40.427196 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" Nov 01 20:47:41.848153 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:41.848109 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:47:45.766712 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:47:45.766672 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 20:47:46.633401 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:46.633367 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:47:49.533163 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:49.533132 4484 scope.go:122] "RemoveContainer" containerID="1fb90a461fa142b7e8c2bd7efc618ee56daa0fb7b2b7e069f2faeefb62e0c11c" Nov 01 20:47:51.042278 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:51.042245 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq"] Nov 01 20:47:51.047908 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:51.047892 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" Nov 01 20:47:51.103132 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:51.103097 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq"] Nov 01 20:47:51.149144 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:51.149112 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-944263\"/\"distinct-2-w3-944263-kserve-self-signed-certs\"" Nov 01 20:47:51.149322 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:51.149196 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-944263\"/\"kube-root-ca.crt\"" Nov 01 20:47:51.149322 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:51.149253 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-944263\"/\"default-dockercfg-9d4fj\"" Nov 01 20:47:51.152087 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:51.152072 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-944263\"/\"openshift-service-ca.crt\"" Nov 01 20:47:51.154738 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:51.154719 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a65d9fd8-6193-4d03-9a3c-123f48b5dcdd-tls-certs\") pod \"distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq\" (UID: \"a65d9fd8-6193-4d03-9a3c-123f48b5dcdd\") " pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" Nov 01 20:47:51.154800 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:51.154776 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a65d9fd8-6193-4d03-9a3c-123f48b5dcdd-dshm\") pod \"distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq\" (UID: \"a65d9fd8-6193-4d03-9a3c-123f48b5dcdd\") " pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" Nov 01 20:47:51.154899 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:51.154883 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h78c8\" (UniqueName: \"kubernetes.io/projected/a65d9fd8-6193-4d03-9a3c-123f48b5dcdd-kube-api-access-h78c8\") pod \"distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq\" (UID: \"a65d9fd8-6193-4d03-9a3c-123f48b5dcdd\") " pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" Nov 01 20:47:51.154947 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:51.154934 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a65d9fd8-6193-4d03-9a3c-123f48b5dcdd-tmp-dir\") pod \"distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq\" (UID: \"a65d9fd8-6193-4d03-9a3c-123f48b5dcdd\") " pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" Nov 01 20:47:51.155058 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:51.155040 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a65d9fd8-6193-4d03-9a3c-123f48b5dcdd-model-cache\") pod \"distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq\" (UID: \"a65d9fd8-6193-4d03-9a3c-123f48b5dcdd\") " pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" Nov 01 20:47:51.155106 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:51.155081 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a65d9fd8-6193-4d03-9a3c-123f48b5dcdd-home\") pod \"distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq\" (UID: \"a65d9fd8-6193-4d03-9a3c-123f48b5dcdd\") " pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" Nov 01 20:47:51.159385 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:51.159361 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:47:51.256616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:51.256581 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h78c8\" (UniqueName: \"kubernetes.io/projected/a65d9fd8-6193-4d03-9a3c-123f48b5dcdd-kube-api-access-h78c8\") pod \"distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq\" (UID: \"a65d9fd8-6193-4d03-9a3c-123f48b5dcdd\") " pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" Nov 01 20:47:51.256793 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:51.256643 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a65d9fd8-6193-4d03-9a3c-123f48b5dcdd-tmp-dir\") pod \"distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq\" (UID: \"a65d9fd8-6193-4d03-9a3c-123f48b5dcdd\") " pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" Nov 01 20:47:51.256793 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:51.256679 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a65d9fd8-6193-4d03-9a3c-123f48b5dcdd-model-cache\") pod \"distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq\" (UID: \"a65d9fd8-6193-4d03-9a3c-123f48b5dcdd\") " pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" Nov 01 20:47:51.256793 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:51.256707 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a65d9fd8-6193-4d03-9a3c-123f48b5dcdd-home\") pod \"distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq\" (UID: \"a65d9fd8-6193-4d03-9a3c-123f48b5dcdd\") " pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" Nov 01 20:47:51.256793 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:51.256750 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a65d9fd8-6193-4d03-9a3c-123f48b5dcdd-tls-certs\") pod \"distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq\" (UID: \"a65d9fd8-6193-4d03-9a3c-123f48b5dcdd\") " pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" Nov 01 20:47:51.256933 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:51.256795 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a65d9fd8-6193-4d03-9a3c-123f48b5dcdd-dshm\") pod \"distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq\" (UID: \"a65d9fd8-6193-4d03-9a3c-123f48b5dcdd\") " pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" Nov 01 20:47:51.257426 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:51.257401 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a65d9fd8-6193-4d03-9a3c-123f48b5dcdd-model-cache\") pod \"distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq\" (UID: \"a65d9fd8-6193-4d03-9a3c-123f48b5dcdd\") " pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" Nov 01 20:47:51.257559 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:51.257532 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a65d9fd8-6193-4d03-9a3c-123f48b5dcdd-tmp-dir\") pod \"distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq\" (UID: \"a65d9fd8-6193-4d03-9a3c-123f48b5dcdd\") " pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" Nov 01 20:47:51.257814 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:51.257798 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a65d9fd8-6193-4d03-9a3c-123f48b5dcdd-home\") pod \"distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq\" (UID: \"a65d9fd8-6193-4d03-9a3c-123f48b5dcdd\") " pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" Nov 01 20:47:51.259027 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:51.258975 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a65d9fd8-6193-4d03-9a3c-123f48b5dcdd-dshm\") pod \"distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq\" (UID: \"a65d9fd8-6193-4d03-9a3c-123f48b5dcdd\") " pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" Nov 01 20:47:51.260009 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:51.259976 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a65d9fd8-6193-4d03-9a3c-123f48b5dcdd-tls-certs\") pod \"distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq\" (UID: \"a65d9fd8-6193-4d03-9a3c-123f48b5dcdd\") " pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" Nov 01 20:47:51.317578 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:51.317506 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h78c8\" (UniqueName: \"kubernetes.io/projected/a65d9fd8-6193-4d03-9a3c-123f48b5dcdd-kube-api-access-h78c8\") pod \"distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq\" (UID: \"a65d9fd8-6193-4d03-9a3c-123f48b5dcdd\") " pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" Nov 01 20:47:51.360142 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:51.360105 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" Nov 01 20:47:51.434204 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:51.434129 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:47:51.736766 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:47:51.736734 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda65d9fd8_6193_4d03_9a3c_123f48b5dcdd.slice/crio-34f31e577b423012adf294027ca20e9e8ae9fe3dfb034e66ec4011132da3917e WatchSource:0}: Error finding container 34f31e577b423012adf294027ca20e9e8ae9fe3dfb034e66ec4011132da3917e: Status 404 returned error can't find the container with id 34f31e577b423012adf294027ca20e9e8ae9fe3dfb034e66ec4011132da3917e Nov 01 20:47:51.737055 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:51.737031 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq"] Nov 01 20:47:52.554591 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:52.554554 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" event={"ID":"a65d9fd8-6193-4d03-9a3c-123f48b5dcdd","Type":"ContainerStarted","Data":"7e1a9a32479449f680c8f44d0b622af78c1b572fbb4eca5a085f71e505ddcaa2"} Nov 01 20:47:52.554979 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:52.554602 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" event={"ID":"a65d9fd8-6193-4d03-9a3c-123f48b5dcdd","Type":"ContainerStarted","Data":"34f31e577b423012adf294027ca20e9e8ae9fe3dfb034e66ec4011132da3917e"} Nov 01 20:47:52.670945 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:52.670894 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" podStartSLOduration=2.670879953 podStartE2EDuration="2.670879953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:47:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:47:52.67044072 +0000 UTC m=+1023.448080867" watchObservedRunningTime="2026-11-01 20:47:52.670879953 +0000 UTC m=+1023.448520088" Nov 01 20:47:53.360585 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.360554 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" Nov 01 20:47:53.549145 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.549113 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk"] Nov 01 20:47:53.554741 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.554708 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk" Nov 01 20:47:53.600062 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.600024 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj"] Nov 01 20:47:53.604288 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.604272 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj" Nov 01 20:47:53.609810 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.609785 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk"] Nov 01 20:47:53.647636 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.647597 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c93e460f-14ce-40bc-93cf-51e81a750554-workload-certs\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk\" (UID: \"c93e460f-14ce-40bc-93cf-51e81a750554\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk" Nov 01 20:47:53.647813 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.647651 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-workload-certs\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj\" (UID: \"8e7768b7-6fa1-4408-a1f5-2862a8a7d907\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj" Nov 01 20:47:53.647813 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.647685 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c93e460f-14ce-40bc-93cf-51e81a750554-istio-envoy\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk\" (UID: \"c93e460f-14ce-40bc-93cf-51e81a750554\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk" Nov 01 20:47:53.647813 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.647784 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-credential-socket\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj\" (UID: \"8e7768b7-6fa1-4408-a1f5-2862a8a7d907\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj" Nov 01 20:47:53.647989 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.647874 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c93e460f-14ce-40bc-93cf-51e81a750554-istio-podinfo\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk\" (UID: \"c93e460f-14ce-40bc-93cf-51e81a750554\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk" Nov 01 20:47:53.647989 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.647927 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c93e460f-14ce-40bc-93cf-51e81a750554-istiod-ca-cert\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk\" (UID: \"c93e460f-14ce-40bc-93cf-51e81a750554\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk" Nov 01 20:47:53.648116 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.648009 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c93e460f-14ce-40bc-93cf-51e81a750554-istio-data\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk\" (UID: \"c93e460f-14ce-40bc-93cf-51e81a750554\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk" Nov 01 20:47:53.648116 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.648040 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-istio-envoy\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj\" (UID: \"8e7768b7-6fa1-4408-a1f5-2862a8a7d907\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj" Nov 01 20:47:53.648116 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.648067 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-workload-socket\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj\" (UID: \"8e7768b7-6fa1-4408-a1f5-2862a8a7d907\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj" Nov 01 20:47:53.648265 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.648122 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-istio-podinfo\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj\" (UID: \"8e7768b7-6fa1-4408-a1f5-2862a8a7d907\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj" Nov 01 20:47:53.648265 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.648176 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c93e460f-14ce-40bc-93cf-51e81a750554-credential-socket\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk\" (UID: \"c93e460f-14ce-40bc-93cf-51e81a750554\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk" Nov 01 20:47:53.648265 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.648211 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c93e460f-14ce-40bc-93cf-51e81a750554-istio-token\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk\" (UID: \"c93e460f-14ce-40bc-93cf-51e81a750554\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk" Nov 01 20:47:53.648265 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.648234 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-istio-data\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj\" (UID: \"8e7768b7-6fa1-4408-a1f5-2862a8a7d907\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj" Nov 01 20:47:53.648459 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.648299 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p29xq\" (UniqueName: \"kubernetes.io/projected/c93e460f-14ce-40bc-93cf-51e81a750554-kube-api-access-p29xq\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk\" (UID: \"c93e460f-14ce-40bc-93cf-51e81a750554\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk" Nov 01 20:47:53.648459 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.648386 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c93e460f-14ce-40bc-93cf-51e81a750554-workload-socket\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk\" (UID: \"c93e460f-14ce-40bc-93cf-51e81a750554\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk" Nov 01 20:47:53.648459 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.648410 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6cxt\" (UniqueName: \"kubernetes.io/projected/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-kube-api-access-m6cxt\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj\" (UID: \"8e7768b7-6fa1-4408-a1f5-2862a8a7d907\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj" Nov 01 20:47:53.648459 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.648445 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-istio-token\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj\" (UID: \"8e7768b7-6fa1-4408-a1f5-2862a8a7d907\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj" Nov 01 20:47:53.648646 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.648490 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-istiod-ca-cert\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj\" (UID: \"8e7768b7-6fa1-4408-a1f5-2862a8a7d907\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj" Nov 01 20:47:53.704395 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.704358 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj"] Nov 01 20:47:53.749554 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.749523 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-istio-token\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj\" (UID: \"8e7768b7-6fa1-4408-a1f5-2862a8a7d907\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj" Nov 01 20:47:53.749789 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.749560 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-istiod-ca-cert\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj\" (UID: \"8e7768b7-6fa1-4408-a1f5-2862a8a7d907\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj" Nov 01 20:47:53.749789 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.749607 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c93e460f-14ce-40bc-93cf-51e81a750554-workload-certs\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk\" (UID: \"c93e460f-14ce-40bc-93cf-51e81a750554\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk" Nov 01 20:47:53.749789 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.749633 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-workload-certs\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj\" (UID: \"8e7768b7-6fa1-4408-a1f5-2862a8a7d907\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj" Nov 01 20:47:53.749789 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.749661 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c93e460f-14ce-40bc-93cf-51e81a750554-istio-envoy\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk\" (UID: \"c93e460f-14ce-40bc-93cf-51e81a750554\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk" Nov 01 20:47:53.749789 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.749692 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-credential-socket\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj\" (UID: \"8e7768b7-6fa1-4408-a1f5-2862a8a7d907\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj" Nov 01 20:47:53.749789 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.749752 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c93e460f-14ce-40bc-93cf-51e81a750554-istio-podinfo\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk\" (UID: \"c93e460f-14ce-40bc-93cf-51e81a750554\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk" Nov 01 20:47:53.750160 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.749909 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c93e460f-14ce-40bc-93cf-51e81a750554-istiod-ca-cert\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk\" (UID: \"c93e460f-14ce-40bc-93cf-51e81a750554\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk" Nov 01 20:47:53.750160 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.749984 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c93e460f-14ce-40bc-93cf-51e81a750554-istio-data\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk\" (UID: \"c93e460f-14ce-40bc-93cf-51e81a750554\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk" Nov 01 20:47:53.750160 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.750031 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-istio-envoy\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj\" (UID: \"8e7768b7-6fa1-4408-a1f5-2862a8a7d907\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj" Nov 01 20:47:53.750160 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.750057 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-workload-socket\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj\" (UID: \"8e7768b7-6fa1-4408-a1f5-2862a8a7d907\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj" Nov 01 20:47:53.750160 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.750112 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-istio-podinfo\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj\" (UID: \"8e7768b7-6fa1-4408-a1f5-2862a8a7d907\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj" Nov 01 20:47:53.750160 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.750114 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c93e460f-14ce-40bc-93cf-51e81a750554-workload-certs\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk\" (UID: \"c93e460f-14ce-40bc-93cf-51e81a750554\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk" Nov 01 20:47:53.750160 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.750127 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-workload-certs\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj\" (UID: \"8e7768b7-6fa1-4408-a1f5-2862a8a7d907\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj" Nov 01 20:47:53.750160 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.750149 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c93e460f-14ce-40bc-93cf-51e81a750554-credential-socket\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk\" (UID: \"c93e460f-14ce-40bc-93cf-51e81a750554\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk" Nov 01 20:47:53.750160 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.750160 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-credential-socket\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj\" (UID: \"8e7768b7-6fa1-4408-a1f5-2862a8a7d907\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj" Nov 01 20:47:53.750621 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.750183 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c93e460f-14ce-40bc-93cf-51e81a750554-istio-token\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk\" (UID: \"c93e460f-14ce-40bc-93cf-51e81a750554\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk" Nov 01 20:47:53.750621 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.750212 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-istio-data\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj\" (UID: \"8e7768b7-6fa1-4408-a1f5-2862a8a7d907\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj" Nov 01 20:47:53.750621 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.750272 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p29xq\" (UniqueName: \"kubernetes.io/projected/c93e460f-14ce-40bc-93cf-51e81a750554-kube-api-access-p29xq\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk\" (UID: \"c93e460f-14ce-40bc-93cf-51e81a750554\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk" Nov 01 20:47:53.750621 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.750347 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c93e460f-14ce-40bc-93cf-51e81a750554-workload-socket\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk\" (UID: \"c93e460f-14ce-40bc-93cf-51e81a750554\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk" Nov 01 20:47:53.750621 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.750384 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6cxt\" (UniqueName: \"kubernetes.io/projected/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-kube-api-access-m6cxt\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj\" (UID: \"8e7768b7-6fa1-4408-a1f5-2862a8a7d907\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj" Nov 01 20:47:53.750621 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.750428 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c93e460f-14ce-40bc-93cf-51e81a750554-istio-data\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk\" (UID: \"c93e460f-14ce-40bc-93cf-51e81a750554\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk" Nov 01 20:47:53.750621 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.750433 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-istiod-ca-cert\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj\" (UID: \"8e7768b7-6fa1-4408-a1f5-2862a8a7d907\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj" Nov 01 20:47:53.751075 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.750700 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c93e460f-14ce-40bc-93cf-51e81a750554-credential-socket\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk\" (UID: \"c93e460f-14ce-40bc-93cf-51e81a750554\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk" Nov 01 20:47:53.751075 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.750713 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-workload-socket\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj\" (UID: \"8e7768b7-6fa1-4408-a1f5-2862a8a7d907\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj" Nov 01 20:47:53.751075 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.750770 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c93e460f-14ce-40bc-93cf-51e81a750554-workload-socket\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk\" (UID: \"c93e460f-14ce-40bc-93cf-51e81a750554\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk" Nov 01 20:47:53.751075 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.750835 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c93e460f-14ce-40bc-93cf-51e81a750554-istiod-ca-cert\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk\" (UID: \"c93e460f-14ce-40bc-93cf-51e81a750554\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk" Nov 01 20:47:53.751282 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.751088 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-istio-data\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj\" (UID: \"8e7768b7-6fa1-4408-a1f5-2862a8a7d907\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj" Nov 01 20:47:53.752383 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.752355 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c93e460f-14ce-40bc-93cf-51e81a750554-istio-envoy\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk\" (UID: \"c93e460f-14ce-40bc-93cf-51e81a750554\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk" Nov 01 20:47:53.752518 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.752367 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-istio-envoy\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj\" (UID: \"8e7768b7-6fa1-4408-a1f5-2862a8a7d907\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj" Nov 01 20:47:53.752657 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.752635 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c93e460f-14ce-40bc-93cf-51e81a750554-istio-podinfo\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk\" (UID: \"c93e460f-14ce-40bc-93cf-51e81a750554\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk" Nov 01 20:47:53.752940 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.752924 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-istio-podinfo\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj\" (UID: \"8e7768b7-6fa1-4408-a1f5-2862a8a7d907\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj" Nov 01 20:47:53.809312 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.809282 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-istio-token\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj\" (UID: \"8e7768b7-6fa1-4408-a1f5-2862a8a7d907\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj" Nov 01 20:47:53.809494 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.809282 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c93e460f-14ce-40bc-93cf-51e81a750554-istio-token\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk\" (UID: \"c93e460f-14ce-40bc-93cf-51e81a750554\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk" Nov 01 20:47:53.809606 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.809582 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6cxt\" (UniqueName: \"kubernetes.io/projected/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-kube-api-access-m6cxt\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj\" (UID: \"8e7768b7-6fa1-4408-a1f5-2862a8a7d907\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj" Nov 01 20:47:53.809858 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.809839 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p29xq\" (UniqueName: \"kubernetes.io/projected/c93e460f-14ce-40bc-93cf-51e81a750554-kube-api-access-p29xq\") pod \"e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk\" (UID: \"c93e460f-14ce-40bc-93cf-51e81a750554\") " pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk" Nov 01 20:47:53.873363 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.873272 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk" Nov 01 20:47:53.919019 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:53.918971 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj" Nov 01 20:47:54.300765 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:47:54.300735 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc93e460f_14ce_40bc_93cf_51e81a750554.slice/crio-99d4940fab79c223381690b66aa27dcae74aced2fd8f8932a42e4104b0c5e6f4 WatchSource:0}: Error finding container 99d4940fab79c223381690b66aa27dcae74aced2fd8f8932a42e4104b0c5e6f4: Status 404 returned error can't find the container with id 99d4940fab79c223381690b66aa27dcae74aced2fd8f8932a42e4104b0c5e6f4 Nov 01 20:47:54.300892 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:54.300781 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk"] Nov 01 20:47:54.303245 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:54.303209 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:47:54.303363 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:54.303293 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:47:54.303363 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:54.303339 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:47:54.352322 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:47:54.352295 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e7768b7_6fa1_4408_a1f5_2862a8a7d907.slice/crio-4fec9df5f5438d6fe7326ec3f16f8bc443a34b1619b7a36541b81dfdd495a113 WatchSource:0}: Error finding container 4fec9df5f5438d6fe7326ec3f16f8bc443a34b1619b7a36541b81dfdd495a113: Status 404 returned error can't find the container with id 4fec9df5f5438d6fe7326ec3f16f8bc443a34b1619b7a36541b81dfdd495a113 Nov 01 20:47:54.352838 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:54.352813 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj"] Nov 01 20:47:54.354956 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:54.354925 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:47:54.355057 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:54.354986 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:47:54.355057 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:54.355042 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:47:54.576914 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:54.576819 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk" event={"ID":"c93e460f-14ce-40bc-93cf-51e81a750554","Type":"ContainerStarted","Data":"4c068cebff320e1ff75553f7197c0adbcd9d04c5033f91bbcec0a9062a2a960f"} Nov 01 20:47:54.576914 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:54.576885 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk" event={"ID":"c93e460f-14ce-40bc-93cf-51e81a750554","Type":"ContainerStarted","Data":"99d4940fab79c223381690b66aa27dcae74aced2fd8f8932a42e4104b0c5e6f4"} Nov 01 20:47:54.579161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:54.579133 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj" event={"ID":"8e7768b7-6fa1-4408-a1f5-2862a8a7d907","Type":"ContainerStarted","Data":"613085aea41e537c2798b514085f2da5f17b07a036ebc4af8f94dbdc48b023a9"} Nov 01 20:47:54.579310 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:54.579182 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj" event={"ID":"8e7768b7-6fa1-4408-a1f5-2862a8a7d907","Type":"ContainerStarted","Data":"4fec9df5f5438d6fe7326ec3f16f8bc443a34b1619b7a36541b81dfdd495a113"} Nov 01 20:47:54.699631 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:54.699573 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk" podStartSLOduration=1.699558531 podStartE2EDuration="1.699558531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:47:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:47:54.695371742 +0000 UTC m=+1025.473011888" watchObservedRunningTime="2026-11-01 20:47:54.699558531 +0000 UTC m=+1025.477198666" Nov 01 20:47:54.822968 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:54.822895 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj" podStartSLOduration=1.822875244 podStartE2EDuration="1.822875244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:47:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:47:54.816460731 +0000 UTC m=+1025.594100867" watchObservedRunningTime="2026-11-01 20:47:54.822875244 +0000 UTC m=+1025.600515377" Nov 01 20:47:54.874365 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:54.874282 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk" Nov 01 20:47:54.922493 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:54.922452 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj" Nov 01 20:47:55.875503 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:55.875469 4484 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 01 20:47:55.875896 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:55.875520 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk" podUID="c93e460f-14ce-40bc-93cf-51e81a750554" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 01 20:47:55.952304 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:55.950496 4484 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 01 20:47:55.952304 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:55.950551 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj" podUID="8e7768b7-6fa1-4408-a1f5-2862a8a7d907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 01 20:47:56.361485 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:56.361414 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:47:56.874871 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:56.874140 4484 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 01 20:47:56.875074 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:56.874898 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk" podUID="c93e460f-14ce-40bc-93cf-51e81a750554" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 01 20:47:56.919573 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:56.919533 4484 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 01 20:47:56.920202 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:56.920168 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj" podUID="8e7768b7-6fa1-4408-a1f5-2862a8a7d907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 01 20:47:57.874331 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:57.874295 4484 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 01 20:47:57.874514 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:57.874357 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk" podUID="c93e460f-14ce-40bc-93cf-51e81a750554" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 01 20:47:57.925025 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:57.924400 4484 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 01 20:47:57.925025 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:57.924469 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj" podUID="8e7768b7-6fa1-4408-a1f5-2862a8a7d907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 01 20:47:58.874470 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:58.874432 4484 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 01 20:47:58.874665 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:58.874497 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk" podUID="c93e460f-14ce-40bc-93cf-51e81a750554" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 01 20:47:58.923021 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:58.921356 4484 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 01 20:47:58.923021 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:58.921418 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj" podUID="8e7768b7-6fa1-4408-a1f5-2862a8a7d907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 01 20:47:59.874693 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:59.874654 4484 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 01 20:47:59.875121 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:59.874724 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk" podUID="c93e460f-14ce-40bc-93cf-51e81a750554" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 01 20:47:59.922021 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:59.920026 4484 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 01 20:47:59.922021 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:47:59.920099 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj" podUID="8e7768b7-6fa1-4408-a1f5-2862a8a7d907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 01 20:48:00.873859 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:00.873818 4484 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 01 20:48:00.874041 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:00.873889 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk" podUID="c93e460f-14ce-40bc-93cf-51e81a750554" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 01 20:48:00.920290 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:00.920252 4484 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 01 20:48:00.920697 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:00.920334 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj" podUID="8e7768b7-6fa1-4408-a1f5-2862a8a7d907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 01 20:48:01.247457 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:01.247357 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:48:01.362436 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:01.362389 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" Nov 01 20:48:01.383904 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:01.383870 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" Nov 01 20:48:01.679517 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:01.679426 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" Nov 01 20:48:01.878493 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:01.878458 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk" Nov 01 20:48:01.878877 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:01.878849 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk" Nov 01 20:48:01.879673 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:01.879655 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk" Nov 01 20:48:01.925611 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:01.925577 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj" Nov 01 20:48:01.926109 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:01.925975 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj" Nov 01 20:48:01.926659 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:01.926639 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj" Nov 01 20:48:06.246239 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:06.246201 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:48:06.978746 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:06.978714 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:48:10.951130 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:10.951097 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:48:15.739573 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:15.739538 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:48:15.822483 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:48:15.822441 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 20:48:20.535389 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:20.535333 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:48:25.402646 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:25.402596 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:48:30.138738 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:30.138688 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:48:34.939956 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:34.939910 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:48:38.863210 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:38.863170 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:48:39.742507 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:39.742466 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:48:44.479622 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:44.479591 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:48:44.653710 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:44.653677 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:48:45.766729 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:48:45.766692 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 20:48:48.156801 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:48.156769 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40cb3cb1-75zv6"] Nov 01 20:48:48.161003 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:48.160977 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40cb3cb1-75zv6" Nov 01 20:48:48.202310 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:48.202274 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/096f520b-a10b-4f4c-afcc-0ea5d143fbd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40cb3cb1-75zv6\" (UID: \"096f520b-a10b-4f4c-afcc-0ea5d143fbd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40cb3cb1-75zv6" Nov 01 20:48:48.202523 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:48.202335 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnx2t\" (UniqueName: \"kubernetes.io/projected/096f520b-a10b-4f4c-afcc-0ea5d143fbd2-kube-api-access-tnx2t\") pod \"maas-api-revoke-sub-40cb3cb1-75zv6\" (UID: \"096f520b-a10b-4f4c-afcc-0ea5d143fbd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40cb3cb1-75zv6" Nov 01 20:48:48.202523 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:48.202416 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/096f520b-a10b-4f4c-afcc-0ea5d143fbd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40cb3cb1-75zv6\" (UID: \"096f520b-a10b-4f4c-afcc-0ea5d143fbd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40cb3cb1-75zv6" Nov 01 20:48:48.217894 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:48.217868 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:48:48.268491 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:48.268457 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:48:48.268491 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:48.268463 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:48:48.268722 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:48.268566 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 20:48:48.272522 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:48.272496 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40cb3cb1-75zv6"] Nov 01 20:48:48.303702 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:48.303664 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/096f520b-a10b-4f4c-afcc-0ea5d143fbd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40cb3cb1-75zv6\" (UID: \"096f520b-a10b-4f4c-afcc-0ea5d143fbd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40cb3cb1-75zv6" Nov 01 20:48:48.303872 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:48.303737 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnx2t\" (UniqueName: \"kubernetes.io/projected/096f520b-a10b-4f4c-afcc-0ea5d143fbd2-kube-api-access-tnx2t\") pod \"maas-api-revoke-sub-40cb3cb1-75zv6\" (UID: \"096f520b-a10b-4f4c-afcc-0ea5d143fbd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40cb3cb1-75zv6" Nov 01 20:48:48.303872 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:48.303785 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/096f520b-a10b-4f4c-afcc-0ea5d143fbd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40cb3cb1-75zv6\" (UID: \"096f520b-a10b-4f4c-afcc-0ea5d143fbd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40cb3cb1-75zv6" Nov 01 20:48:48.304417 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:48.304398 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/096f520b-a10b-4f4c-afcc-0ea5d143fbd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40cb3cb1-75zv6\" (UID: \"096f520b-a10b-4f4c-afcc-0ea5d143fbd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40cb3cb1-75zv6" Nov 01 20:48:48.306103 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:48.306086 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/096f520b-a10b-4f4c-afcc-0ea5d143fbd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40cb3cb1-75zv6\" (UID: \"096f520b-a10b-4f4c-afcc-0ea5d143fbd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40cb3cb1-75zv6" Nov 01 20:48:48.361895 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:48.361852 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnx2t\" (UniqueName: \"kubernetes.io/projected/096f520b-a10b-4f4c-afcc-0ea5d143fbd2-kube-api-access-tnx2t\") pod \"maas-api-revoke-sub-40cb3cb1-75zv6\" (UID: \"096f520b-a10b-4f4c-afcc-0ea5d143fbd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40cb3cb1-75zv6" Nov 01 20:48:48.475191 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:48.475106 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40cb3cb1-75zv6" Nov 01 20:48:48.851643 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:48.851613 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40cb3cb1-75zv6"] Nov 01 20:48:48.853105 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:48:48.853080 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod096f520b_a10b_4f4c_afcc_0ea5d143fbd2.slice/crio-3f2071620eaf85d9b7b875fb1e6d833665d04cc3ae7c24ae816fc991dbc71321 WatchSource:0}: Error finding container 3f2071620eaf85d9b7b875fb1e6d833665d04cc3ae7c24ae816fc991dbc71321: Status 404 returned error can't find the container with id 3f2071620eaf85d9b7b875fb1e6d833665d04cc3ae7c24ae816fc991dbc71321 Nov 01 20:48:49.038981 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:49.038946 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40cb3cb1-75zv6" event={"ID":"096f520b-a10b-4f4c-afcc-0ea5d143fbd2","Type":"ContainerStarted","Data":"3f2071620eaf85d9b7b875fb1e6d833665d04cc3ae7c24ae816fc991dbc71321"} Nov 01 20:48:49.332049 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:49.332010 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:48:50.056354 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:50.056318 4484 generic.go:356] "Generic (PLEG): container finished" podID="096f520b-a10b-4f4c-afcc-0ea5d143fbd2" containerID="0fec2966bfe38d8d66c884c2c0769ddac9b3ab0e19f36d896ca179cfd4943305" exitCode=0 Nov 01 20:48:50.056556 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:50.056375 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40cb3cb1-75zv6" event={"ID":"096f520b-a10b-4f4c-afcc-0ea5d143fbd2","Type":"ContainerDied","Data":"0fec2966bfe38d8d66c884c2c0769ddac9b3ab0e19f36d896ca179cfd4943305"} Nov 01 20:48:51.228334 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:51.228308 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40cb3cb1-75zv6" Nov 01 20:48:51.242423 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:51.242393 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/096f520b-a10b-4f4c-afcc-0ea5d143fbd2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/096f520b-a10b-4f4c-afcc-0ea5d143fbd2-maas-api-openshift-ca\") pod \"096f520b-a10b-4f4c-afcc-0ea5d143fbd2\" (UID: \"096f520b-a10b-4f4c-afcc-0ea5d143fbd2\") " Nov 01 20:48:51.242600 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:51.242481 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/096f520b-a10b-4f4c-afcc-0ea5d143fbd2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/096f520b-a10b-4f4c-afcc-0ea5d143fbd2-maas-api-xks-ca\") pod \"096f520b-a10b-4f4c-afcc-0ea5d143fbd2\" (UID: \"096f520b-a10b-4f4c-afcc-0ea5d143fbd2\") " Nov 01 20:48:51.242677 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:51.242651 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/096f520b-a10b-4f4c-afcc-0ea5d143fbd2-kube-api-access-tnx2t\" (UniqueName: \"kubernetes.io/projected/096f520b-a10b-4f4c-afcc-0ea5d143fbd2-kube-api-access-tnx2t\") pod \"096f520b-a10b-4f4c-afcc-0ea5d143fbd2\" (UID: \"096f520b-a10b-4f4c-afcc-0ea5d143fbd2\") " Nov 01 20:48:51.242835 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:51.242806 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/096f520b-a10b-4f4c-afcc-0ea5d143fbd2-maas-api-openshift-ca" pod "096f520b-a10b-4f4c-afcc-0ea5d143fbd2" (UID: "096f520b-a10b-4f4c-afcc-0ea5d143fbd2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:48:51.243170 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:51.243146 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/096f520b-a10b-4f4c-afcc-0ea5d143fbd2-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:48:51.244633 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:51.244610 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/096f520b-a10b-4f4c-afcc-0ea5d143fbd2-maas-api-xks-ca" pod "096f520b-a10b-4f4c-afcc-0ea5d143fbd2" (UID: "096f520b-a10b-4f4c-afcc-0ea5d143fbd2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:48:51.244843 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:51.244821 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/096f520b-a10b-4f4c-afcc-0ea5d143fbd2-kube-api-access-tnx2t" pod "096f520b-a10b-4f4c-afcc-0ea5d143fbd2" (UID: "096f520b-a10b-4f4c-afcc-0ea5d143fbd2"). InnerVolumeSpecName "kube-api-access-tnx2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:48:51.344052 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:51.343920 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnx2t\" (UniqueName: \"kubernetes.io/projected/096f520b-a10b-4f4c-afcc-0ea5d143fbd2-kube-api-access-tnx2t\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:48:51.344052 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:51.343950 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/096f520b-a10b-4f4c-afcc-0ea5d143fbd2-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:48:52.071626 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:52.071596 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40cb3cb1-75zv6" Nov 01 20:48:52.071803 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:52.071626 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40cb3cb1-75zv6" event={"ID":"096f520b-a10b-4f4c-afcc-0ea5d143fbd2","Type":"ContainerDied","Data":"3f2071620eaf85d9b7b875fb1e6d833665d04cc3ae7c24ae816fc991dbc71321"} Nov 01 20:48:52.071803 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:52.071679 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f2071620eaf85d9b7b875fb1e6d833665d04cc3ae7c24ae816fc991dbc71321" Nov 01 20:48:54.127270 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:54.127227 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:48:59.065546 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:48:59.065511 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:49:02.544462 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:02.544427 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48a2ffe7-8nlnq"] Nov 01 20:49:02.545080 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:02.545065 4484 cpu_manager.go:414] "containerMap: removing container" podUID="096f520b-a10b-4f4c-afcc-0ea5d143fbd2" containerName="revoke-keys" Nov 01 20:49:02.545140 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:02.545082 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="096f520b-a10b-4f4c-afcc-0ea5d143fbd2" containerName="revoke-keys" Nov 01 20:49:02.545210 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:02.545201 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="096f520b-a10b-4f4c-afcc-0ea5d143fbd2" containerName="revoke-keys" Nov 01 20:49:02.549061 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:02.549040 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48a2ffe7-8nlnq" Nov 01 20:49:02.602122 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:02.602091 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:49:02.602289 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:02.602176 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 20:49:02.603639 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:02.603615 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48a2ffe7-8nlnq"] Nov 01 20:49:02.683444 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:02.683408 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:49:02.683444 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:02.683409 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:49:02.725779 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:02.725742 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdj9v\" (UniqueName: \"kubernetes.io/projected/1479472c-ffe5-49cc-98b9-5fba410f0c4c-kube-api-access-wdj9v\") pod \"maas-api-revoke-sub-48a2ffe7-8nlnq\" (UID: \"1479472c-ffe5-49cc-98b9-5fba410f0c4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48a2ffe7-8nlnq" Nov 01 20:49:02.726038 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:02.725796 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1479472c-ffe5-49cc-98b9-5fba410f0c4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48a2ffe7-8nlnq\" (UID: \"1479472c-ffe5-49cc-98b9-5fba410f0c4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48a2ffe7-8nlnq" Nov 01 20:49:02.726038 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:02.725822 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1479472c-ffe5-49cc-98b9-5fba410f0c4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48a2ffe7-8nlnq\" (UID: \"1479472c-ffe5-49cc-98b9-5fba410f0c4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48a2ffe7-8nlnq" Nov 01 20:49:02.827044 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:02.826910 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdj9v\" (UniqueName: \"kubernetes.io/projected/1479472c-ffe5-49cc-98b9-5fba410f0c4c-kube-api-access-wdj9v\") pod \"maas-api-revoke-sub-48a2ffe7-8nlnq\" (UID: \"1479472c-ffe5-49cc-98b9-5fba410f0c4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48a2ffe7-8nlnq" Nov 01 20:49:02.827044 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:02.826974 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1479472c-ffe5-49cc-98b9-5fba410f0c4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48a2ffe7-8nlnq\" (UID: \"1479472c-ffe5-49cc-98b9-5fba410f0c4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48a2ffe7-8nlnq" Nov 01 20:49:02.827254 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:02.827049 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1479472c-ffe5-49cc-98b9-5fba410f0c4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48a2ffe7-8nlnq\" (UID: \"1479472c-ffe5-49cc-98b9-5fba410f0c4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48a2ffe7-8nlnq" Nov 01 20:49:02.827658 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:02.827628 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1479472c-ffe5-49cc-98b9-5fba410f0c4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48a2ffe7-8nlnq\" (UID: \"1479472c-ffe5-49cc-98b9-5fba410f0c4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48a2ffe7-8nlnq" Nov 01 20:49:02.829411 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:02.829390 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1479472c-ffe5-49cc-98b9-5fba410f0c4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48a2ffe7-8nlnq\" (UID: \"1479472c-ffe5-49cc-98b9-5fba410f0c4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48a2ffe7-8nlnq" Nov 01 20:49:02.885713 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:02.885678 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdj9v\" (UniqueName: \"kubernetes.io/projected/1479472c-ffe5-49cc-98b9-5fba410f0c4c-kube-api-access-wdj9v\") pod \"maas-api-revoke-sub-48a2ffe7-8nlnq\" (UID: \"1479472c-ffe5-49cc-98b9-5fba410f0c4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48a2ffe7-8nlnq" Nov 01 20:49:03.162813 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:03.162731 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48a2ffe7-8nlnq" Nov 01 20:49:03.539765 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:49:03.539736 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1479472c_ffe5_49cc_98b9_5fba410f0c4c.slice/crio-6e338ab667432ec0a889d48f39be7a251976b96da70daa1737ed7c9766759870 WatchSource:0}: Error finding container 6e338ab667432ec0a889d48f39be7a251976b96da70daa1737ed7c9766759870: Status 404 returned error can't find the container with id 6e338ab667432ec0a889d48f39be7a251976b96da70daa1737ed7c9766759870 Nov 01 20:49:03.540293 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:03.540268 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48a2ffe7-8nlnq"] Nov 01 20:49:03.847866 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:03.847801 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:49:04.195509 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:04.195421 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48a2ffe7-8nlnq" event={"ID":"1479472c-ffe5-49cc-98b9-5fba410f0c4c","Type":"ContainerStarted","Data":"0f53a479ccfa75fede0b3687eeb4f0dc68c8800a3c16030b1f40b4df0b6d48a1"} Nov 01 20:49:04.195509 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:04.195482 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48a2ffe7-8nlnq" event={"ID":"1479472c-ffe5-49cc-98b9-5fba410f0c4c","Type":"ContainerStarted","Data":"6e338ab667432ec0a889d48f39be7a251976b96da70daa1737ed7c9766759870"} Nov 01 20:49:04.310843 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:04.310786 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48a2ffe7-8nlnq" podStartSLOduration=2.310768512 podStartE2EDuration="2.310768512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:49:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:49:04.309985387 +0000 UTC m=+1095.087625534" watchObservedRunningTime="2026-11-01 20:49:04.310768512 +0000 UTC m=+1095.088408647" Nov 01 20:49:05.204381 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:05.204351 4484 generic.go:356] "Generic (PLEG): container finished" podID="1479472c-ffe5-49cc-98b9-5fba410f0c4c" containerID="0f53a479ccfa75fede0b3687eeb4f0dc68c8800a3c16030b1f40b4df0b6d48a1" exitCode=0 Nov 01 20:49:05.204761 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:05.204447 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48a2ffe7-8nlnq" event={"ID":"1479472c-ffe5-49cc-98b9-5fba410f0c4c","Type":"ContainerDied","Data":"0f53a479ccfa75fede0b3687eeb4f0dc68c8800a3c16030b1f40b4df0b6d48a1"} Nov 01 20:49:06.182678 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:06.182643 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq"] Nov 01 20:49:06.187223 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:06.187199 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" Nov 01 20:49:06.244748 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:06.244716 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq"] Nov 01 20:49:06.291392 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:06.291320 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-dd038b\"/\"openshift-service-ca.crt\"" Nov 01 20:49:06.291571 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:06.291436 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-dd038b\"/\"kube-root-ca.crt\"" Nov 01 20:49:06.293885 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:06.291582 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-dd038b\"/\"default-dockercfg-hdkp4\"" Nov 01 20:49:06.295013 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:06.294961 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-dd038b\"/\"distinct-2-w6-dd038b-kserve-self-signed-certs\"" Nov 01 20:49:06.304876 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:06.304748 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:49:06.389027 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:06.382187 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e175b298-2c23-45d4-8f9c-0c9b9f6aab92-tmp-dir\") pod \"distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq\" (UID: \"e175b298-2c23-45d4-8f9c-0c9b9f6aab92\") " pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" Nov 01 20:49:06.389027 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:06.382271 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e175b298-2c23-45d4-8f9c-0c9b9f6aab92-dshm\") pod \"distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq\" (UID: \"e175b298-2c23-45d4-8f9c-0c9b9f6aab92\") " pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" Nov 01 20:49:06.389027 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:06.382412 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e175b298-2c23-45d4-8f9c-0c9b9f6aab92-tls-certs\") pod \"distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq\" (UID: \"e175b298-2c23-45d4-8f9c-0c9b9f6aab92\") " pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" Nov 01 20:49:06.389027 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:06.382460 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e175b298-2c23-45d4-8f9c-0c9b9f6aab92-home\") pod \"distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq\" (UID: \"e175b298-2c23-45d4-8f9c-0c9b9f6aab92\") " pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" Nov 01 20:49:06.389027 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:06.382529 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e175b298-2c23-45d4-8f9c-0c9b9f6aab92-model-cache\") pod \"distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq\" (UID: \"e175b298-2c23-45d4-8f9c-0c9b9f6aab92\") " pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" Nov 01 20:49:06.389027 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:06.382639 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5nvb\" (UniqueName: \"kubernetes.io/projected/e175b298-2c23-45d4-8f9c-0c9b9f6aab92-kube-api-access-w5nvb\") pod \"distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq\" (UID: \"e175b298-2c23-45d4-8f9c-0c9b9f6aab92\") " pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" Nov 01 20:49:06.476754 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:06.476725 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48a2ffe7-8nlnq" Nov 01 20:49:06.484189 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:06.484159 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e175b298-2c23-45d4-8f9c-0c9b9f6aab92-tmp-dir\") pod \"distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq\" (UID: \"e175b298-2c23-45d4-8f9c-0c9b9f6aab92\") " pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" Nov 01 20:49:06.484354 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:06.484220 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e175b298-2c23-45d4-8f9c-0c9b9f6aab92-dshm\") pod \"distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq\" (UID: \"e175b298-2c23-45d4-8f9c-0c9b9f6aab92\") " pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" Nov 01 20:49:06.484354 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:06.484278 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e175b298-2c23-45d4-8f9c-0c9b9f6aab92-tls-certs\") pod \"distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq\" (UID: \"e175b298-2c23-45d4-8f9c-0c9b9f6aab92\") " pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" Nov 01 20:49:06.484354 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:06.484311 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e175b298-2c23-45d4-8f9c-0c9b9f6aab92-home\") pod \"distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq\" (UID: \"e175b298-2c23-45d4-8f9c-0c9b9f6aab92\") " pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" Nov 01 20:49:06.484531 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:06.484381 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e175b298-2c23-45d4-8f9c-0c9b9f6aab92-model-cache\") pod \"distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq\" (UID: \"e175b298-2c23-45d4-8f9c-0c9b9f6aab92\") " pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" Nov 01 20:49:06.484531 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:06.484482 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5nvb\" (UniqueName: \"kubernetes.io/projected/e175b298-2c23-45d4-8f9c-0c9b9f6aab92-kube-api-access-w5nvb\") pod \"distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq\" (UID: \"e175b298-2c23-45d4-8f9c-0c9b9f6aab92\") " pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" Nov 01 20:49:06.484642 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:06.484627 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e175b298-2c23-45d4-8f9c-0c9b9f6aab92-tmp-dir\") pod \"distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq\" (UID: \"e175b298-2c23-45d4-8f9c-0c9b9f6aab92\") " pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" Nov 01 20:49:06.484746 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:06.484714 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e175b298-2c23-45d4-8f9c-0c9b9f6aab92-home\") pod \"distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq\" (UID: \"e175b298-2c23-45d4-8f9c-0c9b9f6aab92\") " pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" Nov 01 20:49:06.484892 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:06.484873 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e175b298-2c23-45d4-8f9c-0c9b9f6aab92-model-cache\") pod \"distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq\" (UID: \"e175b298-2c23-45d4-8f9c-0c9b9f6aab92\") " pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" Nov 01 20:49:06.487105 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:06.487084 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e175b298-2c23-45d4-8f9c-0c9b9f6aab92-dshm\") pod \"distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq\" (UID: \"e175b298-2c23-45d4-8f9c-0c9b9f6aab92\") " pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" Nov 01 20:49:06.487503 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:06.487482 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e175b298-2c23-45d4-8f9c-0c9b9f6aab92-tls-certs\") pod \"distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq\" (UID: \"e175b298-2c23-45d4-8f9c-0c9b9f6aab92\") " pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" Nov 01 20:49:06.596021 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:06.594685 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1479472c-ffe5-49cc-98b9-5fba410f0c4c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1479472c-ffe5-49cc-98b9-5fba410f0c4c-maas-api-xks-ca\") pod \"1479472c-ffe5-49cc-98b9-5fba410f0c4c\" (UID: \"1479472c-ffe5-49cc-98b9-5fba410f0c4c\") " Nov 01 20:49:06.596021 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:06.594844 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1479472c-ffe5-49cc-98b9-5fba410f0c4c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1479472c-ffe5-49cc-98b9-5fba410f0c4c-maas-api-openshift-ca\") pod \"1479472c-ffe5-49cc-98b9-5fba410f0c4c\" (UID: \"1479472c-ffe5-49cc-98b9-5fba410f0c4c\") " Nov 01 20:49:06.596021 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:06.594952 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1479472c-ffe5-49cc-98b9-5fba410f0c4c-kube-api-access-wdj9v\" (UniqueName: \"kubernetes.io/projected/1479472c-ffe5-49cc-98b9-5fba410f0c4c-kube-api-access-wdj9v\") pod \"1479472c-ffe5-49cc-98b9-5fba410f0c4c\" (UID: \"1479472c-ffe5-49cc-98b9-5fba410f0c4c\") " Nov 01 20:49:06.646875 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:06.646225 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1479472c-ffe5-49cc-98b9-5fba410f0c4c-kube-api-access-wdj9v" pod "1479472c-ffe5-49cc-98b9-5fba410f0c4c" (UID: "1479472c-ffe5-49cc-98b9-5fba410f0c4c"). InnerVolumeSpecName "kube-api-access-wdj9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:49:06.648754 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:06.647819 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5nvb\" (UniqueName: \"kubernetes.io/projected/e175b298-2c23-45d4-8f9c-0c9b9f6aab92-kube-api-access-w5nvb\") pod \"distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq\" (UID: \"e175b298-2c23-45d4-8f9c-0c9b9f6aab92\") " pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" Nov 01 20:49:06.648754 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:06.648503 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1479472c-ffe5-49cc-98b9-5fba410f0c4c-maas-api-xks-ca" pod "1479472c-ffe5-49cc-98b9-5fba410f0c4c" (UID: "1479472c-ffe5-49cc-98b9-5fba410f0c4c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:49:06.660034 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:06.657273 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1479472c-ffe5-49cc-98b9-5fba410f0c4c-maas-api-openshift-ca" pod "1479472c-ffe5-49cc-98b9-5fba410f0c4c" (UID: "1479472c-ffe5-49cc-98b9-5fba410f0c4c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:49:06.697229 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:06.697144 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1479472c-ffe5-49cc-98b9-5fba410f0c4c-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:49:06.697229 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:06.697178 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdj9v\" (UniqueName: \"kubernetes.io/projected/1479472c-ffe5-49cc-98b9-5fba410f0c4c-kube-api-access-wdj9v\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:49:06.697229 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:06.697189 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1479472c-ffe5-49cc-98b9-5fba410f0c4c-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:49:06.800467 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:06.800432 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" Nov 01 20:49:07.190910 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:49:07.190881 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode175b298_2c23_45d4_8f9c_0c9b9f6aab92.slice/crio-81c64095598fdb4a7d88ed5cbbc49ff9e687b2f67fc0d5104c73ac933e48d14a WatchSource:0}: Error finding container 81c64095598fdb4a7d88ed5cbbc49ff9e687b2f67fc0d5104c73ac933e48d14a: Status 404 returned error can't find the container with id 81c64095598fdb4a7d88ed5cbbc49ff9e687b2f67fc0d5104c73ac933e48d14a Nov 01 20:49:07.192765 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:07.192733 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq"] Nov 01 20:49:07.222260 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:07.222219 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48a2ffe7-8nlnq" event={"ID":"1479472c-ffe5-49cc-98b9-5fba410f0c4c","Type":"ContainerDied","Data":"6e338ab667432ec0a889d48f39be7a251976b96da70daa1737ed7c9766759870"} Nov 01 20:49:07.222411 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:07.222279 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e338ab667432ec0a889d48f39be7a251976b96da70daa1737ed7c9766759870" Nov 01 20:49:07.222411 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:07.222346 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48a2ffe7-8nlnq" Nov 01 20:49:07.227279 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:07.227237 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" event={"ID":"e175b298-2c23-45d4-8f9c-0c9b9f6aab92","Type":"ContainerStarted","Data":"81c64095598fdb4a7d88ed5cbbc49ff9e687b2f67fc0d5104c73ac933e48d14a"} Nov 01 20:49:08.235682 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:08.235646 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" event={"ID":"e175b298-2c23-45d4-8f9c-0c9b9f6aab92","Type":"ContainerStarted","Data":"955e23e9322ec7be3ec445fa9c82cdd1ba700e26a4d4c404cf4bdfa1f51bf859"} Nov 01 20:49:08.353767 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:08.353713 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" podStartSLOduration=2.353695076 podStartE2EDuration="2.353695076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:49:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:49:08.350391409 +0000 UTC m=+1099.128031546" watchObservedRunningTime="2026-11-01 20:49:08.353695076 +0000 UTC m=+1099.131335211" Nov 01 20:49:08.637199 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:08.637168 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:49:08.800903 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:08.800856 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" Nov 01 20:49:13.447223 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:13.447184 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:49:15.772554 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:49:15.772508 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 20:49:16.801431 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:16.801392 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" Nov 01 20:49:16.814430 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:16.814399 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" Nov 01 20:49:17.311678 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:17.311651 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" Nov 01 20:49:18.346627 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:18.346583 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:49:23.151442 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.151396 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:49:23.379624 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.379585 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh"] Nov 01 20:49:23.380169 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.380154 4484 cpu_manager.go:414] "containerMap: removing container" podUID="1479472c-ffe5-49cc-98b9-5fba410f0c4c" containerName="revoke-keys" Nov 01 20:49:23.380217 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.380170 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1479472c-ffe5-49cc-98b9-5fba410f0c4c" containerName="revoke-keys" Nov 01 20:49:23.380302 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.380293 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="1479472c-ffe5-49cc-98b9-5fba410f0c4c" containerName="revoke-keys" Nov 01 20:49:23.384090 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.384074 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh" Nov 01 20:49:23.426448 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.426366 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx"] Nov 01 20:49:23.430638 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.430615 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx" Nov 01 20:49:23.452366 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.452339 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/388311ca-26b2-4bd9-b355-a5b4c7b459cb-istio-data\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh\" (UID: \"388311ca-26b2-4bd9-b355-a5b4c7b459cb\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh" Nov 01 20:49:23.452505 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.452387 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/388311ca-26b2-4bd9-b355-a5b4c7b459cb-workload-certs\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh\" (UID: \"388311ca-26b2-4bd9-b355-a5b4c7b459cb\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh" Nov 01 20:49:23.452581 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.452562 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/388311ca-26b2-4bd9-b355-a5b4c7b459cb-istio-token\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh\" (UID: \"388311ca-26b2-4bd9-b355-a5b4c7b459cb\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh" Nov 01 20:49:23.452638 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.452598 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/388311ca-26b2-4bd9-b355-a5b4c7b459cb-istio-podinfo\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh\" (UID: \"388311ca-26b2-4bd9-b355-a5b4c7b459cb\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh" Nov 01 20:49:23.452638 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.452621 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf5q9\" (UniqueName: \"kubernetes.io/projected/388311ca-26b2-4bd9-b355-a5b4c7b459cb-kube-api-access-zf5q9\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh\" (UID: \"388311ca-26b2-4bd9-b355-a5b4c7b459cb\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh" Nov 01 20:49:23.452753 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.452739 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/388311ca-26b2-4bd9-b355-a5b4c7b459cb-credential-socket\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh\" (UID: \"388311ca-26b2-4bd9-b355-a5b4c7b459cb\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh" Nov 01 20:49:23.452791 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.452772 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/388311ca-26b2-4bd9-b355-a5b4c7b459cb-workload-socket\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh\" (UID: \"388311ca-26b2-4bd9-b355-a5b4c7b459cb\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh" Nov 01 20:49:23.452840 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.452827 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/388311ca-26b2-4bd9-b355-a5b4c7b459cb-istiod-ca-cert\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh\" (UID: \"388311ca-26b2-4bd9-b355-a5b4c7b459cb\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh" Nov 01 20:49:23.452876 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.452852 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/388311ca-26b2-4bd9-b355-a5b4c7b459cb-istio-envoy\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh\" (UID: \"388311ca-26b2-4bd9-b355-a5b4c7b459cb\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh" Nov 01 20:49:23.482211 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.482177 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh"] Nov 01 20:49:23.547198 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.547165 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx"] Nov 01 20:49:23.554498 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.554464 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/388311ca-26b2-4bd9-b355-a5b4c7b459cb-istio-data\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh\" (UID: \"388311ca-26b2-4bd9-b355-a5b4c7b459cb\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh" Nov 01 20:49:23.554778 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.554755 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95617e0d-4bf3-435d-86ec-b6c87a0996b9-istio-data\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx\" (UID: \"95617e0d-4bf3-435d-86ec-b6c87a0996b9\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx" Nov 01 20:49:23.554956 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.554902 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/388311ca-26b2-4bd9-b355-a5b4c7b459cb-workload-certs\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh\" (UID: \"388311ca-26b2-4bd9-b355-a5b4c7b459cb\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh" Nov 01 20:49:23.555129 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.555111 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95617e0d-4bf3-435d-86ec-b6c87a0996b9-workload-socket\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx\" (UID: \"95617e0d-4bf3-435d-86ec-b6c87a0996b9\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx" Nov 01 20:49:23.555301 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.555286 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c757m\" (UniqueName: \"kubernetes.io/projected/95617e0d-4bf3-435d-86ec-b6c87a0996b9-kube-api-access-c757m\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx\" (UID: \"95617e0d-4bf3-435d-86ec-b6c87a0996b9\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx" Nov 01 20:49:23.555462 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.555446 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/388311ca-26b2-4bd9-b355-a5b4c7b459cb-istio-token\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh\" (UID: \"388311ca-26b2-4bd9-b355-a5b4c7b459cb\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh" Nov 01 20:49:23.555591 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.555576 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/388311ca-26b2-4bd9-b355-a5b4c7b459cb-istio-podinfo\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh\" (UID: \"388311ca-26b2-4bd9-b355-a5b4c7b459cb\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh" Nov 01 20:49:23.555697 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.555684 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf5q9\" (UniqueName: \"kubernetes.io/projected/388311ca-26b2-4bd9-b355-a5b4c7b459cb-kube-api-access-zf5q9\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh\" (UID: \"388311ca-26b2-4bd9-b355-a5b4c7b459cb\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh" Nov 01 20:49:23.555816 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.555803 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95617e0d-4bf3-435d-86ec-b6c87a0996b9-istio-envoy\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx\" (UID: \"95617e0d-4bf3-435d-86ec-b6c87a0996b9\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx" Nov 01 20:49:23.555927 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.555912 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95617e0d-4bf3-435d-86ec-b6c87a0996b9-credential-socket\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx\" (UID: \"95617e0d-4bf3-435d-86ec-b6c87a0996b9\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx" Nov 01 20:49:23.556065 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.556049 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/388311ca-26b2-4bd9-b355-a5b4c7b459cb-credential-socket\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh\" (UID: \"388311ca-26b2-4bd9-b355-a5b4c7b459cb\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh" Nov 01 20:49:23.556189 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.556172 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/388311ca-26b2-4bd9-b355-a5b4c7b459cb-workload-socket\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh\" (UID: \"388311ca-26b2-4bd9-b355-a5b4c7b459cb\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh" Nov 01 20:49:23.556305 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.556280 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95617e0d-4bf3-435d-86ec-b6c87a0996b9-istio-token\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx\" (UID: \"95617e0d-4bf3-435d-86ec-b6c87a0996b9\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx" Nov 01 20:49:23.556412 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.556335 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/388311ca-26b2-4bd9-b355-a5b4c7b459cb-istiod-ca-cert\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh\" (UID: \"388311ca-26b2-4bd9-b355-a5b4c7b459cb\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh" Nov 01 20:49:23.556412 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.556362 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/388311ca-26b2-4bd9-b355-a5b4c7b459cb-istio-envoy\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh\" (UID: \"388311ca-26b2-4bd9-b355-a5b4c7b459cb\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh" Nov 01 20:49:23.556412 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.556389 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95617e0d-4bf3-435d-86ec-b6c87a0996b9-workload-certs\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx\" (UID: \"95617e0d-4bf3-435d-86ec-b6c87a0996b9\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx" Nov 01 20:49:23.556512 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.556462 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95617e0d-4bf3-435d-86ec-b6c87a0996b9-istiod-ca-cert\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx\" (UID: \"95617e0d-4bf3-435d-86ec-b6c87a0996b9\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx" Nov 01 20:49:23.556550 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.556531 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95617e0d-4bf3-435d-86ec-b6c87a0996b9-istio-podinfo\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx\" (UID: \"95617e0d-4bf3-435d-86ec-b6c87a0996b9\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx" Nov 01 20:49:23.557111 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.557063 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/388311ca-26b2-4bd9-b355-a5b4c7b459cb-istio-data\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh\" (UID: \"388311ca-26b2-4bd9-b355-a5b4c7b459cb\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh" Nov 01 20:49:23.557210 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.557120 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/388311ca-26b2-4bd9-b355-a5b4c7b459cb-workload-certs\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh\" (UID: \"388311ca-26b2-4bd9-b355-a5b4c7b459cb\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh" Nov 01 20:49:23.557455 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.557435 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/388311ca-26b2-4bd9-b355-a5b4c7b459cb-credential-socket\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh\" (UID: \"388311ca-26b2-4bd9-b355-a5b4c7b459cb\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh" Nov 01 20:49:23.557730 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.557704 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/388311ca-26b2-4bd9-b355-a5b4c7b459cb-istiod-ca-cert\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh\" (UID: \"388311ca-26b2-4bd9-b355-a5b4c7b459cb\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh" Nov 01 20:49:23.557839 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.557788 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/388311ca-26b2-4bd9-b355-a5b4c7b459cb-workload-socket\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh\" (UID: \"388311ca-26b2-4bd9-b355-a5b4c7b459cb\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh" Nov 01 20:49:23.559885 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.559861 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/388311ca-26b2-4bd9-b355-a5b4c7b459cb-istio-envoy\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh\" (UID: \"388311ca-26b2-4bd9-b355-a5b4c7b459cb\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh" Nov 01 20:49:23.559885 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.559877 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/388311ca-26b2-4bd9-b355-a5b4c7b459cb-istio-podinfo\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh\" (UID: \"388311ca-26b2-4bd9-b355-a5b4c7b459cb\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh" Nov 01 20:49:23.616371 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.616334 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/388311ca-26b2-4bd9-b355-a5b4c7b459cb-istio-token\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh\" (UID: \"388311ca-26b2-4bd9-b355-a5b4c7b459cb\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh" Nov 01 20:49:23.616648 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.616622 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf5q9\" (UniqueName: \"kubernetes.io/projected/388311ca-26b2-4bd9-b355-a5b4c7b459cb-kube-api-access-zf5q9\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh\" (UID: \"388311ca-26b2-4bd9-b355-a5b4c7b459cb\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh" Nov 01 20:49:23.657622 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.657586 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95617e0d-4bf3-435d-86ec-b6c87a0996b9-istio-podinfo\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx\" (UID: \"95617e0d-4bf3-435d-86ec-b6c87a0996b9\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx" Nov 01 20:49:23.657796 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.657641 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95617e0d-4bf3-435d-86ec-b6c87a0996b9-istio-data\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx\" (UID: \"95617e0d-4bf3-435d-86ec-b6c87a0996b9\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx" Nov 01 20:49:23.657796 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.657670 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95617e0d-4bf3-435d-86ec-b6c87a0996b9-workload-socket\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx\" (UID: \"95617e0d-4bf3-435d-86ec-b6c87a0996b9\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx" Nov 01 20:49:23.657796 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.657712 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c757m\" (UniqueName: \"kubernetes.io/projected/95617e0d-4bf3-435d-86ec-b6c87a0996b9-kube-api-access-c757m\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx\" (UID: \"95617e0d-4bf3-435d-86ec-b6c87a0996b9\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx" Nov 01 20:49:23.657796 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.657775 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95617e0d-4bf3-435d-86ec-b6c87a0996b9-istio-envoy\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx\" (UID: \"95617e0d-4bf3-435d-86ec-b6c87a0996b9\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx" Nov 01 20:49:23.657796 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.657797 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95617e0d-4bf3-435d-86ec-b6c87a0996b9-credential-socket\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx\" (UID: \"95617e0d-4bf3-435d-86ec-b6c87a0996b9\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx" Nov 01 20:49:23.658111 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.657823 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95617e0d-4bf3-435d-86ec-b6c87a0996b9-istio-token\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx\" (UID: \"95617e0d-4bf3-435d-86ec-b6c87a0996b9\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx" Nov 01 20:49:23.658111 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.657921 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95617e0d-4bf3-435d-86ec-b6c87a0996b9-workload-certs\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx\" (UID: \"95617e0d-4bf3-435d-86ec-b6c87a0996b9\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx" Nov 01 20:49:23.658228 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.658156 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95617e0d-4bf3-435d-86ec-b6c87a0996b9-istio-data\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx\" (UID: \"95617e0d-4bf3-435d-86ec-b6c87a0996b9\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx" Nov 01 20:49:23.658228 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.658199 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95617e0d-4bf3-435d-86ec-b6c87a0996b9-workload-socket\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx\" (UID: \"95617e0d-4bf3-435d-86ec-b6c87a0996b9\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx" Nov 01 20:49:23.658312 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.658227 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95617e0d-4bf3-435d-86ec-b6c87a0996b9-istiod-ca-cert\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx\" (UID: \"95617e0d-4bf3-435d-86ec-b6c87a0996b9\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx" Nov 01 20:49:23.658312 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.658286 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95617e0d-4bf3-435d-86ec-b6c87a0996b9-credential-socket\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx\" (UID: \"95617e0d-4bf3-435d-86ec-b6c87a0996b9\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx" Nov 01 20:49:23.658415 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.658393 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95617e0d-4bf3-435d-86ec-b6c87a0996b9-workload-certs\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx\" (UID: \"95617e0d-4bf3-435d-86ec-b6c87a0996b9\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx" Nov 01 20:49:23.658777 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.658759 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95617e0d-4bf3-435d-86ec-b6c87a0996b9-istiod-ca-cert\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx\" (UID: \"95617e0d-4bf3-435d-86ec-b6c87a0996b9\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx" Nov 01 20:49:23.660522 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.660494 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95617e0d-4bf3-435d-86ec-b6c87a0996b9-istio-envoy\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx\" (UID: \"95617e0d-4bf3-435d-86ec-b6c87a0996b9\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx" Nov 01 20:49:23.660661 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.660641 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95617e0d-4bf3-435d-86ec-b6c87a0996b9-istio-podinfo\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx\" (UID: \"95617e0d-4bf3-435d-86ec-b6c87a0996b9\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx" Nov 01 20:49:23.699182 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.699100 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh" Nov 01 20:49:23.715290 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.715259 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95617e0d-4bf3-435d-86ec-b6c87a0996b9-istio-token\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx\" (UID: \"95617e0d-4bf3-435d-86ec-b6c87a0996b9\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx" Nov 01 20:49:23.715457 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.715439 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c757m\" (UniqueName: \"kubernetes.io/projected/95617e0d-4bf3-435d-86ec-b6c87a0996b9-kube-api-access-c757m\") pod \"e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx\" (UID: \"95617e0d-4bf3-435d-86ec-b6c87a0996b9\") " pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx" Nov 01 20:49:23.744463 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:23.744421 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx" Nov 01 20:49:24.128077 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:49:24.128048 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod388311ca_26b2_4bd9_b355_a5b4c7b459cb.slice/crio-bdb8eaa00f57d4ad6c31bbd9bb82dfa60fa474ccc92e516b0c060b6e63f22e62 WatchSource:0}: Error finding container bdb8eaa00f57d4ad6c31bbd9bb82dfa60fa474ccc92e516b0c060b6e63f22e62: Status 404 returned error can't find the container with id bdb8eaa00f57d4ad6c31bbd9bb82dfa60fa474ccc92e516b0c060b6e63f22e62 Nov 01 20:49:24.128534 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:24.128469 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh"] Nov 01 20:49:24.130452 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:24.130421 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:49:24.130515 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:24.130486 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:49:24.130550 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:24.130518 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:49:24.181481 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:49:24.181450 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95617e0d_4bf3_435d_86ec_b6c87a0996b9.slice/crio-2ebf960f365afc7cdf88dbd0b5043bdb1e26345d68121e79de614afd83b852be WatchSource:0}: Error finding container 2ebf960f365afc7cdf88dbd0b5043bdb1e26345d68121e79de614afd83b852be: Status 404 returned error can't find the container with id 2ebf960f365afc7cdf88dbd0b5043bdb1e26345d68121e79de614afd83b852be Nov 01 20:49:24.181844 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:24.181822 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx"] Nov 01 20:49:24.184488 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:24.184457 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:49:24.184577 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:24.184516 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:49:24.184577 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:24.184545 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:49:24.359128 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:24.359088 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:49:24.361539 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:24.361511 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx" event={"ID":"95617e0d-4bf3-435d-86ec-b6c87a0996b9","Type":"ContainerStarted","Data":"f1fca49a911b5a444a2588075e0dde0e95de2ea943a5e48f78b6946afefbce7f"} Nov 01 20:49:24.361691 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:24.361565 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx" event={"ID":"95617e0d-4bf3-435d-86ec-b6c87a0996b9","Type":"ContainerStarted","Data":"2ebf960f365afc7cdf88dbd0b5043bdb1e26345d68121e79de614afd83b852be"} Nov 01 20:49:24.363628 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:24.363599 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh" event={"ID":"388311ca-26b2-4bd9-b355-a5b4c7b459cb","Type":"ContainerStarted","Data":"9f1f72313446b073366578987cc4a9fef5c87d44bb84830c48a12af23e622def"} Nov 01 20:49:24.363755 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:24.363707 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh" event={"ID":"388311ca-26b2-4bd9-b355-a5b4c7b459cb","Type":"ContainerStarted","Data":"bdb8eaa00f57d4ad6c31bbd9bb82dfa60fa474ccc92e516b0c060b6e63f22e62"} Nov 01 20:49:24.495273 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:24.495219 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx" podStartSLOduration=1.495205725 podStartE2EDuration="1.495205725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:49:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:49:24.491306259 +0000 UTC m=+1115.268946397" watchObservedRunningTime="2026-11-01 20:49:24.495205725 +0000 UTC m=+1115.272845864" Nov 01 20:49:24.542500 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:24.542461 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq"] Nov 01 20:49:24.549460 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:24.548700 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq" Nov 01 20:49:24.607871 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:24.605870 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq"] Nov 01 20:49:24.657020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:24.656587 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh" podStartSLOduration=1.656563259 podStartE2EDuration="1.656563259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:49:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:49:24.621233036 +0000 UTC m=+1115.398873174" watchObservedRunningTime="2026-11-01 20:49:24.656563259 +0000 UTC m=+1115.434203395" Nov 01 20:49:24.730536 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:24.718097 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh" Nov 01 20:49:24.758205 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:24.757935 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx" Nov 01 20:49:24.760532 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:24.760498 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-credential-socket\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq\" (UID: \"50906dd8-8d76-4fcb-8415-f8f32cd47ffc\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq" Nov 01 20:49:24.760691 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:24.760672 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-workload-certs\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq\" (UID: \"50906dd8-8d76-4fcb-8415-f8f32cd47ffc\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq" Nov 01 20:49:24.760737 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:24.760723 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-workload-socket\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq\" (UID: \"50906dd8-8d76-4fcb-8415-f8f32cd47ffc\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq" Nov 01 20:49:24.760807 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:24.760783 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-istio-envoy\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq\" (UID: \"50906dd8-8d76-4fcb-8415-f8f32cd47ffc\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq" Nov 01 20:49:24.760912 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:24.760894 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-istiod-ca-cert\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq\" (UID: \"50906dd8-8d76-4fcb-8415-f8f32cd47ffc\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq" Nov 01 20:49:24.761050 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:24.761034 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-istio-podinfo\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq\" (UID: \"50906dd8-8d76-4fcb-8415-f8f32cd47ffc\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq" Nov 01 20:49:24.761101 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:24.761087 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-istio-token\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq\" (UID: \"50906dd8-8d76-4fcb-8415-f8f32cd47ffc\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq" Nov 01 20:49:24.761189 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:24.761175 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv9mq\" (UniqueName: \"kubernetes.io/projected/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-kube-api-access-nv9mq\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq\" (UID: \"50906dd8-8d76-4fcb-8415-f8f32cd47ffc\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq" Nov 01 20:49:24.761232 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:24.761217 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-istio-data\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq\" (UID: \"50906dd8-8d76-4fcb-8415-f8f32cd47ffc\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq" Nov 01 20:49:24.862227 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:24.862193 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-credential-socket\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq\" (UID: \"50906dd8-8d76-4fcb-8415-f8f32cd47ffc\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq" Nov 01 20:49:24.862401 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:24.862292 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-workload-certs\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq\" (UID: \"50906dd8-8d76-4fcb-8415-f8f32cd47ffc\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq" Nov 01 20:49:24.862401 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:24.862322 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-workload-socket\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq\" (UID: \"50906dd8-8d76-4fcb-8415-f8f32cd47ffc\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq" Nov 01 20:49:24.862401 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:24.862340 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-istio-envoy\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq\" (UID: \"50906dd8-8d76-4fcb-8415-f8f32cd47ffc\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq" Nov 01 20:49:24.862401 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:24.862393 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-istiod-ca-cert\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq\" (UID: \"50906dd8-8d76-4fcb-8415-f8f32cd47ffc\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq" Nov 01 20:49:24.862594 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:24.862459 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-istio-podinfo\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq\" (UID: \"50906dd8-8d76-4fcb-8415-f8f32cd47ffc\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq" Nov 01 20:49:24.862594 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:24.862504 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-istio-token\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq\" (UID: \"50906dd8-8d76-4fcb-8415-f8f32cd47ffc\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq" Nov 01 20:49:24.862594 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:24.862583 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv9mq\" (UniqueName: \"kubernetes.io/projected/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-kube-api-access-nv9mq\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq\" (UID: \"50906dd8-8d76-4fcb-8415-f8f32cd47ffc\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq" Nov 01 20:49:24.862760 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:24.862630 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-istio-data\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq\" (UID: \"50906dd8-8d76-4fcb-8415-f8f32cd47ffc\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq" Nov 01 20:49:24.862760 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:24.862676 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-credential-socket\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq\" (UID: \"50906dd8-8d76-4fcb-8415-f8f32cd47ffc\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq" Nov 01 20:49:24.862760 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:24.862745 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-workload-certs\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq\" (UID: \"50906dd8-8d76-4fcb-8415-f8f32cd47ffc\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq" Nov 01 20:49:24.863171 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:24.863149 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-workload-socket\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq\" (UID: \"50906dd8-8d76-4fcb-8415-f8f32cd47ffc\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq" Nov 01 20:49:24.863320 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:24.863300 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-istio-data\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq\" (UID: \"50906dd8-8d76-4fcb-8415-f8f32cd47ffc\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq" Nov 01 20:49:24.863476 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:24.863459 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-istiod-ca-cert\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq\" (UID: \"50906dd8-8d76-4fcb-8415-f8f32cd47ffc\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq" Nov 01 20:49:24.864807 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:24.864777 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-istio-envoy\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq\" (UID: \"50906dd8-8d76-4fcb-8415-f8f32cd47ffc\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq" Nov 01 20:49:24.865936 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:24.865897 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-istio-podinfo\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq\" (UID: \"50906dd8-8d76-4fcb-8415-f8f32cd47ffc\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq" Nov 01 20:49:24.920888 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:24.920852 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-istio-token\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq\" (UID: \"50906dd8-8d76-4fcb-8415-f8f32cd47ffc\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq" Nov 01 20:49:24.921139 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:24.921114 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv9mq\" (UniqueName: \"kubernetes.io/projected/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-kube-api-access-nv9mq\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq\" (UID: \"50906dd8-8d76-4fcb-8415-f8f32cd47ffc\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq" Nov 01 20:49:24.929847 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:24.929819 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq" Nov 01 20:49:25.092237 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:25.092152 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b23881c3-9m222"] Nov 01 20:49:25.097695 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:25.097665 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23881c3-9m222" Nov 01 20:49:25.150939 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:25.150906 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:49:25.151176 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:25.151067 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:49:25.151176 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:25.151112 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:49:25.151176 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:25.151115 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 20:49:25.152100 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:25.152077 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b23881c3-9m222"] Nov 01 20:49:25.167074 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:25.167040 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msc8c\" (UniqueName: \"kubernetes.io/projected/2704b115-3839-4088-bc5b-fb6fc97760ee-kube-api-access-msc8c\") pod \"maas-api-revoke-sub-b23881c3-9m222\" (UID: \"2704b115-3839-4088-bc5b-fb6fc97760ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23881c3-9m222" Nov 01 20:49:25.167269 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:25.167240 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2704b115-3839-4088-bc5b-fb6fc97760ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b23881c3-9m222\" (UID: \"2704b115-3839-4088-bc5b-fb6fc97760ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23881c3-9m222" Nov 01 20:49:25.167480 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:25.167460 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2704b115-3839-4088-bc5b-fb6fc97760ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b23881c3-9m222\" (UID: \"2704b115-3839-4088-bc5b-fb6fc97760ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23881c3-9m222" Nov 01 20:49:25.269828 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:25.269776 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2704b115-3839-4088-bc5b-fb6fc97760ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b23881c3-9m222\" (UID: \"2704b115-3839-4088-bc5b-fb6fc97760ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23881c3-9m222" Nov 01 20:49:25.270369 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:25.269878 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msc8c\" (UniqueName: \"kubernetes.io/projected/2704b115-3839-4088-bc5b-fb6fc97760ee-kube-api-access-msc8c\") pod \"maas-api-revoke-sub-b23881c3-9m222\" (UID: \"2704b115-3839-4088-bc5b-fb6fc97760ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23881c3-9m222" Nov 01 20:49:25.270369 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:25.270031 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2704b115-3839-4088-bc5b-fb6fc97760ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b23881c3-9m222\" (UID: \"2704b115-3839-4088-bc5b-fb6fc97760ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23881c3-9m222" Nov 01 20:49:25.280028 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:25.271308 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2704b115-3839-4088-bc5b-fb6fc97760ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b23881c3-9m222\" (UID: \"2704b115-3839-4088-bc5b-fb6fc97760ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23881c3-9m222" Nov 01 20:49:25.280028 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:25.274589 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2704b115-3839-4088-bc5b-fb6fc97760ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b23881c3-9m222\" (UID: \"2704b115-3839-4088-bc5b-fb6fc97760ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23881c3-9m222" Nov 01 20:49:25.356132 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:25.355664 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msc8c\" (UniqueName: \"kubernetes.io/projected/2704b115-3839-4088-bc5b-fb6fc97760ee-kube-api-access-msc8c\") pod \"maas-api-revoke-sub-b23881c3-9m222\" (UID: \"2704b115-3839-4088-bc5b-fb6fc97760ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23881c3-9m222" Nov 01 20:49:25.363662 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:49:25.363622 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50906dd8_8d76_4fcb_8415_f8f32cd47ffc.slice/crio-933787bfbcd161ad0a94fd956a8729306dd6b6ae5e8523e40f4e85aec6a16a16 WatchSource:0}: Error finding container 933787bfbcd161ad0a94fd956a8729306dd6b6ae5e8523e40f4e85aec6a16a16: Status 404 returned error can't find the container with id 933787bfbcd161ad0a94fd956a8729306dd6b6ae5e8523e40f4e85aec6a16a16 Nov 01 20:49:25.366090 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:25.366055 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:49:25.366181 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:25.366118 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:49:25.366181 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:25.366156 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:49:25.373706 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:25.373683 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq"] Nov 01 20:49:25.419030 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:25.417425 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23881c3-9m222" Nov 01 20:49:25.766953 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:25.764842 4484 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 01 20:49:25.766953 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:25.764904 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh" podUID="388311ca-26b2-4bd9-b355-a5b4c7b459cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 01 20:49:25.769211 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:25.769175 4484 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 01 20:49:25.769349 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:25.769233 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx" podUID="95617e0d-4bf3-435d-86ec-b6c87a0996b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 01 20:49:25.941019 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:25.939496 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:49:25.991286 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:25.990496 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b23881c3-9m222"] Nov 01 20:49:26.096038 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:26.095983 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:49:26.382327 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:26.382239 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23881c3-9m222" event={"ID":"2704b115-3839-4088-bc5b-fb6fc97760ee","Type":"ContainerStarted","Data":"77c90d70ef4495f275da969e3106f9075b13aead67cdaea64b0cbddbc2fd4729"} Nov 01 20:49:26.384252 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:26.384030 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq" event={"ID":"50906dd8-8d76-4fcb-8415-f8f32cd47ffc","Type":"ContainerStarted","Data":"7129fcd315cf108e1128365007816b2d9879a2a0ee5c21d8a4794338bc86c9ee"} Nov 01 20:49:26.384252 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:26.384086 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq" event={"ID":"50906dd8-8d76-4fcb-8415-f8f32cd47ffc","Type":"ContainerStarted","Data":"933787bfbcd161ad0a94fd956a8729306dd6b6ae5e8523e40f4e85aec6a16a16"} Nov 01 20:49:26.523022 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:26.515508 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq" podStartSLOduration=2.515485333 podStartE2EDuration="2.515485333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:49:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:49:26.502386302 +0000 UTC m=+1117.280026436" watchObservedRunningTime="2026-11-01 20:49:26.515485333 +0000 UTC m=+1117.293125471" Nov 01 20:49:26.702138 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:26.701405 4484 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 01 20:49:26.702138 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:26.701463 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh" podUID="388311ca-26b2-4bd9-b355-a5b4c7b459cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 01 20:49:26.745567 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:26.745434 4484 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 01 20:49:26.745567 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:26.745496 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx" podUID="95617e0d-4bf3-435d-86ec-b6c87a0996b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 01 20:49:26.934754 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:26.933255 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq" Nov 01 20:49:26.961025 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:26.960829 4484 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 01 20:49:26.961025 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:26.960904 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq" podUID="50906dd8-8d76-4fcb-8415-f8f32cd47ffc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 01 20:49:27.399252 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:27.399219 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23881c3-9m222" event={"ID":"2704b115-3839-4088-bc5b-fb6fc97760ee","Type":"ContainerStarted","Data":"0ddc06b83759c2a680a9823fcaa63fdcdae437040acef41710457b2936e664d1"} Nov 01 20:49:27.595165 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:27.595106 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23881c3-9m222" podStartSLOduration=2.5950877549999998 podStartE2EDuration="2.595087755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:49:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:49:27.588697783 +0000 UTC m=+1118.366337919" watchObservedRunningTime="2026-11-01 20:49:27.595087755 +0000 UTC m=+1118.372727889" Nov 01 20:49:27.700215 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:27.700123 4484 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 01 20:49:27.700215 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:27.700172 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh" podUID="388311ca-26b2-4bd9-b355-a5b4c7b459cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 01 20:49:27.747026 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:27.746474 4484 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 01 20:49:27.747026 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:27.746541 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx" podUID="95617e0d-4bf3-435d-86ec-b6c87a0996b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 01 20:49:27.933024 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:27.932416 4484 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 01 20:49:27.933024 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:27.932478 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq" podUID="50906dd8-8d76-4fcb-8415-f8f32cd47ffc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 01 20:49:28.111020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:28.108520 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:49:28.439128 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:28.437704 4484 generic.go:356] "Generic (PLEG): container finished" podID="2704b115-3839-4088-bc5b-fb6fc97760ee" containerID="0ddc06b83759c2a680a9823fcaa63fdcdae437040acef41710457b2936e664d1" exitCode=0 Nov 01 20:49:28.440180 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:28.440063 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23881c3-9m222" event={"ID":"2704b115-3839-4088-bc5b-fb6fc97760ee","Type":"ContainerDied","Data":"0ddc06b83759c2a680a9823fcaa63fdcdae437040acef41710457b2936e664d1"} Nov 01 20:49:28.699875 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:28.699776 4484 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 01 20:49:28.699875 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:28.699838 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh" podUID="388311ca-26b2-4bd9-b355-a5b4c7b459cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 01 20:49:28.769152 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:28.769116 4484 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 01 20:49:28.769445 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:28.769419 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx" podUID="95617e0d-4bf3-435d-86ec-b6c87a0996b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 01 20:49:28.935551 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:28.935504 4484 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 01 20:49:28.935851 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:28.935805 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq" podUID="50906dd8-8d76-4fcb-8415-f8f32cd47ffc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 01 20:49:29.712021 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:29.711422 4484 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 01 20:49:29.712021 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:29.711484 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh" podUID="388311ca-26b2-4bd9-b355-a5b4c7b459cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 01 20:49:29.797080 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:29.759119 4484 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 01 20:49:29.797080 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:29.759175 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx" podUID="95617e0d-4bf3-435d-86ec-b6c87a0996b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 01 20:49:29.992359 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:29.992312 4484 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 01 20:49:29.992687 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:29.992656 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq" podUID="50906dd8-8d76-4fcb-8415-f8f32cd47ffc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 01 20:49:30.080646 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:30.080546 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23881c3-9m222" Nov 01 20:49:30.099667 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:30.099623 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2704b115-3839-4088-bc5b-fb6fc97760ee-kube-api-access-msc8c\" (UniqueName: \"kubernetes.io/projected/2704b115-3839-4088-bc5b-fb6fc97760ee-kube-api-access-msc8c\") pod \"2704b115-3839-4088-bc5b-fb6fc97760ee\" (UID: \"2704b115-3839-4088-bc5b-fb6fc97760ee\") " Nov 01 20:49:30.099817 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:30.099703 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2704b115-3839-4088-bc5b-fb6fc97760ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2704b115-3839-4088-bc5b-fb6fc97760ee-maas-api-xks-ca\") pod \"2704b115-3839-4088-bc5b-fb6fc97760ee\" (UID: \"2704b115-3839-4088-bc5b-fb6fc97760ee\") " Nov 01 20:49:30.102458 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:30.102416 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2704b115-3839-4088-bc5b-fb6fc97760ee-maas-api-xks-ca" pod "2704b115-3839-4088-bc5b-fb6fc97760ee" (UID: "2704b115-3839-4088-bc5b-fb6fc97760ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:49:30.102644 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:30.102498 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2704b115-3839-4088-bc5b-fb6fc97760ee-kube-api-access-msc8c" pod "2704b115-3839-4088-bc5b-fb6fc97760ee" (UID: "2704b115-3839-4088-bc5b-fb6fc97760ee"). InnerVolumeSpecName "kube-api-access-msc8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:49:30.200690 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:30.200638 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2704b115-3839-4088-bc5b-fb6fc97760ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2704b115-3839-4088-bc5b-fb6fc97760ee-maas-api-openshift-ca\") pod \"2704b115-3839-4088-bc5b-fb6fc97760ee\" (UID: \"2704b115-3839-4088-bc5b-fb6fc97760ee\") " Nov 01 20:49:30.201085 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:30.201067 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msc8c\" (UniqueName: \"kubernetes.io/projected/2704b115-3839-4088-bc5b-fb6fc97760ee-kube-api-access-msc8c\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:49:30.201085 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:30.201085 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2704b115-3839-4088-bc5b-fb6fc97760ee-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:49:30.201398 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:30.201377 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2704b115-3839-4088-bc5b-fb6fc97760ee-maas-api-openshift-ca" pod "2704b115-3839-4088-bc5b-fb6fc97760ee" (UID: "2704b115-3839-4088-bc5b-fb6fc97760ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:49:30.304021 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:30.303274 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2704b115-3839-4088-bc5b-fb6fc97760ee-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:49:30.619022 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:30.617289 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:49:30.720025 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:30.700258 4484 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 01 20:49:30.720025 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:30.700317 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh" podUID="388311ca-26b2-4bd9-b355-a5b4c7b459cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 01 20:49:30.766050 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:30.750397 4484 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 01 20:49:30.766050 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:30.750473 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx" podUID="95617e0d-4bf3-435d-86ec-b6c87a0996b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 01 20:49:30.923073 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:30.897786 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23881c3-9m222" event={"ID":"2704b115-3839-4088-bc5b-fb6fc97760ee","Type":"ContainerDied","Data":"77c90d70ef4495f275da969e3106f9075b13aead67cdaea64b0cbddbc2fd4729"} Nov 01 20:49:30.923073 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:30.897853 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77c90d70ef4495f275da969e3106f9075b13aead67cdaea64b0cbddbc2fd4729" Nov 01 20:49:30.923073 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:30.897925 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23881c3-9m222" Nov 01 20:49:30.940095 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:30.939922 4484 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 01 20:49:30.940095 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:30.939985 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq" podUID="50906dd8-8d76-4fcb-8415-f8f32cd47ffc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 01 20:49:31.749019 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:31.747403 4484 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 01 20:49:31.749019 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:31.747463 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh" podUID="388311ca-26b2-4bd9-b355-a5b4c7b459cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 01 20:49:31.767035 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:31.762307 4484 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 01 20:49:31.767035 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:31.762368 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx" podUID="95617e0d-4bf3-435d-86ec-b6c87a0996b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 01 20:49:31.954934 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:31.952131 4484 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 01 20:49:31.954934 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:31.952196 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq" podUID="50906dd8-8d76-4fcb-8415-f8f32cd47ffc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 01 20:49:32.700355 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:32.700301 4484 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 01 20:49:32.700578 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:32.700369 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh" podUID="388311ca-26b2-4bd9-b355-a5b4c7b459cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 01 20:49:32.931291 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:32.931250 4484 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 01 20:49:32.931787 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:32.931316 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq" podUID="50906dd8-8d76-4fcb-8415-f8f32cd47ffc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 01 20:49:33.174979 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:33.174645 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:49:33.544365 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:33.544328 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx" Nov 01 20:49:33.545857 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:33.545830 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx" Nov 01 20:49:33.546880 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:33.546859 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx" Nov 01 20:49:33.704656 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:33.704624 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh" Nov 01 20:49:33.704939 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:33.704920 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh" Nov 01 20:49:33.705791 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:33.705772 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh" Nov 01 20:49:33.930877 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:33.930800 4484 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 01 20:49:33.930877 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:33.930850 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq" podUID="50906dd8-8d76-4fcb-8415-f8f32cd47ffc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 01 20:49:34.934827 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:34.934797 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq" Nov 01 20:49:34.935280 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:34.935262 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq" Nov 01 20:49:34.935890 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:34.935872 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq" Nov 01 20:49:35.830168 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:35.830131 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0393ace-lv62v"] Nov 01 20:49:35.830784 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:35.830769 4484 cpu_manager.go:414] "containerMap: removing container" podUID="2704b115-3839-4088-bc5b-fb6fc97760ee" containerName="revoke-keys" Nov 01 20:49:35.830838 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:35.830784 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2704b115-3839-4088-bc5b-fb6fc97760ee" containerName="revoke-keys" Nov 01 20:49:35.830912 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:35.830902 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="2704b115-3839-4088-bc5b-fb6fc97760ee" containerName="revoke-keys" Nov 01 20:49:35.834807 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:35.834787 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0393ace-lv62v" Nov 01 20:49:35.847520 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:35.847496 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d235567c-ae00-4a30-ac29-3ef4cc1e4bbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0393ace-lv62v\" (UID: \"d235567c-ae00-4a30-ac29-3ef4cc1e4bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0393ace-lv62v" Nov 01 20:49:35.847699 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:35.847681 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kvhf\" (UniqueName: \"kubernetes.io/projected/d235567c-ae00-4a30-ac29-3ef4cc1e4bbb-kube-api-access-7kvhf\") pod \"maas-api-revoke-sub-c0393ace-lv62v\" (UID: \"d235567c-ae00-4a30-ac29-3ef4cc1e4bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0393ace-lv62v" Nov 01 20:49:35.847758 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:35.847744 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d235567c-ae00-4a30-ac29-3ef4cc1e4bbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0393ace-lv62v\" (UID: \"d235567c-ae00-4a30-ac29-3ef4cc1e4bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0393ace-lv62v" Nov 01 20:49:35.890812 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:35.890779 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0393ace-lv62v"] Nov 01 20:49:35.936603 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:35.936569 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 20:49:35.936603 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:35.936599 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:49:35.936603 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:35.936585 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:49:35.937094 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:35.936682 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:49:35.948816 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:35.948786 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kvhf\" (UniqueName: \"kubernetes.io/projected/d235567c-ae00-4a30-ac29-3ef4cc1e4bbb-kube-api-access-7kvhf\") pod \"maas-api-revoke-sub-c0393ace-lv62v\" (UID: \"d235567c-ae00-4a30-ac29-3ef4cc1e4bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0393ace-lv62v" Nov 01 20:49:35.948954 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:35.948846 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d235567c-ae00-4a30-ac29-3ef4cc1e4bbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0393ace-lv62v\" (UID: \"d235567c-ae00-4a30-ac29-3ef4cc1e4bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0393ace-lv62v" Nov 01 20:49:35.948954 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:35.948916 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d235567c-ae00-4a30-ac29-3ef4cc1e4bbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0393ace-lv62v\" (UID: \"d235567c-ae00-4a30-ac29-3ef4cc1e4bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0393ace-lv62v" Nov 01 20:49:35.949565 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:35.949542 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d235567c-ae00-4a30-ac29-3ef4cc1e4bbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0393ace-lv62v\" (UID: \"d235567c-ae00-4a30-ac29-3ef4cc1e4bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0393ace-lv62v" Nov 01 20:49:35.951404 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:35.951382 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d235567c-ae00-4a30-ac29-3ef4cc1e4bbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0393ace-lv62v\" (UID: \"d235567c-ae00-4a30-ac29-3ef4cc1e4bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0393ace-lv62v" Nov 01 20:49:36.006253 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:36.006222 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kvhf\" (UniqueName: \"kubernetes.io/projected/d235567c-ae00-4a30-ac29-3ef4cc1e4bbb-kube-api-access-7kvhf\") pod \"maas-api-revoke-sub-c0393ace-lv62v\" (UID: \"d235567c-ae00-4a30-ac29-3ef4cc1e4bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0393ace-lv62v" Nov 01 20:49:36.148463 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:36.148359 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0393ace-lv62v" Nov 01 20:49:36.531228 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:49:36.531196 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd235567c_ae00_4a30_ac29_3ef4cc1e4bbb.slice/crio-5372a8a3540bb9945c156b6a665d1a4e739b090f55bd4e20e85e4c1abdb67649 WatchSource:0}: Error finding container 5372a8a3540bb9945c156b6a665d1a4e739b090f55bd4e20e85e4c1abdb67649: Status 404 returned error can't find the container with id 5372a8a3540bb9945c156b6a665d1a4e739b090f55bd4e20e85e4c1abdb67649 Nov 01 20:49:36.531583 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:36.531553 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0393ace-lv62v"] Nov 01 20:49:37.054111 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:37.054065 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0393ace-lv62v" event={"ID":"d235567c-ae00-4a30-ac29-3ef4cc1e4bbb","Type":"ContainerStarted","Data":"2f8a615f8064bdf305c7e3389389f5c5e0eb090a289042145801356573514750"} Nov 01 20:49:37.054613 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:37.054132 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0393ace-lv62v" event={"ID":"d235567c-ae00-4a30-ac29-3ef4cc1e4bbb","Type":"ContainerStarted","Data":"5372a8a3540bb9945c156b6a665d1a4e739b090f55bd4e20e85e4c1abdb67649"} Nov 01 20:49:37.177186 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:37.177066 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0393ace-lv62v" podStartSLOduration=2.177046435 podStartE2EDuration="2.177046435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:49:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:49:37.169955473 +0000 UTC m=+1127.947595610" watchObservedRunningTime="2026-11-01 20:49:37.177046435 +0000 UTC m=+1127.954686571" Nov 01 20:49:37.931873 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:37.931840 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:49:38.063878 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:38.063846 4484 generic.go:356] "Generic (PLEG): container finished" podID="d235567c-ae00-4a30-ac29-3ef4cc1e4bbb" containerID="2f8a615f8064bdf305c7e3389389f5c5e0eb090a289042145801356573514750" exitCode=0 Nov 01 20:49:38.064315 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:38.063936 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0393ace-lv62v" event={"ID":"d235567c-ae00-4a30-ac29-3ef4cc1e4bbb","Type":"ContainerDied","Data":"2f8a615f8064bdf305c7e3389389f5c5e0eb090a289042145801356573514750"} Nov 01 20:49:39.203613 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:39.203566 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:49:39.248362 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:39.248338 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0393ace-lv62v" Nov 01 20:49:39.290147 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:39.290115 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d235567c-ae00-4a30-ac29-3ef4cc1e4bbb-kube-api-access-7kvhf\" (UniqueName: \"kubernetes.io/projected/d235567c-ae00-4a30-ac29-3ef4cc1e4bbb-kube-api-access-7kvhf\") pod \"d235567c-ae00-4a30-ac29-3ef4cc1e4bbb\" (UID: \"d235567c-ae00-4a30-ac29-3ef4cc1e4bbb\") " Nov 01 20:49:39.290300 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:39.290285 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d235567c-ae00-4a30-ac29-3ef4cc1e4bbb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d235567c-ae00-4a30-ac29-3ef4cc1e4bbb-maas-api-xks-ca\") pod \"d235567c-ae00-4a30-ac29-3ef4cc1e4bbb\" (UID: \"d235567c-ae00-4a30-ac29-3ef4cc1e4bbb\") " Nov 01 20:49:39.290371 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:39.290322 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d235567c-ae00-4a30-ac29-3ef4cc1e4bbb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d235567c-ae00-4a30-ac29-3ef4cc1e4bbb-maas-api-openshift-ca\") pod \"d235567c-ae00-4a30-ac29-3ef4cc1e4bbb\" (UID: \"d235567c-ae00-4a30-ac29-3ef4cc1e4bbb\") " Nov 01 20:49:39.290692 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:39.290664 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d235567c-ae00-4a30-ac29-3ef4cc1e4bbb-maas-api-openshift-ca" pod "d235567c-ae00-4a30-ac29-3ef4cc1e4bbb" (UID: "d235567c-ae00-4a30-ac29-3ef4cc1e4bbb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:49:39.292430 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:39.292402 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d235567c-ae00-4a30-ac29-3ef4cc1e4bbb-maas-api-xks-ca" pod "d235567c-ae00-4a30-ac29-3ef4cc1e4bbb" (UID: "d235567c-ae00-4a30-ac29-3ef4cc1e4bbb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:49:39.292532 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:39.292434 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d235567c-ae00-4a30-ac29-3ef4cc1e4bbb-kube-api-access-7kvhf" pod "d235567c-ae00-4a30-ac29-3ef4cc1e4bbb" (UID: "d235567c-ae00-4a30-ac29-3ef4cc1e4bbb"). InnerVolumeSpecName "kube-api-access-7kvhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:49:39.368218 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:39.368126 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:49:39.391790 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:39.391761 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d235567c-ae00-4a30-ac29-3ef4cc1e4bbb-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:49:39.391790 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:39.391786 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kvhf\" (UniqueName: \"kubernetes.io/projected/d235567c-ae00-4a30-ac29-3ef4cc1e4bbb-kube-api-access-7kvhf\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:49:39.391790 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:39.391795 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d235567c-ae00-4a30-ac29-3ef4cc1e4bbb-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:49:40.079681 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:40.079653 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0393ace-lv62v" Nov 01 20:49:40.079844 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:40.079685 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0393ace-lv62v" event={"ID":"d235567c-ae00-4a30-ac29-3ef4cc1e4bbb","Type":"ContainerDied","Data":"5372a8a3540bb9945c156b6a665d1a4e739b090f55bd4e20e85e4c1abdb67649"} Nov 01 20:49:40.079844 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:40.079728 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5372a8a3540bb9945c156b6a665d1a4e739b090f55bd4e20e85e4c1abdb67649" Nov 01 20:49:42.744775 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:42.744730 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:49:45.767866 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:49:45.767829 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 20:49:47.497121 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:47.497027 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:49:48.920020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:48.908095 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" podUID="fe8307db-264d-4b8a-b097-f180bc255a9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:49:49.011152 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:49.011097 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" podUID="17a9ac24-19cb-48c4-99b5-d74ce8aac0a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:49:49.289022 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:49.285099 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" podUID="70b4dacf-b3ac-46ec-99e8-a44f05587e7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:49:49.302970 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:49.302925 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" podUID="e175b298-2c23-45d4-8f9c-0c9b9f6aab92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:49:49.471019 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:49.426319 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" podUID="9f19be29-2c84-4ae4-9674-acca4905f4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:49:49.677206 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:49.676112 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" podUID="a65d9fd8-6193-4d03-9a3c-123f48b5dcdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:49:49.752214 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:49.752149 4484 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:49:49.752551 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:49.752521 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh" podUID="388311ca-26b2-4bd9-b355-a5b4c7b459cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:49:49.785032 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:49.784095 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" podUID="311f9699-6585-4a2c-8c20-e6b696b2f857" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:49:49.940221 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:49.940125 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" podUID="fe8307db-264d-4b8a-b097-f180bc255a9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:49:49.959033 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:49.941941 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" Nov 01 20:49:50.040445 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:50.040379 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" podUID="17a9ac24-19cb-48c4-99b5-d74ce8aac0a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:49:50.040826 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:50.040800 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" Nov 01 20:49:50.384021 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:50.309379 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" podUID="e175b298-2c23-45d4-8f9c-0c9b9f6aab92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:49:50.384021 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:50.326297 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" Nov 01 20:49:50.538077 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:50.538033 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" podUID="9f19be29-2c84-4ae4-9674-acca4905f4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:49:50.637093 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:50.550848 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" podUID="70b4dacf-b3ac-46ec-99e8-a44f05587e7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:49:50.662788 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:50.604462 4484 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" podUID="9f19be29-2c84-4ae4-9674-acca4905f4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:49:50.713315 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:50.713276 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" Nov 01 20:49:50.714070 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:50.714048 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" Nov 01 20:49:50.818022 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:50.805337 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" podUID="311f9699-6585-4a2c-8c20-e6b696b2f857" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:49:50.864190 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:50.805576 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" podUID="a65d9fd8-6193-4d03-9a3c-123f48b5dcdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:49:51.049025 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:51.036113 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" podUID="fe8307db-264d-4b8a-b097-f180bc255a9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:49:51.049025 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:51.039701 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" Nov 01 20:49:51.071029 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:51.070668 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" podUID="17a9ac24-19cb-48c4-99b5-d74ce8aac0a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:49:51.075030 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:51.072622 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" Nov 01 20:49:51.211445 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:51.211416 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" Nov 01 20:49:51.213581 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:51.213269 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" Nov 01 20:49:51.231167 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:51.231136 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" Nov 01 20:49:51.234864 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:51.234834 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" Nov 01 20:49:51.239095 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:51.239067 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" Nov 01 20:49:51.239381 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:51.239357 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" Nov 01 20:49:51.245162 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:51.244970 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" Nov 01 20:49:52.680876 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:52.680791 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:49:54.103362 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:54.103333 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt"] Nov 01 20:49:54.103974 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:54.103958 4484 cpu_manager.go:414] "containerMap: removing container" podUID="d235567c-ae00-4a30-ac29-3ef4cc1e4bbb" containerName="revoke-keys" Nov 01 20:49:54.104044 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:54.103975 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d235567c-ae00-4a30-ac29-3ef4cc1e4bbb" containerName="revoke-keys" Nov 01 20:49:54.104150 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:54.104138 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="d235567c-ae00-4a30-ac29-3ef4cc1e4bbb" containerName="revoke-keys" Nov 01 20:49:54.109338 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:54.109306 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt" Nov 01 20:49:54.209802 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:54.209770 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt"] Nov 01 20:49:54.235684 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:54.235648 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b446c1d-56a2-4b12-923e-2ca1874321ed-workload-socket\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt\" (UID: \"7b446c1d-56a2-4b12-923e-2ca1874321ed\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt" Nov 01 20:49:54.236006 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:54.235812 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b446c1d-56a2-4b12-923e-2ca1874321ed-workload-certs\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt\" (UID: \"7b446c1d-56a2-4b12-923e-2ca1874321ed\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt" Nov 01 20:49:54.236006 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:54.235842 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b446c1d-56a2-4b12-923e-2ca1874321ed-istio-podinfo\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt\" (UID: \"7b446c1d-56a2-4b12-923e-2ca1874321ed\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt" Nov 01 20:49:54.236006 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:54.235870 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b446c1d-56a2-4b12-923e-2ca1874321ed-istiod-ca-cert\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt\" (UID: \"7b446c1d-56a2-4b12-923e-2ca1874321ed\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt" Nov 01 20:49:54.236369 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:54.236009 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjjg6\" (UniqueName: \"kubernetes.io/projected/7b446c1d-56a2-4b12-923e-2ca1874321ed-kube-api-access-qjjg6\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt\" (UID: \"7b446c1d-56a2-4b12-923e-2ca1874321ed\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt" Nov 01 20:49:54.236369 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:54.236128 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b446c1d-56a2-4b12-923e-2ca1874321ed-istio-envoy\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt\" (UID: \"7b446c1d-56a2-4b12-923e-2ca1874321ed\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt" Nov 01 20:49:54.236369 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:54.236162 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b446c1d-56a2-4b12-923e-2ca1874321ed-istio-data\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt\" (UID: \"7b446c1d-56a2-4b12-923e-2ca1874321ed\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt" Nov 01 20:49:54.236369 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:54.236292 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b446c1d-56a2-4b12-923e-2ca1874321ed-istio-token\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt\" (UID: \"7b446c1d-56a2-4b12-923e-2ca1874321ed\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt" Nov 01 20:49:54.236369 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:54.236318 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b446c1d-56a2-4b12-923e-2ca1874321ed-credential-socket\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt\" (UID: \"7b446c1d-56a2-4b12-923e-2ca1874321ed\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt" Nov 01 20:49:54.337926 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:54.337894 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b446c1d-56a2-4b12-923e-2ca1874321ed-istio-token\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt\" (UID: \"7b446c1d-56a2-4b12-923e-2ca1874321ed\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt" Nov 01 20:49:54.337926 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:54.337938 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b446c1d-56a2-4b12-923e-2ca1874321ed-credential-socket\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt\" (UID: \"7b446c1d-56a2-4b12-923e-2ca1874321ed\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt" Nov 01 20:49:54.338320 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:54.337976 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b446c1d-56a2-4b12-923e-2ca1874321ed-workload-socket\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt\" (UID: \"7b446c1d-56a2-4b12-923e-2ca1874321ed\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt" Nov 01 20:49:54.338320 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:54.338094 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b446c1d-56a2-4b12-923e-2ca1874321ed-workload-certs\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt\" (UID: \"7b446c1d-56a2-4b12-923e-2ca1874321ed\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt" Nov 01 20:49:54.338320 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:54.338119 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b446c1d-56a2-4b12-923e-2ca1874321ed-istio-podinfo\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt\" (UID: \"7b446c1d-56a2-4b12-923e-2ca1874321ed\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt" Nov 01 20:49:54.338320 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:54.338146 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b446c1d-56a2-4b12-923e-2ca1874321ed-istiod-ca-cert\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt\" (UID: \"7b446c1d-56a2-4b12-923e-2ca1874321ed\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt" Nov 01 20:49:54.338320 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:54.338217 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjjg6\" (UniqueName: \"kubernetes.io/projected/7b446c1d-56a2-4b12-923e-2ca1874321ed-kube-api-access-qjjg6\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt\" (UID: \"7b446c1d-56a2-4b12-923e-2ca1874321ed\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt" Nov 01 20:49:54.338320 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:54.338302 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b446c1d-56a2-4b12-923e-2ca1874321ed-istio-envoy\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt\" (UID: \"7b446c1d-56a2-4b12-923e-2ca1874321ed\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt" Nov 01 20:49:54.338687 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:54.338330 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b446c1d-56a2-4b12-923e-2ca1874321ed-istio-data\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt\" (UID: \"7b446c1d-56a2-4b12-923e-2ca1874321ed\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt" Nov 01 20:49:54.338687 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:54.338427 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b446c1d-56a2-4b12-923e-2ca1874321ed-credential-socket\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt\" (UID: \"7b446c1d-56a2-4b12-923e-2ca1874321ed\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt" Nov 01 20:49:54.338800 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:54.338751 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b446c1d-56a2-4b12-923e-2ca1874321ed-workload-socket\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt\" (UID: \"7b446c1d-56a2-4b12-923e-2ca1874321ed\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt" Nov 01 20:49:54.339036 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:54.338983 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b446c1d-56a2-4b12-923e-2ca1874321ed-istio-data\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt\" (UID: \"7b446c1d-56a2-4b12-923e-2ca1874321ed\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt" Nov 01 20:49:54.339314 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:54.339284 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b446c1d-56a2-4b12-923e-2ca1874321ed-workload-certs\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt\" (UID: \"7b446c1d-56a2-4b12-923e-2ca1874321ed\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt" Nov 01 20:49:54.339454 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:54.339435 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b446c1d-56a2-4b12-923e-2ca1874321ed-istiod-ca-cert\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt\" (UID: \"7b446c1d-56a2-4b12-923e-2ca1874321ed\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt" Nov 01 20:49:54.340838 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:54.340812 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b446c1d-56a2-4b12-923e-2ca1874321ed-istio-podinfo\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt\" (UID: \"7b446c1d-56a2-4b12-923e-2ca1874321ed\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt" Nov 01 20:49:54.340946 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:54.340897 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b446c1d-56a2-4b12-923e-2ca1874321ed-istio-envoy\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt\" (UID: \"7b446c1d-56a2-4b12-923e-2ca1874321ed\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt" Nov 01 20:49:54.395434 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:54.395364 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b446c1d-56a2-4b12-923e-2ca1874321ed-istio-token\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt\" (UID: \"7b446c1d-56a2-4b12-923e-2ca1874321ed\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt" Nov 01 20:49:54.395574 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:54.395436 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjjg6\" (UniqueName: \"kubernetes.io/projected/7b446c1d-56a2-4b12-923e-2ca1874321ed-kube-api-access-qjjg6\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt\" (UID: \"7b446c1d-56a2-4b12-923e-2ca1874321ed\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt" Nov 01 20:49:54.424085 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:54.424058 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt" Nov 01 20:49:54.856223 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:49:54.856183 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b446c1d_56a2_4b12_923e_2ca1874321ed.slice/crio-cf17d18f551abff3aec52c8dcfe8ec476e3867ab3addb8f4aae0acc3878e2f9c WatchSource:0}: Error finding container cf17d18f551abff3aec52c8dcfe8ec476e3867ab3addb8f4aae0acc3878e2f9c: Status 404 returned error can't find the container with id cf17d18f551abff3aec52c8dcfe8ec476e3867ab3addb8f4aae0acc3878e2f9c Nov 01 20:49:54.856959 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:54.856936 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt"] Nov 01 20:49:54.858674 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:54.858644 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:49:54.858762 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:54.858706 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:49:54.858762 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:54.858738 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:49:55.085835 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:55.085802 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt" event={"ID":"7b446c1d-56a2-4b12-923e-2ca1874321ed","Type":"ContainerStarted","Data":"6d987a419cf3fe3d71bf6b2bdb40cfdae84adfabf76629186e44033cd121104c"} Nov 01 20:49:55.086028 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:55.085856 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt" event={"ID":"7b446c1d-56a2-4b12-923e-2ca1874321ed","Type":"ContainerStarted","Data":"cf17d18f551abff3aec52c8dcfe8ec476e3867ab3addb8f4aae0acc3878e2f9c"} Nov 01 20:49:55.206361 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:55.206248 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt" podStartSLOduration=1.2062338989999999 podStartE2EDuration="1.206233899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:49:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:49:55.202060811 +0000 UTC m=+1145.979700949" watchObservedRunningTime="2026-11-01 20:49:55.206233899 +0000 UTC m=+1145.983874098" Nov 01 20:49:55.424927 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:55.424888 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt" Nov 01 20:49:55.426302 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:55.426268 4484 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 01 20:49:55.426432 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:55.426327 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt" podUID="7b446c1d-56a2-4b12-923e-2ca1874321ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 01 20:49:55.549017 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:55.544733 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:49:56.234273 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:56.234224 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:49:56.428436 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:56.428390 4484 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 01 20:49:56.428625 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:56.428453 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt" podUID="7b446c1d-56a2-4b12-923e-2ca1874321ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 01 20:49:56.446185 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:56.446143 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-406eeae3-hzzpn"] Nov 01 20:49:56.450853 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:56.450830 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-406eeae3-hzzpn" Nov 01 20:49:56.532023 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:56.521757 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-406eeae3-hzzpn"] Nov 01 20:49:56.534231 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:56.534187 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:49:56.551340 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:56.551303 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:49:56.551340 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:56.551318 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:49:56.551617 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:56.551409 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 20:49:56.585113 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:56.585081 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d39c82f-ec44-4e41-a05b-649095064039-maas-api-xks-ca\") pod \"maas-api-revoke-sub-406eeae3-hzzpn\" (UID: \"4d39c82f-ec44-4e41-a05b-649095064039\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-406eeae3-hzzpn" Nov 01 20:49:56.585278 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:56.585216 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d39c82f-ec44-4e41-a05b-649095064039-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-406eeae3-hzzpn\" (UID: \"4d39c82f-ec44-4e41-a05b-649095064039\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-406eeae3-hzzpn" Nov 01 20:49:56.585278 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:56.585258 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz52k\" (UniqueName: \"kubernetes.io/projected/4d39c82f-ec44-4e41-a05b-649095064039-kube-api-access-nz52k\") pod \"maas-api-revoke-sub-406eeae3-hzzpn\" (UID: \"4d39c82f-ec44-4e41-a05b-649095064039\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-406eeae3-hzzpn" Nov 01 20:49:56.687009 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:56.686956 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d39c82f-ec44-4e41-a05b-649095064039-maas-api-xks-ca\") pod \"maas-api-revoke-sub-406eeae3-hzzpn\" (UID: \"4d39c82f-ec44-4e41-a05b-649095064039\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-406eeae3-hzzpn" Nov 01 20:49:56.687215 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:56.687074 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d39c82f-ec44-4e41-a05b-649095064039-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-406eeae3-hzzpn\" (UID: \"4d39c82f-ec44-4e41-a05b-649095064039\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-406eeae3-hzzpn" Nov 01 20:49:56.687215 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:56.687108 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz52k\" (UniqueName: \"kubernetes.io/projected/4d39c82f-ec44-4e41-a05b-649095064039-kube-api-access-nz52k\") pod \"maas-api-revoke-sub-406eeae3-hzzpn\" (UID: \"4d39c82f-ec44-4e41-a05b-649095064039\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-406eeae3-hzzpn" Nov 01 20:49:56.687749 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:56.687725 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d39c82f-ec44-4e41-a05b-649095064039-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-406eeae3-hzzpn\" (UID: \"4d39c82f-ec44-4e41-a05b-649095064039\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-406eeae3-hzzpn" Nov 01 20:49:56.689316 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:56.689292 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d39c82f-ec44-4e41-a05b-649095064039-maas-api-xks-ca\") pod \"maas-api-revoke-sub-406eeae3-hzzpn\" (UID: \"4d39c82f-ec44-4e41-a05b-649095064039\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-406eeae3-hzzpn" Nov 01 20:49:56.744691 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:56.744657 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz52k\" (UniqueName: \"kubernetes.io/projected/4d39c82f-ec44-4e41-a05b-649095064039-kube-api-access-nz52k\") pod \"maas-api-revoke-sub-406eeae3-hzzpn\" (UID: \"4d39c82f-ec44-4e41-a05b-649095064039\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-406eeae3-hzzpn" Nov 01 20:49:56.765598 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:56.765569 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-406eeae3-hzzpn" Nov 01 20:49:57.053300 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:57.053272 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b8846bc-cwg7c"] Nov 01 20:49:57.057501 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:57.057474 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b8846bc-cwg7c"] Nov 01 20:49:57.171817 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:49:57.171787 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d39c82f_ec44_4e41_a05b_649095064039.slice/crio-ae1559f0cf30ecdde951857c5a21888ab942e76e50f84bb47cabfef00dd54f48 WatchSource:0}: Error finding container ae1559f0cf30ecdde951857c5a21888ab942e76e50f84bb47cabfef00dd54f48: Status 404 returned error can't find the container with id ae1559f0cf30ecdde951857c5a21888ab942e76e50f84bb47cabfef00dd54f48 Nov 01 20:49:57.172951 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:57.172927 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-406eeae3-hzzpn"] Nov 01 20:49:57.373983 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:57.373948 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d3a7b07-765e-4eec-8948-ff6366bb6c9a" path="/var/lib/kubelet/pods/9d3a7b07-765e-4eec-8948-ff6366bb6c9a/volumes" Nov 01 20:49:57.425713 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:57.425151 4484 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 01 20:49:57.425713 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:57.425210 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt" podUID="7b446c1d-56a2-4b12-923e-2ca1874321ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 01 20:49:57.487219 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:57.487185 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:49:58.133792 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:58.133682 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-406eeae3-hzzpn" event={"ID":"4d39c82f-ec44-4e41-a05b-649095064039","Type":"ContainerStarted","Data":"2d07c40870b4d710599ff7fc3cd2a97fd130c0efca3c2f4a4e6298e7b99b9c53"} Nov 01 20:49:58.133792 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:58.133784 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-406eeae3-hzzpn" event={"ID":"4d39c82f-ec44-4e41-a05b-649095064039","Type":"ContainerStarted","Data":"ae1559f0cf30ecdde951857c5a21888ab942e76e50f84bb47cabfef00dd54f48"} Nov 01 20:49:58.251552 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:58.251496 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-406eeae3-hzzpn" podStartSLOduration=2.251482007 podStartE2EDuration="2.251482007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:49:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:49:58.248622357 +0000 UTC m=+1149.026262515" watchObservedRunningTime="2026-11-01 20:49:58.251482007 +0000 UTC m=+1149.029122148" Nov 01 20:49:58.428268 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:58.428172 4484 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 01 20:49:58.428268 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:58.428240 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt" podUID="7b446c1d-56a2-4b12-923e-2ca1874321ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 01 20:49:59.144227 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:59.144190 4484 generic.go:356] "Generic (PLEG): container finished" podID="4d39c82f-ec44-4e41-a05b-649095064039" containerID="2d07c40870b4d710599ff7fc3cd2a97fd130c0efca3c2f4a4e6298e7b99b9c53" exitCode=0 Nov 01 20:49:59.144412 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:59.144273 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-406eeae3-hzzpn" event={"ID":"4d39c82f-ec44-4e41-a05b-649095064039","Type":"ContainerDied","Data":"2d07c40870b4d710599ff7fc3cd2a97fd130c0efca3c2f4a4e6298e7b99b9c53"} Nov 01 20:49:59.425270 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:59.425181 4484 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 01 20:49:59.425528 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:49:59.425497 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt" podUID="7b446c1d-56a2-4b12-923e-2ca1874321ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 01 20:50:00.367448 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:00.367423 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-406eeae3-hzzpn" Nov 01 20:50:00.428468 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:00.428377 4484 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 01 20:50:00.428648 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:00.428481 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt" podUID="7b446c1d-56a2-4b12-923e-2ca1874321ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 01 20:50:00.469622 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:00.469593 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d39c82f-ec44-4e41-a05b-649095064039-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d39c82f-ec44-4e41-a05b-649095064039-maas-api-xks-ca\") pod \"4d39c82f-ec44-4e41-a05b-649095064039\" (UID: \"4d39c82f-ec44-4e41-a05b-649095064039\") " Nov 01 20:50:00.469808 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:00.469796 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d39c82f-ec44-4e41-a05b-649095064039-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d39c82f-ec44-4e41-a05b-649095064039-maas-api-openshift-ca\") pod \"4d39c82f-ec44-4e41-a05b-649095064039\" (UID: \"4d39c82f-ec44-4e41-a05b-649095064039\") " Nov 01 20:50:00.469920 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:00.469911 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d39c82f-ec44-4e41-a05b-649095064039-kube-api-access-nz52k\" (UniqueName: \"kubernetes.io/projected/4d39c82f-ec44-4e41-a05b-649095064039-kube-api-access-nz52k\") pod \"4d39c82f-ec44-4e41-a05b-649095064039\" (UID: \"4d39c82f-ec44-4e41-a05b-649095064039\") " Nov 01 20:50:00.470564 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:00.470521 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d39c82f-ec44-4e41-a05b-649095064039-maas-api-openshift-ca" pod "4d39c82f-ec44-4e41-a05b-649095064039" (UID: "4d39c82f-ec44-4e41-a05b-649095064039"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:50:00.471915 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:00.471874 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d39c82f-ec44-4e41-a05b-649095064039-maas-api-xks-ca" pod "4d39c82f-ec44-4e41-a05b-649095064039" (UID: "4d39c82f-ec44-4e41-a05b-649095064039"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:50:00.472971 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:00.472945 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d39c82f-ec44-4e41-a05b-649095064039-kube-api-access-nz52k" pod "4d39c82f-ec44-4e41-a05b-649095064039" (UID: "4d39c82f-ec44-4e41-a05b-649095064039"). InnerVolumeSpecName "kube-api-access-nz52k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:50:00.571350 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:00.571318 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d39c82f-ec44-4e41-a05b-649095064039-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:00.571350 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:00.571344 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d39c82f-ec44-4e41-a05b-649095064039-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:00.571350 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:00.571355 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz52k\" (UniqueName: \"kubernetes.io/projected/4d39c82f-ec44-4e41-a05b-649095064039-kube-api-access-nz52k\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:01.047204 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:01.047169 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06fecfbb-7wr8m"] Nov 01 20:50:01.053544 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:01.053510 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06fecfbb-7wr8m"] Nov 01 20:50:01.160564 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:01.160524 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-406eeae3-hzzpn" event={"ID":"4d39c82f-ec44-4e41-a05b-649095064039","Type":"ContainerDied","Data":"ae1559f0cf30ecdde951857c5a21888ab942e76e50f84bb47cabfef00dd54f48"} Nov 01 20:50:01.160751 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:01.160611 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae1559f0cf30ecdde951857c5a21888ab942e76e50f84bb47cabfef00dd54f48" Nov 01 20:50:01.160751 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:01.160695 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-406eeae3-hzzpn" Nov 01 20:50:01.372727 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:01.372639 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04242230-db36-48bc-b020-06b79d6ddab4" path="/var/lib/kubelet/pods/04242230-db36-48bc-b020-06b79d6ddab4/volumes" Nov 01 20:50:01.425440 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:01.425405 4484 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 01 20:50:01.425618 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:01.425460 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt" podUID="7b446c1d-56a2-4b12-923e-2ca1874321ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 01 20:50:02.352500 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:02.352431 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:50:02.850440 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:02.850387 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt" Nov 01 20:50:02.851378 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:02.851355 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt" Nov 01 20:50:02.852426 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:02.852401 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt" Nov 01 20:50:07.138282 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:07.138246 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:50:11.827200 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:11.827160 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a0cd91b-zvz2c"] Nov 01 20:50:11.828179 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:11.828157 4484 cpu_manager.go:414] "containerMap: removing container" podUID="4d39c82f-ec44-4e41-a05b-649095064039" containerName="revoke-keys" Nov 01 20:50:11.828270 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:11.828181 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d39c82f-ec44-4e41-a05b-649095064039" containerName="revoke-keys" Nov 01 20:50:11.828422 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:11.828407 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d39c82f-ec44-4e41-a05b-649095064039" containerName="revoke-keys" Nov 01 20:50:11.832772 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:11.832749 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0cd91b-zvz2c" Nov 01 20:50:11.904262 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:11.904215 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a0cd91b-zvz2c"] Nov 01 20:50:11.907034 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:11.906964 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:50:11.907197 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:11.907037 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:50:11.907197 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:11.907190 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:50:11.930124 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:11.930086 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 20:50:11.944480 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:11.944439 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:50:11.964006 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:11.963946 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d4f5a1e-364c-4282-ab9c-3b664313da65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a0cd91b-zvz2c\" (UID: \"8d4f5a1e-364c-4282-ab9c-3b664313da65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0cd91b-zvz2c" Nov 01 20:50:11.964357 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:11.964328 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d4f5a1e-364c-4282-ab9c-3b664313da65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a0cd91b-zvz2c\" (UID: \"8d4f5a1e-364c-4282-ab9c-3b664313da65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0cd91b-zvz2c" Nov 01 20:50:11.964689 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:11.964665 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kckz5\" (UniqueName: \"kubernetes.io/projected/8d4f5a1e-364c-4282-ab9c-3b664313da65-kube-api-access-kckz5\") pod \"maas-api-revoke-sub-0a0cd91b-zvz2c\" (UID: \"8d4f5a1e-364c-4282-ab9c-3b664313da65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0cd91b-zvz2c" Nov 01 20:50:12.066157 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:12.066117 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kckz5\" (UniqueName: \"kubernetes.io/projected/8d4f5a1e-364c-4282-ab9c-3b664313da65-kube-api-access-kckz5\") pod \"maas-api-revoke-sub-0a0cd91b-zvz2c\" (UID: \"8d4f5a1e-364c-4282-ab9c-3b664313da65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0cd91b-zvz2c" Nov 01 20:50:12.066358 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:12.066285 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d4f5a1e-364c-4282-ab9c-3b664313da65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a0cd91b-zvz2c\" (UID: \"8d4f5a1e-364c-4282-ab9c-3b664313da65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0cd91b-zvz2c" Nov 01 20:50:12.066358 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:12.066304 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d4f5a1e-364c-4282-ab9c-3b664313da65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a0cd91b-zvz2c\" (UID: \"8d4f5a1e-364c-4282-ab9c-3b664313da65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0cd91b-zvz2c" Nov 01 20:50:12.067036 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:12.067006 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d4f5a1e-364c-4282-ab9c-3b664313da65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a0cd91b-zvz2c\" (UID: \"8d4f5a1e-364c-4282-ab9c-3b664313da65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0cd91b-zvz2c" Nov 01 20:50:12.068850 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:12.068824 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d4f5a1e-364c-4282-ab9c-3b664313da65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a0cd91b-zvz2c\" (UID: \"8d4f5a1e-364c-4282-ab9c-3b664313da65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0cd91b-zvz2c" Nov 01 20:50:12.123856 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:12.123788 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kckz5\" (UniqueName: \"kubernetes.io/projected/8d4f5a1e-364c-4282-ab9c-3b664313da65-kube-api-access-kckz5\") pod \"maas-api-revoke-sub-0a0cd91b-zvz2c\" (UID: \"8d4f5a1e-364c-4282-ab9c-3b664313da65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0cd91b-zvz2c" Nov 01 20:50:12.147578 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:12.147539 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0cd91b-zvz2c" Nov 01 20:50:12.539190 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:12.539159 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a0cd91b-zvz2c"] Nov 01 20:50:12.540457 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:50:12.540427 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d4f5a1e_364c_4282_ab9c_3b664313da65.slice/crio-e71d9a29d2ad09bf34c6412df69a230f50b5c5b07795b5b70aba2cdd4a63fba3 WatchSource:0}: Error finding container e71d9a29d2ad09bf34c6412df69a230f50b5c5b07795b5b70aba2cdd4a63fba3: Status 404 returned error can't find the container with id e71d9a29d2ad09bf34c6412df69a230f50b5c5b07795b5b70aba2cdd4a63fba3 Nov 01 20:50:13.317897 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:13.317866 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0cd91b-zvz2c" event={"ID":"8d4f5a1e-364c-4282-ab9c-3b664313da65","Type":"ContainerStarted","Data":"14e3daeb011e99bda8862b4b26336c48301a0ab44e8266039e90cb6f34da8705"} Nov 01 20:50:13.318417 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:13.318086 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0cd91b-zvz2c" event={"ID":"8d4f5a1e-364c-4282-ab9c-3b664313da65","Type":"ContainerStarted","Data":"e71d9a29d2ad09bf34c6412df69a230f50b5c5b07795b5b70aba2cdd4a63fba3"} Nov 01 20:50:13.437385 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:13.437320 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0cd91b-zvz2c" podStartSLOduration=2.437299872 podStartE2EDuration="2.437299872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:50:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:50:13.435841739 +0000 UTC m=+1164.213481875" watchObservedRunningTime="2026-11-01 20:50:13.437299872 +0000 UTC m=+1164.214940008" Nov 01 20:50:14.092929 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:14.092885 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8f6bbf1-6ljkf"] Nov 01 20:50:14.118020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:14.108705 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8f6bbf1-6ljkf"] Nov 01 20:50:14.414078 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:14.413697 4484 generic.go:356] "Generic (PLEG): container finished" podID="8d4f5a1e-364c-4282-ab9c-3b664313da65" containerID="14e3daeb011e99bda8862b4b26336c48301a0ab44e8266039e90cb6f34da8705" exitCode=0 Nov 01 20:50:14.414078 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:14.413927 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0cd91b-zvz2c" event={"ID":"8d4f5a1e-364c-4282-ab9c-3b664313da65","Type":"ContainerDied","Data":"14e3daeb011e99bda8862b4b26336c48301a0ab44e8266039e90cb6f34da8705"} Nov 01 20:50:15.372613 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:15.372582 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="226c28bf-b17c-4eae-819a-03fb580dc870" path="/var/lib/kubelet/pods/226c28bf-b17c-4eae-819a-03fb580dc870/volumes" Nov 01 20:50:15.596669 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:15.596639 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0cd91b-zvz2c" Nov 01 20:50:15.646625 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:15.646531 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d4f5a1e-364c-4282-ab9c-3b664313da65-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d4f5a1e-364c-4282-ab9c-3b664313da65-maas-api-openshift-ca\") pod \"8d4f5a1e-364c-4282-ab9c-3b664313da65\" (UID: \"8d4f5a1e-364c-4282-ab9c-3b664313da65\") " Nov 01 20:50:15.646762 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:15.646753 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d4f5a1e-364c-4282-ab9c-3b664313da65-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d4f5a1e-364c-4282-ab9c-3b664313da65-maas-api-xks-ca\") pod \"8d4f5a1e-364c-4282-ab9c-3b664313da65\" (UID: \"8d4f5a1e-364c-4282-ab9c-3b664313da65\") " Nov 01 20:50:15.646801 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:15.646787 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d4f5a1e-364c-4282-ab9c-3b664313da65-kube-api-access-kckz5\" (UniqueName: \"kubernetes.io/projected/8d4f5a1e-364c-4282-ab9c-3b664313da65-kube-api-access-kckz5\") pod \"8d4f5a1e-364c-4282-ab9c-3b664313da65\" (UID: \"8d4f5a1e-364c-4282-ab9c-3b664313da65\") " Nov 01 20:50:15.646940 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:15.646914 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d4f5a1e-364c-4282-ab9c-3b664313da65-maas-api-openshift-ca" pod "8d4f5a1e-364c-4282-ab9c-3b664313da65" (UID: "8d4f5a1e-364c-4282-ab9c-3b664313da65"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:50:15.647319 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:15.647294 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d4f5a1e-364c-4282-ab9c-3b664313da65-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:15.648803 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:15.648785 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d4f5a1e-364c-4282-ab9c-3b664313da65-maas-api-xks-ca" pod "8d4f5a1e-364c-4282-ab9c-3b664313da65" (UID: "8d4f5a1e-364c-4282-ab9c-3b664313da65"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:50:15.648873 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:15.648851 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d4f5a1e-364c-4282-ab9c-3b664313da65-kube-api-access-kckz5" pod "8d4f5a1e-364c-4282-ab9c-3b664313da65" (UID: "8d4f5a1e-364c-4282-ab9c-3b664313da65"). InnerVolumeSpecName "kube-api-access-kckz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:50:15.748816 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:15.748766 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d4f5a1e-364c-4282-ab9c-3b664313da65-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:15.748816 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:15.748809 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kckz5\" (UniqueName: \"kubernetes.io/projected/8d4f5a1e-364c-4282-ab9c-3b664313da65-kube-api-access-kckz5\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:15.769595 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:50:15.769555 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 20:50:16.429082 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:16.429037 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0cd91b-zvz2c" event={"ID":"8d4f5a1e-364c-4282-ab9c-3b664313da65","Type":"ContainerDied","Data":"e71d9a29d2ad09bf34c6412df69a230f50b5c5b07795b5b70aba2cdd4a63fba3"} Nov 01 20:50:16.429241 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:16.429105 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e71d9a29d2ad09bf34c6412df69a230f50b5c5b07795b5b70aba2cdd4a63fba3" Nov 01 20:50:16.429241 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:16.429053 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0cd91b-zvz2c" Nov 01 20:50:16.844083 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:16.844052 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:50:16.965485 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:16.965449 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1d4200f-4b5mk"] Nov 01 20:50:16.966195 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:16.966175 4484 cpu_manager.go:414] "containerMap: removing container" podUID="8d4f5a1e-364c-4282-ab9c-3b664313da65" containerName="revoke-keys" Nov 01 20:50:16.966195 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:16.966195 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d4f5a1e-364c-4282-ab9c-3b664313da65" containerName="revoke-keys" Nov 01 20:50:16.966367 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:16.966360 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d4f5a1e-364c-4282-ab9c-3b664313da65" containerName="revoke-keys" Nov 01 20:50:16.970045 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:16.970026 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1d4200f-4b5mk" Nov 01 20:50:17.023130 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:17.023093 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:50:17.023130 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:17.023125 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 20:50:17.023377 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:17.023138 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:50:17.023377 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:17.023195 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:50:17.025269 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:17.025248 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1d4200f-4b5mk"] Nov 01 20:50:17.078641 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:17.078600 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9194558-fad7-4439-b531-278d6b4bea39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1d4200f-4b5mk\" (UID: \"b9194558-fad7-4439-b531-278d6b4bea39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1d4200f-4b5mk" Nov 01 20:50:17.078871 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:17.078842 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqh7n\" (UniqueName: \"kubernetes.io/projected/b9194558-fad7-4439-b531-278d6b4bea39-kube-api-access-sqh7n\") pod \"maas-api-revoke-sub-e1d4200f-4b5mk\" (UID: \"b9194558-fad7-4439-b531-278d6b4bea39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1d4200f-4b5mk" Nov 01 20:50:17.078965 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:17.078949 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9194558-fad7-4439-b531-278d6b4bea39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1d4200f-4b5mk\" (UID: \"b9194558-fad7-4439-b531-278d6b4bea39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1d4200f-4b5mk" Nov 01 20:50:17.180559 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:17.180462 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9194558-fad7-4439-b531-278d6b4bea39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1d4200f-4b5mk\" (UID: \"b9194558-fad7-4439-b531-278d6b4bea39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1d4200f-4b5mk" Nov 01 20:50:17.180756 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:17.180620 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqh7n\" (UniqueName: \"kubernetes.io/projected/b9194558-fad7-4439-b531-278d6b4bea39-kube-api-access-sqh7n\") pod \"maas-api-revoke-sub-e1d4200f-4b5mk\" (UID: \"b9194558-fad7-4439-b531-278d6b4bea39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1d4200f-4b5mk" Nov 01 20:50:17.180756 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:17.180678 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9194558-fad7-4439-b531-278d6b4bea39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1d4200f-4b5mk\" (UID: \"b9194558-fad7-4439-b531-278d6b4bea39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1d4200f-4b5mk" Nov 01 20:50:17.181410 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:17.181376 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9194558-fad7-4439-b531-278d6b4bea39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1d4200f-4b5mk\" (UID: \"b9194558-fad7-4439-b531-278d6b4bea39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1d4200f-4b5mk" Nov 01 20:50:17.183124 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:17.183098 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9194558-fad7-4439-b531-278d6b4bea39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1d4200f-4b5mk\" (UID: \"b9194558-fad7-4439-b531-278d6b4bea39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1d4200f-4b5mk" Nov 01 20:50:17.238392 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:17.238358 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqh7n\" (UniqueName: \"kubernetes.io/projected/b9194558-fad7-4439-b531-278d6b4bea39-kube-api-access-sqh7n\") pod \"maas-api-revoke-sub-e1d4200f-4b5mk\" (UID: \"b9194558-fad7-4439-b531-278d6b4bea39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1d4200f-4b5mk" Nov 01 20:50:17.283323 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:17.283283 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1d4200f-4b5mk" Nov 01 20:50:17.669352 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:50:17.669315 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9194558_fad7_4439_b531_278d6b4bea39.slice/crio-e34a3a4df3064e324e4aefc7f8fe75bc51fcd0e604ab54216b4ac1a063e77912 WatchSource:0}: Error finding container e34a3a4df3064e324e4aefc7f8fe75bc51fcd0e604ab54216b4ac1a063e77912: Status 404 returned error can't find the container with id e34a3a4df3064e324e4aefc7f8fe75bc51fcd0e604ab54216b4ac1a063e77912 Nov 01 20:50:17.670103 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:17.670072 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1d4200f-4b5mk"] Nov 01 20:50:18.449629 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:18.449538 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1d4200f-4b5mk" event={"ID":"b9194558-fad7-4439-b531-278d6b4bea39","Type":"ContainerStarted","Data":"cc82af2585ec718e50f15c8a0dede339daaa04577714d7ab0e3896c054fc2cfc"} Nov 01 20:50:18.449629 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:18.449593 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1d4200f-4b5mk" event={"ID":"b9194558-fad7-4439-b531-278d6b4bea39","Type":"ContainerStarted","Data":"e34a3a4df3064e324e4aefc7f8fe75bc51fcd0e604ab54216b4ac1a063e77912"} Nov 01 20:50:18.568539 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:18.568470 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1d4200f-4b5mk" podStartSLOduration=2.568450704 podStartE2EDuration="2.568450704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:50:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:50:18.564208266 +0000 UTC m=+1169.341848416" watchObservedRunningTime="2026-11-01 20:50:18.568450704 +0000 UTC m=+1169.346090841" Nov 01 20:50:18.836185 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:18.836153 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:50:19.458668 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:19.458633 4484 generic.go:356] "Generic (PLEG): container finished" podID="b9194558-fad7-4439-b531-278d6b4bea39" containerID="cc82af2585ec718e50f15c8a0dede339daaa04577714d7ab0e3896c054fc2cfc" exitCode=0 Nov 01 20:50:19.459147 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:19.458739 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1d4200f-4b5mk" event={"ID":"b9194558-fad7-4439-b531-278d6b4bea39","Type":"ContainerDied","Data":"cc82af2585ec718e50f15c8a0dede339daaa04577714d7ab0e3896c054fc2cfc"} Nov 01 20:50:20.629359 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:20.629332 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1d4200f-4b5mk" Nov 01 20:50:20.729784 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:20.729700 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9194558-fad7-4439-b531-278d6b4bea39-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9194558-fad7-4439-b531-278d6b4bea39-maas-api-openshift-ca\") pod \"b9194558-fad7-4439-b531-278d6b4bea39\" (UID: \"b9194558-fad7-4439-b531-278d6b4bea39\") " Nov 01 20:50:20.729784 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:20.729751 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9194558-fad7-4439-b531-278d6b4bea39-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9194558-fad7-4439-b531-278d6b4bea39-maas-api-xks-ca\") pod \"b9194558-fad7-4439-b531-278d6b4bea39\" (UID: \"b9194558-fad7-4439-b531-278d6b4bea39\") " Nov 01 20:50:20.730052 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:20.729982 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9194558-fad7-4439-b531-278d6b4bea39-kube-api-access-sqh7n\" (UniqueName: \"kubernetes.io/projected/b9194558-fad7-4439-b531-278d6b4bea39-kube-api-access-sqh7n\") pod \"b9194558-fad7-4439-b531-278d6b4bea39\" (UID: \"b9194558-fad7-4439-b531-278d6b4bea39\") " Nov 01 20:50:20.730144 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:20.730116 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9194558-fad7-4439-b531-278d6b4bea39-maas-api-openshift-ca" pod "b9194558-fad7-4439-b531-278d6b4bea39" (UID: "b9194558-fad7-4439-b531-278d6b4bea39"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:50:20.730533 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:20.730510 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9194558-fad7-4439-b531-278d6b4bea39-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:20.731940 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:20.731920 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9194558-fad7-4439-b531-278d6b4bea39-maas-api-xks-ca" pod "b9194558-fad7-4439-b531-278d6b4bea39" (UID: "b9194558-fad7-4439-b531-278d6b4bea39"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:50:20.732067 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:20.731934 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9194558-fad7-4439-b531-278d6b4bea39-kube-api-access-sqh7n" pod "b9194558-fad7-4439-b531-278d6b4bea39" (UID: "b9194558-fad7-4439-b531-278d6b4bea39"). InnerVolumeSpecName "kube-api-access-sqh7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:50:20.832030 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:20.831945 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqh7n\" (UniqueName: \"kubernetes.io/projected/b9194558-fad7-4439-b531-278d6b4bea39-kube-api-access-sqh7n\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:20.832030 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:20.831989 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9194558-fad7-4439-b531-278d6b4bea39-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:21.047937 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:21.047901 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a7a217d-w4jzg"] Nov 01 20:50:21.051896 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:21.051870 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a7a217d-w4jzg"] Nov 01 20:50:21.372868 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:21.372791 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dee25581-08e6-472d-b589-ccd44b94bd5f" path="/var/lib/kubelet/pods/dee25581-08e6-472d-b589-ccd44b94bd5f/volumes" Nov 01 20:50:21.478318 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:21.478290 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1d4200f-4b5mk" Nov 01 20:50:21.478466 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:21.478330 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1d4200f-4b5mk" event={"ID":"b9194558-fad7-4439-b531-278d6b4bea39","Type":"ContainerDied","Data":"e34a3a4df3064e324e4aefc7f8fe75bc51fcd0e604ab54216b4ac1a063e77912"} Nov 01 20:50:21.478466 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:21.478385 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e34a3a4df3064e324e4aefc7f8fe75bc51fcd0e604ab54216b4ac1a063e77912" Nov 01 20:50:21.630556 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:21.630447 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:50:22.426908 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:22.426874 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bff043fa-lzr84"] Nov 01 20:50:22.427588 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:22.427573 4484 cpu_manager.go:414] "containerMap: removing container" podUID="b9194558-fad7-4439-b531-278d6b4bea39" containerName="revoke-keys" Nov 01 20:50:22.427640 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:22.427590 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9194558-fad7-4439-b531-278d6b4bea39" containerName="revoke-keys" Nov 01 20:50:22.427706 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:22.427698 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9194558-fad7-4439-b531-278d6b4bea39" containerName="revoke-keys" Nov 01 20:50:22.431649 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:22.431623 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bff043fa-lzr84" Nov 01 20:50:22.487631 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:22.487597 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bff043fa-lzr84"] Nov 01 20:50:22.532561 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:22.532518 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:50:22.532561 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:22.532518 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:50:22.532561 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:22.532520 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:50:22.535173 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:22.535147 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 20:50:22.559956 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:22.559924 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pc2k\" (UniqueName: \"kubernetes.io/projected/3fc560bd-a1ae-4889-8ca9-2c03d2ef39b4-kube-api-access-2pc2k\") pod \"maas-api-revoke-sub-bff043fa-lzr84\" (UID: \"3fc560bd-a1ae-4889-8ca9-2c03d2ef39b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bff043fa-lzr84" Nov 01 20:50:22.560136 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:22.559976 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fc560bd-a1ae-4889-8ca9-2c03d2ef39b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bff043fa-lzr84\" (UID: \"3fc560bd-a1ae-4889-8ca9-2c03d2ef39b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bff043fa-lzr84" Nov 01 20:50:22.560405 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:22.560344 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fc560bd-a1ae-4889-8ca9-2c03d2ef39b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bff043fa-lzr84\" (UID: \"3fc560bd-a1ae-4889-8ca9-2c03d2ef39b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bff043fa-lzr84" Nov 01 20:50:22.662281 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:22.662247 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fc560bd-a1ae-4889-8ca9-2c03d2ef39b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bff043fa-lzr84\" (UID: \"3fc560bd-a1ae-4889-8ca9-2c03d2ef39b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bff043fa-lzr84" Nov 01 20:50:22.662793 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:22.662297 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pc2k\" (UniqueName: \"kubernetes.io/projected/3fc560bd-a1ae-4889-8ca9-2c03d2ef39b4-kube-api-access-2pc2k\") pod \"maas-api-revoke-sub-bff043fa-lzr84\" (UID: \"3fc560bd-a1ae-4889-8ca9-2c03d2ef39b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bff043fa-lzr84" Nov 01 20:50:22.662793 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:22.662335 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fc560bd-a1ae-4889-8ca9-2c03d2ef39b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bff043fa-lzr84\" (UID: \"3fc560bd-a1ae-4889-8ca9-2c03d2ef39b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bff043fa-lzr84" Nov 01 20:50:22.663226 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:22.663206 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fc560bd-a1ae-4889-8ca9-2c03d2ef39b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bff043fa-lzr84\" (UID: \"3fc560bd-a1ae-4889-8ca9-2c03d2ef39b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bff043fa-lzr84" Nov 01 20:50:22.665186 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:22.665158 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fc560bd-a1ae-4889-8ca9-2c03d2ef39b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bff043fa-lzr84\" (UID: \"3fc560bd-a1ae-4889-8ca9-2c03d2ef39b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bff043fa-lzr84" Nov 01 20:50:22.720025 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:22.719915 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pc2k\" (UniqueName: \"kubernetes.io/projected/3fc560bd-a1ae-4889-8ca9-2c03d2ef39b4-kube-api-access-2pc2k\") pod \"maas-api-revoke-sub-bff043fa-lzr84\" (UID: \"3fc560bd-a1ae-4889-8ca9-2c03d2ef39b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bff043fa-lzr84" Nov 01 20:50:22.745659 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:22.745623 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bff043fa-lzr84" Nov 01 20:50:23.125780 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:50:23.125743 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fc560bd_a1ae_4889_8ca9_2c03d2ef39b4.slice/crio-cc7412d9d458d2a778cfe593e381e419dc3df8ce3cfca7ea8bbb338e99f8ea0a WatchSource:0}: Error finding container cc7412d9d458d2a778cfe593e381e419dc3df8ce3cfca7ea8bbb338e99f8ea0a: Status 404 returned error can't find the container with id cc7412d9d458d2a778cfe593e381e419dc3df8ce3cfca7ea8bbb338e99f8ea0a Nov 01 20:50:23.126492 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:23.126468 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bff043fa-lzr84"] Nov 01 20:50:23.495537 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:23.495501 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bff043fa-lzr84" event={"ID":"3fc560bd-a1ae-4889-8ca9-2c03d2ef39b4","Type":"ContainerStarted","Data":"153f0ce574bb8ab8efda1f57031712ec0329d1c04da9284dc17e55a179500ca1"} Nov 01 20:50:23.495709 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:23.495555 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bff043fa-lzr84" event={"ID":"3fc560bd-a1ae-4889-8ca9-2c03d2ef39b4","Type":"ContainerStarted","Data":"cc7412d9d458d2a778cfe593e381e419dc3df8ce3cfca7ea8bbb338e99f8ea0a"} Nov 01 20:50:23.611633 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:23.611581 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bff043fa-lzr84" podStartSLOduration=1.611568213 podStartE2EDuration="1.611568213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:50:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:50:23.609375052 +0000 UTC m=+1174.387015187" watchObservedRunningTime="2026-11-01 20:50:23.611568213 +0000 UTC m=+1174.389208347" Nov 01 20:50:24.504850 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:24.504814 4484 generic.go:356] "Generic (PLEG): container finished" podID="3fc560bd-a1ae-4889-8ca9-2c03d2ef39b4" containerID="153f0ce574bb8ab8efda1f57031712ec0329d1c04da9284dc17e55a179500ca1" exitCode=0 Nov 01 20:50:24.505271 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:24.504914 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bff043fa-lzr84" event={"ID":"3fc560bd-a1ae-4889-8ca9-2c03d2ef39b4","Type":"ContainerDied","Data":"153f0ce574bb8ab8efda1f57031712ec0329d1c04da9284dc17e55a179500ca1"} Nov 01 20:50:25.047894 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:25.047861 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2e93c2f-nnclz"] Nov 01 20:50:25.052092 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:25.052065 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2e93c2f-nnclz"] Nov 01 20:50:25.372302 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:25.372225 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d87c9034-5ef9-4323-be7b-040921d4bf35" path="/var/lib/kubelet/pods/d87c9034-5ef9-4323-be7b-040921d4bf35/volumes" Nov 01 20:50:25.683434 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:25.683403 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc"] Nov 01 20:50:25.688950 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:25.688927 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc" Nov 01 20:50:25.690772 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:25.690746 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bff043fa-lzr84" Nov 01 20:50:25.700172 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:25.700148 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fc560bd-a1ae-4889-8ca9-2c03d2ef39b4-kube-api-access-2pc2k\" (UniqueName: \"kubernetes.io/projected/3fc560bd-a1ae-4889-8ca9-2c03d2ef39b4-kube-api-access-2pc2k\") pod \"3fc560bd-a1ae-4889-8ca9-2c03d2ef39b4\" (UID: \"3fc560bd-a1ae-4889-8ca9-2c03d2ef39b4\") " Nov 01 20:50:25.700300 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:25.700273 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fc560bd-a1ae-4889-8ca9-2c03d2ef39b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fc560bd-a1ae-4889-8ca9-2c03d2ef39b4-maas-api-xks-ca\") pod \"3fc560bd-a1ae-4889-8ca9-2c03d2ef39b4\" (UID: \"3fc560bd-a1ae-4889-8ca9-2c03d2ef39b4\") " Nov 01 20:50:25.700425 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:25.700406 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fc560bd-a1ae-4889-8ca9-2c03d2ef39b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fc560bd-a1ae-4889-8ca9-2c03d2ef39b4-maas-api-openshift-ca\") pod \"3fc560bd-a1ae-4889-8ca9-2c03d2ef39b4\" (UID: \"3fc560bd-a1ae-4889-8ca9-2c03d2ef39b4\") " Nov 01 20:50:25.700683 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:25.700658 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5142a1b6-ef5b-4381-ae7d-676a82db925a-istio-podinfo\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc\" (UID: \"5142a1b6-ef5b-4381-ae7d-676a82db925a\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc" Nov 01 20:50:25.701867 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:25.700742 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5142a1b6-ef5b-4381-ae7d-676a82db925a-credential-socket\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc\" (UID: \"5142a1b6-ef5b-4381-ae7d-676a82db925a\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc" Nov 01 20:50:25.701867 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:25.700859 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fc560bd-a1ae-4889-8ca9-2c03d2ef39b4-maas-api-openshift-ca" pod "3fc560bd-a1ae-4889-8ca9-2c03d2ef39b4" (UID: "3fc560bd-a1ae-4889-8ca9-2c03d2ef39b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:50:25.701867 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:25.700926 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5142a1b6-ef5b-4381-ae7d-676a82db925a-istio-envoy\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc\" (UID: \"5142a1b6-ef5b-4381-ae7d-676a82db925a\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc" Nov 01 20:50:25.701867 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:25.701041 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5142a1b6-ef5b-4381-ae7d-676a82db925a-workload-certs\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc\" (UID: \"5142a1b6-ef5b-4381-ae7d-676a82db925a\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc" Nov 01 20:50:25.701867 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:25.701096 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5142a1b6-ef5b-4381-ae7d-676a82db925a-istio-data\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc\" (UID: \"5142a1b6-ef5b-4381-ae7d-676a82db925a\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc" Nov 01 20:50:25.701867 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:25.701128 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5142a1b6-ef5b-4381-ae7d-676a82db925a-istiod-ca-cert\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc\" (UID: \"5142a1b6-ef5b-4381-ae7d-676a82db925a\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc" Nov 01 20:50:25.701867 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:25.701156 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5142a1b6-ef5b-4381-ae7d-676a82db925a-istio-token\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc\" (UID: \"5142a1b6-ef5b-4381-ae7d-676a82db925a\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc" Nov 01 20:50:25.701867 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:25.701243 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5142a1b6-ef5b-4381-ae7d-676a82db925a-workload-socket\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc\" (UID: \"5142a1b6-ef5b-4381-ae7d-676a82db925a\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc" Nov 01 20:50:25.701867 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:25.701303 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwq9r\" (UniqueName: \"kubernetes.io/projected/5142a1b6-ef5b-4381-ae7d-676a82db925a-kube-api-access-xwq9r\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc\" (UID: \"5142a1b6-ef5b-4381-ae7d-676a82db925a\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc" Nov 01 20:50:25.701867 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:25.701409 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fc560bd-a1ae-4889-8ca9-2c03d2ef39b4-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:25.703081 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:25.702758 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fc560bd-a1ae-4889-8ca9-2c03d2ef39b4-maas-api-xks-ca" pod "3fc560bd-a1ae-4889-8ca9-2c03d2ef39b4" (UID: "3fc560bd-a1ae-4889-8ca9-2c03d2ef39b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:50:25.703081 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:25.702861 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fc560bd-a1ae-4889-8ca9-2c03d2ef39b4-kube-api-access-2pc2k" pod "3fc560bd-a1ae-4889-8ca9-2c03d2ef39b4" (UID: "3fc560bd-a1ae-4889-8ca9-2c03d2ef39b4"). InnerVolumeSpecName "kube-api-access-2pc2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:50:25.786244 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:25.786213 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc"] Nov 01 20:50:25.803174 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:25.803145 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5142a1b6-ef5b-4381-ae7d-676a82db925a-istio-data\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc\" (UID: \"5142a1b6-ef5b-4381-ae7d-676a82db925a\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc" Nov 01 20:50:25.803355 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:25.803187 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5142a1b6-ef5b-4381-ae7d-676a82db925a-istiod-ca-cert\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc\" (UID: \"5142a1b6-ef5b-4381-ae7d-676a82db925a\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc" Nov 01 20:50:25.803355 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:25.803255 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5142a1b6-ef5b-4381-ae7d-676a82db925a-istio-token\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc\" (UID: \"5142a1b6-ef5b-4381-ae7d-676a82db925a\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc" Nov 01 20:50:25.803476 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:25.803353 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5142a1b6-ef5b-4381-ae7d-676a82db925a-workload-socket\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc\" (UID: \"5142a1b6-ef5b-4381-ae7d-676a82db925a\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc" Nov 01 20:50:25.803476 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:25.803410 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwq9r\" (UniqueName: \"kubernetes.io/projected/5142a1b6-ef5b-4381-ae7d-676a82db925a-kube-api-access-xwq9r\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc\" (UID: \"5142a1b6-ef5b-4381-ae7d-676a82db925a\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc" Nov 01 20:50:25.803476 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:25.803464 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5142a1b6-ef5b-4381-ae7d-676a82db925a-istio-podinfo\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc\" (UID: \"5142a1b6-ef5b-4381-ae7d-676a82db925a\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc" Nov 01 20:50:25.803614 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:25.803536 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5142a1b6-ef5b-4381-ae7d-676a82db925a-credential-socket\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc\" (UID: \"5142a1b6-ef5b-4381-ae7d-676a82db925a\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc" Nov 01 20:50:25.803671 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:25.803637 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5142a1b6-ef5b-4381-ae7d-676a82db925a-istio-envoy\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc\" (UID: \"5142a1b6-ef5b-4381-ae7d-676a82db925a\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc" Nov 01 20:50:25.803801 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:25.803714 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5142a1b6-ef5b-4381-ae7d-676a82db925a-workload-certs\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc\" (UID: \"5142a1b6-ef5b-4381-ae7d-676a82db925a\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc" Nov 01 20:50:25.803872 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:25.803808 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fc560bd-a1ae-4889-8ca9-2c03d2ef39b4-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:25.803872 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:25.803827 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pc2k\" (UniqueName: \"kubernetes.io/projected/3fc560bd-a1ae-4889-8ca9-2c03d2ef39b4-kube-api-access-2pc2k\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:25.805158 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:25.804461 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5142a1b6-ef5b-4381-ae7d-676a82db925a-istio-data\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc\" (UID: \"5142a1b6-ef5b-4381-ae7d-676a82db925a\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc" Nov 01 20:50:25.805158 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:25.804691 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5142a1b6-ef5b-4381-ae7d-676a82db925a-workload-socket\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc\" (UID: \"5142a1b6-ef5b-4381-ae7d-676a82db925a\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc" Nov 01 20:50:25.805158 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:25.804894 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5142a1b6-ef5b-4381-ae7d-676a82db925a-credential-socket\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc\" (UID: \"5142a1b6-ef5b-4381-ae7d-676a82db925a\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc" Nov 01 20:50:25.805158 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:25.805112 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5142a1b6-ef5b-4381-ae7d-676a82db925a-workload-certs\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc\" (UID: \"5142a1b6-ef5b-4381-ae7d-676a82db925a\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc" Nov 01 20:50:25.805458 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:25.805114 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5142a1b6-ef5b-4381-ae7d-676a82db925a-istiod-ca-cert\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc\" (UID: \"5142a1b6-ef5b-4381-ae7d-676a82db925a\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc" Nov 01 20:50:25.807013 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:25.806973 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5142a1b6-ef5b-4381-ae7d-676a82db925a-istio-podinfo\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc\" (UID: \"5142a1b6-ef5b-4381-ae7d-676a82db925a\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc" Nov 01 20:50:25.807135 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:25.807117 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5142a1b6-ef5b-4381-ae7d-676a82db925a-istio-envoy\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc\" (UID: \"5142a1b6-ef5b-4381-ae7d-676a82db925a\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc" Nov 01 20:50:25.860978 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:25.860945 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5142a1b6-ef5b-4381-ae7d-676a82db925a-istio-token\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc\" (UID: \"5142a1b6-ef5b-4381-ae7d-676a82db925a\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc" Nov 01 20:50:25.861160 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:25.861127 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwq9r\" (UniqueName: \"kubernetes.io/projected/5142a1b6-ef5b-4381-ae7d-676a82db925a-kube-api-access-xwq9r\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc\" (UID: \"5142a1b6-ef5b-4381-ae7d-676a82db925a\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc" Nov 01 20:50:26.004243 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:26.004210 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc" Nov 01 20:50:26.438531 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:50:26.438500 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5142a1b6_ef5b_4381_ae7d_676a82db925a.slice/crio-08785a5a6c5b39ac7abde82fe8ab3666d73093629911585be1f7ee5f2f06ae40 WatchSource:0}: Error finding container 08785a5a6c5b39ac7abde82fe8ab3666d73093629911585be1f7ee5f2f06ae40: Status 404 returned error can't find the container with id 08785a5a6c5b39ac7abde82fe8ab3666d73093629911585be1f7ee5f2f06ae40 Nov 01 20:50:26.439989 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:26.439961 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc"] Nov 01 20:50:26.441630 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:26.441595 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:50:26.441701 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:26.441662 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:50:26.441738 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:26.441698 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:50:26.463483 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:26.463454 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:50:26.586108 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:26.586075 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc" event={"ID":"5142a1b6-ef5b-4381-ae7d-676a82db925a","Type":"ContainerStarted","Data":"08785a5a6c5b39ac7abde82fe8ab3666d73093629911585be1f7ee5f2f06ae40"} Nov 01 20:50:26.588016 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:26.587622 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bff043fa-lzr84" event={"ID":"3fc560bd-a1ae-4889-8ca9-2c03d2ef39b4","Type":"ContainerDied","Data":"cc7412d9d458d2a778cfe593e381e419dc3df8ce3cfca7ea8bbb338e99f8ea0a"} Nov 01 20:50:26.588016 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:26.587663 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc7412d9d458d2a778cfe593e381e419dc3df8ce3cfca7ea8bbb338e99f8ea0a" Nov 01 20:50:26.588016 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:26.587673 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bff043fa-lzr84" Nov 01 20:50:27.596939 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:27.596901 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc" event={"ID":"5142a1b6-ef5b-4381-ae7d-676a82db925a","Type":"ContainerStarted","Data":"1d1c0c13cacb2c3f418350d97268ea4fb0feebe629d99eca3f73bcb3af17ff7d"} Nov 01 20:50:27.825022 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:27.820678 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc" podStartSLOduration=2.82065664 podStartE2EDuration="2.82065664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:50:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:50:27.765345892 +0000 UTC m=+1178.542986039" watchObservedRunningTime="2026-11-01 20:50:27.82065664 +0000 UTC m=+1178.598296776" Nov 01 20:50:28.026020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:28.015368 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc" Nov 01 20:50:28.028168 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:28.028129 4484 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 01 20:50:28.028328 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:28.028199 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc" podUID="5142a1b6-ef5b-4381-ae7d-676a82db925a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 01 20:50:29.023021 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:29.017263 4484 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 01 20:50:29.023021 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:29.017320 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc" podUID="5142a1b6-ef5b-4381-ae7d-676a82db925a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 01 20:50:30.020461 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:30.020418 4484 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 01 20:50:30.028123 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:30.028079 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc" podUID="5142a1b6-ef5b-4381-ae7d-676a82db925a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 01 20:50:31.024515 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:31.024476 4484 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 01 20:50:31.024849 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:31.024820 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc" podUID="5142a1b6-ef5b-4381-ae7d-676a82db925a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 01 20:50:31.348725 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:31.348622 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:50:32.019172 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:32.019132 4484 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 01 20:50:32.019455 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:32.019426 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc" podUID="5142a1b6-ef5b-4381-ae7d-676a82db925a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 01 20:50:32.053891 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:32.053852 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0902bc45-ptppj"] Nov 01 20:50:32.130022 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:32.128490 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0902bc45-ptppj"] Nov 01 20:50:33.005211 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:33.005179 4484 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 01 20:50:33.005608 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:33.005237 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc" podUID="5142a1b6-ef5b-4381-ae7d-676a82db925a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 01 20:50:33.372488 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:33.372405 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5780b494-dc4c-47a7-b2df-52af36460639" path="/var/lib/kubelet/pods/5780b494-dc4c-47a7-b2df-52af36460639/volumes" Nov 01 20:50:34.096695 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:34.096659 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc" Nov 01 20:50:34.097598 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:34.097569 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc" Nov 01 20:50:34.098580 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:34.098532 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc" Nov 01 20:50:34.507098 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:34.507057 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9d537cc-9vsb4"] Nov 01 20:50:34.507736 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:34.507722 4484 cpu_manager.go:414] "containerMap: removing container" podUID="3fc560bd-a1ae-4889-8ca9-2c03d2ef39b4" containerName="revoke-keys" Nov 01 20:50:34.507789 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:34.507737 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fc560bd-a1ae-4889-8ca9-2c03d2ef39b4" containerName="revoke-keys" Nov 01 20:50:34.507871 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:34.507862 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fc560bd-a1ae-4889-8ca9-2c03d2ef39b4" containerName="revoke-keys" Nov 01 20:50:34.512476 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:34.512452 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9d537cc-9vsb4" Nov 01 20:50:34.569007 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:34.568955 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:50:34.569163 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:34.569049 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:50:34.572961 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:34.572934 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:50:34.573174 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:34.573156 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 20:50:34.583891 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:34.583856 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9d537cc-9vsb4"] Nov 01 20:50:34.671778 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:34.671743 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhzk8\" (UniqueName: \"kubernetes.io/projected/039b7fe4-4e78-41a5-aa66-61e5999c85bb-kube-api-access-bhzk8\") pod \"maas-api-revoke-sub-c9d537cc-9vsb4\" (UID: \"039b7fe4-4e78-41a5-aa66-61e5999c85bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9d537cc-9vsb4" Nov 01 20:50:34.671962 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:34.671909 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/039b7fe4-4e78-41a5-aa66-61e5999c85bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9d537cc-9vsb4\" (UID: \"039b7fe4-4e78-41a5-aa66-61e5999c85bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9d537cc-9vsb4" Nov 01 20:50:34.672110 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:34.672091 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/039b7fe4-4e78-41a5-aa66-61e5999c85bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9d537cc-9vsb4\" (UID: \"039b7fe4-4e78-41a5-aa66-61e5999c85bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9d537cc-9vsb4" Nov 01 20:50:34.773337 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:34.773222 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/039b7fe4-4e78-41a5-aa66-61e5999c85bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9d537cc-9vsb4\" (UID: \"039b7fe4-4e78-41a5-aa66-61e5999c85bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9d537cc-9vsb4" Nov 01 20:50:34.773529 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:34.773338 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhzk8\" (UniqueName: \"kubernetes.io/projected/039b7fe4-4e78-41a5-aa66-61e5999c85bb-kube-api-access-bhzk8\") pod \"maas-api-revoke-sub-c9d537cc-9vsb4\" (UID: \"039b7fe4-4e78-41a5-aa66-61e5999c85bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9d537cc-9vsb4" Nov 01 20:50:34.773529 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:34.773448 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/039b7fe4-4e78-41a5-aa66-61e5999c85bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9d537cc-9vsb4\" (UID: \"039b7fe4-4e78-41a5-aa66-61e5999c85bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9d537cc-9vsb4" Nov 01 20:50:34.774032 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:34.773987 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/039b7fe4-4e78-41a5-aa66-61e5999c85bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9d537cc-9vsb4\" (UID: \"039b7fe4-4e78-41a5-aa66-61e5999c85bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9d537cc-9vsb4" Nov 01 20:50:34.775871 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:34.775848 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/039b7fe4-4e78-41a5-aa66-61e5999c85bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9d537cc-9vsb4\" (UID: \"039b7fe4-4e78-41a5-aa66-61e5999c85bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9d537cc-9vsb4" Nov 01 20:50:34.831056 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:34.830984 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhzk8\" (UniqueName: \"kubernetes.io/projected/039b7fe4-4e78-41a5-aa66-61e5999c85bb-kube-api-access-bhzk8\") pod \"maas-api-revoke-sub-c9d537cc-9vsb4\" (UID: \"039b7fe4-4e78-41a5-aa66-61e5999c85bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9d537cc-9vsb4" Nov 01 20:50:35.127976 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:35.127883 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9d537cc-9vsb4" Nov 01 20:50:36.038021 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:36.037830 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9d537cc-9vsb4"] Nov 01 20:50:36.201040 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:50:36.187148 4484 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod039b7fe4_4e78_41a5_aa66_61e5999c85bb.slice/crio-f61de8e035b3b5e935ebf508391f5cb4ace122ef510731b7d650e9f559eead59\": RecentStats: unable to find data in memory cache]" Nov 01 20:50:36.201040 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:50:36.199556 4484 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod039b7fe4_4e78_41a5_aa66_61e5999c85bb.slice/crio-f61de8e035b3b5e935ebf508391f5cb4ace122ef510731b7d650e9f559eead59\": RecentStats: unable to find data in memory cache]" Nov 01 20:50:37.026023 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:36.996967 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:50:37.116154 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:37.116100 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" podUID="17a9ac24-19cb-48c4-99b5-d74ce8aac0a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:50:37.216948 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:37.216856 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" podUID="70b4dacf-b3ac-46ec-99e8-a44f05587e7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:50:37.258845 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:37.257212 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" podUID="a65d9fd8-6193-4d03-9a3c-123f48b5dcdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:50:37.261156 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:37.259897 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" podUID="fe8307db-264d-4b8a-b097-f180bc255a9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:50:37.261481 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:37.261456 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" podUID="9f19be29-2c84-4ae4-9674-acca4905f4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:50:37.353378 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:37.353082 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" podUID="e175b298-2c23-45d4-8f9c-0c9b9f6aab92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:50:37.424137 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:37.423172 4484 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" podUID="311f9699-6585-4a2c-8c20-e6b696b2f857" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:50:37.461075 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:37.461034 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9d537cc-9vsb4" event={"ID":"039b7fe4-4e78-41a5-aa66-61e5999c85bb","Type":"ContainerStarted","Data":"cafe933880332ee2ab7d9ff0d1687c4171d93ef8d92e4920fb2f7a62cedaaf75"} Nov 01 20:50:37.462174 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:37.461286 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9d537cc-9vsb4" event={"ID":"039b7fe4-4e78-41a5-aa66-61e5999c85bb","Type":"ContainerStarted","Data":"f61de8e035b3b5e935ebf508391f5cb4ace122ef510731b7d650e9f559eead59"} Nov 01 20:50:37.586658 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:37.586583 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9d537cc-9vsb4" podStartSLOduration=3.586561957 podStartE2EDuration="3.586561957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:50:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:50:37.578820401 +0000 UTC m=+1188.356460537" watchObservedRunningTime="2026-11-01 20:50:37.586561957 +0000 UTC m=+1188.364202093" Nov 01 20:50:37.890748 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:37.889107 4484 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" podUID="e175b298-2c23-45d4-8f9c-0c9b9f6aab92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 20:50:38.133333 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:38.126325 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" podUID="17a9ac24-19cb-48c4-99b5-d74ce8aac0a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 01 20:50:38.133333 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:38.126984 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" Nov 01 20:50:38.272023 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:38.241513 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" podUID="70b4dacf-b3ac-46ec-99e8-a44f05587e7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 01 20:50:38.294178 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:38.273595 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" podUID="9f19be29-2c84-4ae4-9674-acca4905f4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:50:38.316038 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:38.273646 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" podUID="311f9699-6585-4a2c-8c20-e6b696b2f857" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 01 20:50:38.319412 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:38.273671 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" podUID="fe8307db-264d-4b8a-b097-f180bc255a9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 01 20:50:38.329493 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:38.278427 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" podUID="a65d9fd8-6193-4d03-9a3c-123f48b5dcdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:50:38.355730 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:38.278457 4484 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" podUID="17a9ac24-19cb-48c4-99b5-d74ce8aac0a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": context deadline exceeded" Nov 01 20:50:38.356029 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:38.309616 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" Nov 01 20:50:38.356226 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:38.356209 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" Nov 01 20:50:38.356369 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:38.356357 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" Nov 01 20:50:38.356498 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:38.356486 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" Nov 01 20:50:38.467113 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:38.415337 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" podUID="e175b298-2c23-45d4-8f9c-0c9b9f6aab92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:50:38.561506 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:38.561475 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" Nov 01 20:50:39.154020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:39.133825 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" podUID="17a9ac24-19cb-48c4-99b5-d74ce8aac0a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 01 20:50:39.178578 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:39.178541 4484 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:50:39.178866 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:39.178838 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv" podUID="3ae0b2c8-270b-4a3f-bb16-523874681ade" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:50:39.325026 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:39.311543 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" podUID="70b4dacf-b3ac-46ec-99e8-a44f05587e7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 01 20:50:39.325026 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:39.311389 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" podUID="9f19be29-2c84-4ae4-9674-acca4905f4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 01 20:50:39.326858 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:39.326817 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" podUID="311f9699-6585-4a2c-8c20-e6b696b2f857" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 01 20:50:39.348858 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:39.341114 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" Nov 01 20:50:39.348858 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:39.342362 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" podUID="fe8307db-264d-4b8a-b097-f180bc255a9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:50:39.375020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:39.356186 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" podUID="a65d9fd8-6193-4d03-9a3c-123f48b5dcdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:50:39.599111 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:39.599056 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" podUID="e175b298-2c23-45d4-8f9c-0c9b9f6aab92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:50:39.873083 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:39.869657 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:50:39.968013 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:39.967816 4484 generic.go:356] "Generic (PLEG): container finished" podID="039b7fe4-4e78-41a5-aa66-61e5999c85bb" containerID="cafe933880332ee2ab7d9ff0d1687c4171d93ef8d92e4920fb2f7a62cedaaf75" exitCode=0 Nov 01 20:50:40.013075 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:40.012776 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9d537cc-9vsb4" event={"ID":"039b7fe4-4e78-41a5-aa66-61e5999c85bb","Type":"ContainerDied","Data":"cafe933880332ee2ab7d9ff0d1687c4171d93ef8d92e4920fb2f7a62cedaaf75"} Nov 01 20:50:40.056169 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:40.056135 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" Nov 01 20:50:40.060408 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:40.060380 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" Nov 01 20:50:40.060665 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:40.060649 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" Nov 01 20:50:40.060797 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:40.060785 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" Nov 01 20:50:40.106417 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:40.106386 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" Nov 01 20:50:40.110587 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:40.110563 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" Nov 01 20:50:40.152617 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:40.152535 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" Nov 01 20:50:41.201438 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:41.201412 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9d537cc-9vsb4" Nov 01 20:50:41.327288 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:41.327207 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/039b7fe4-4e78-41a5-aa66-61e5999c85bb-kube-api-access-bhzk8\" (UniqueName: \"kubernetes.io/projected/039b7fe4-4e78-41a5-aa66-61e5999c85bb-kube-api-access-bhzk8\") pod \"039b7fe4-4e78-41a5-aa66-61e5999c85bb\" (UID: \"039b7fe4-4e78-41a5-aa66-61e5999c85bb\") " Nov 01 20:50:41.327698 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:41.327672 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/039b7fe4-4e78-41a5-aa66-61e5999c85bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/039b7fe4-4e78-41a5-aa66-61e5999c85bb-maas-api-xks-ca\") pod \"039b7fe4-4e78-41a5-aa66-61e5999c85bb\" (UID: \"039b7fe4-4e78-41a5-aa66-61e5999c85bb\") " Nov 01 20:50:41.328235 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:41.328212 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/039b7fe4-4e78-41a5-aa66-61e5999c85bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/039b7fe4-4e78-41a5-aa66-61e5999c85bb-maas-api-openshift-ca\") pod \"039b7fe4-4e78-41a5-aa66-61e5999c85bb\" (UID: \"039b7fe4-4e78-41a5-aa66-61e5999c85bb\") " Nov 01 20:50:41.330187 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:41.330154 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/039b7fe4-4e78-41a5-aa66-61e5999c85bb-kube-api-access-bhzk8" pod "039b7fe4-4e78-41a5-aa66-61e5999c85bb" (UID: "039b7fe4-4e78-41a5-aa66-61e5999c85bb"). InnerVolumeSpecName "kube-api-access-bhzk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:50:41.338170 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:41.338134 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/039b7fe4-4e78-41a5-aa66-61e5999c85bb-maas-api-openshift-ca" pod "039b7fe4-4e78-41a5-aa66-61e5999c85bb" (UID: "039b7fe4-4e78-41a5-aa66-61e5999c85bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:50:41.339029 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:41.338989 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/039b7fe4-4e78-41a5-aa66-61e5999c85bb-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:41.339123 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:41.339030 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhzk8\" (UniqueName: \"kubernetes.io/projected/039b7fe4-4e78-41a5-aa66-61e5999c85bb-kube-api-access-bhzk8\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:41.341620 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:41.341587 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/039b7fe4-4e78-41a5-aa66-61e5999c85bb-maas-api-xks-ca" pod "039b7fe4-4e78-41a5-aa66-61e5999c85bb" (UID: "039b7fe4-4e78-41a5-aa66-61e5999c85bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:50:41.440431 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:41.440401 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/039b7fe4-4e78-41a5-aa66-61e5999c85bb-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:41.607298 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:41.607204 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78648dee-bptz9"] Nov 01 20:50:41.607950 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:41.607925 4484 cpu_manager.go:414] "containerMap: removing container" podUID="039b7fe4-4e78-41a5-aa66-61e5999c85bb" containerName="revoke-keys" Nov 01 20:50:41.607950 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:41.607946 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="039b7fe4-4e78-41a5-aa66-61e5999c85bb" containerName="revoke-keys" Nov 01 20:50:41.608147 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:41.608116 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="039b7fe4-4e78-41a5-aa66-61e5999c85bb" containerName="revoke-keys" Nov 01 20:50:41.612176 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:41.612152 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78648dee-bptz9" Nov 01 20:50:41.660443 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:41.660404 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78648dee-bptz9"] Nov 01 20:50:41.745262 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:41.745228 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/100bce4a-f707-4097-8902-50d8d064c4fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78648dee-bptz9\" (UID: \"100bce4a-f707-4097-8902-50d8d064c4fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78648dee-bptz9" Nov 01 20:50:41.745417 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:41.745396 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gljqm\" (UniqueName: \"kubernetes.io/projected/100bce4a-f707-4097-8902-50d8d064c4fd-kube-api-access-gljqm\") pod \"maas-api-revoke-sub-78648dee-bptz9\" (UID: \"100bce4a-f707-4097-8902-50d8d064c4fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78648dee-bptz9" Nov 01 20:50:41.745510 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:41.745483 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/100bce4a-f707-4097-8902-50d8d064c4fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78648dee-bptz9\" (UID: \"100bce4a-f707-4097-8902-50d8d064c4fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78648dee-bptz9" Nov 01 20:50:41.847178 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:41.847139 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/100bce4a-f707-4097-8902-50d8d064c4fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78648dee-bptz9\" (UID: \"100bce4a-f707-4097-8902-50d8d064c4fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78648dee-bptz9" Nov 01 20:50:41.847336 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:41.847293 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gljqm\" (UniqueName: \"kubernetes.io/projected/100bce4a-f707-4097-8902-50d8d064c4fd-kube-api-access-gljqm\") pod \"maas-api-revoke-sub-78648dee-bptz9\" (UID: \"100bce4a-f707-4097-8902-50d8d064c4fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78648dee-bptz9" Nov 01 20:50:41.847336 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:41.847331 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/100bce4a-f707-4097-8902-50d8d064c4fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78648dee-bptz9\" (UID: \"100bce4a-f707-4097-8902-50d8d064c4fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78648dee-bptz9" Nov 01 20:50:41.847989 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:41.847970 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/100bce4a-f707-4097-8902-50d8d064c4fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78648dee-bptz9\" (UID: \"100bce4a-f707-4097-8902-50d8d064c4fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78648dee-bptz9" Nov 01 20:50:41.849602 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:41.849577 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/100bce4a-f707-4097-8902-50d8d064c4fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78648dee-bptz9\" (UID: \"100bce4a-f707-4097-8902-50d8d064c4fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78648dee-bptz9" Nov 01 20:50:41.912175 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:41.912099 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gljqm\" (UniqueName: \"kubernetes.io/projected/100bce4a-f707-4097-8902-50d8d064c4fd-kube-api-access-gljqm\") pod \"maas-api-revoke-sub-78648dee-bptz9\" (UID: \"100bce4a-f707-4097-8902-50d8d064c4fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78648dee-bptz9" Nov 01 20:50:41.927976 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:41.927942 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78648dee-bptz9" Nov 01 20:50:42.060427 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:42.060069 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9d537cc-9vsb4" event={"ID":"039b7fe4-4e78-41a5-aa66-61e5999c85bb","Type":"ContainerDied","Data":"f61de8e035b3b5e935ebf508391f5cb4ace122ef510731b7d650e9f559eead59"} Nov 01 20:50:42.060427 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:42.060152 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f61de8e035b3b5e935ebf508391f5cb4ace122ef510731b7d650e9f559eead59" Nov 01 20:50:42.060427 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:42.060217 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9d537cc-9vsb4" Nov 01 20:50:42.137058 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:42.136393 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:50:42.354855 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:42.354816 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78648dee-bptz9"] Nov 01 20:50:42.355290 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:50:42.354942 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod100bce4a_f707_4097_8902_50d8d064c4fd.slice/crio-26728c26eccc3273b141a4022d57e74c1ab7aa8fcaa358594fa0e83cd05811ac WatchSource:0}: Error finding container 26728c26eccc3273b141a4022d57e74c1ab7aa8fcaa358594fa0e83cd05811ac: Status 404 returned error can't find the container with id 26728c26eccc3273b141a4022d57e74c1ab7aa8fcaa358594fa0e83cd05811ac Nov 01 20:50:43.070796 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:43.070759 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78648dee-bptz9" event={"ID":"100bce4a-f707-4097-8902-50d8d064c4fd","Type":"ContainerStarted","Data":"c45e275f9e2c27873a964593824d5ebb931a73f5ab5ccfc134329194767b79e1"} Nov 01 20:50:43.070979 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:43.070825 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78648dee-bptz9" event={"ID":"100bce4a-f707-4097-8902-50d8d064c4fd","Type":"ContainerStarted","Data":"26728c26eccc3273b141a4022d57e74c1ab7aa8fcaa358594fa0e83cd05811ac"} Nov 01 20:50:43.194452 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:43.194403 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78648dee-bptz9" podStartSLOduration=2.19438984 podStartE2EDuration="2.19438984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:50:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:50:43.190430416 +0000 UTC m=+1193.968070556" watchObservedRunningTime="2026-11-01 20:50:43.19438984 +0000 UTC m=+1193.972029975" Nov 01 20:50:44.083479 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:44.083443 4484 generic.go:356] "Generic (PLEG): container finished" podID="100bce4a-f707-4097-8902-50d8d064c4fd" containerID="c45e275f9e2c27873a964593824d5ebb931a73f5ab5ccfc134329194767b79e1" exitCode=0 Nov 01 20:50:44.083884 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:44.083495 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78648dee-bptz9" event={"ID":"100bce4a-f707-4097-8902-50d8d064c4fd","Type":"ContainerDied","Data":"c45e275f9e2c27873a964593824d5ebb931a73f5ab5ccfc134329194767b79e1"} Nov 01 20:50:45.228754 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:45.228702 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c4eee07-2wkrt"] Nov 01 20:50:45.238266 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:45.238238 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c4eee07-2wkrt" Nov 01 20:50:45.289756 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:45.289722 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c4eee07-2wkrt"] Nov 01 20:50:45.340586 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:45.340556 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77f7ce89-0b88-475b-b7b7-a9de5d77f34b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c4eee07-2wkrt\" (UID: \"77f7ce89-0b88-475b-b7b7-a9de5d77f34b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c4eee07-2wkrt" Nov 01 20:50:45.340725 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:45.340593 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77f7ce89-0b88-475b-b7b7-a9de5d77f34b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c4eee07-2wkrt\" (UID: \"77f7ce89-0b88-475b-b7b7-a9de5d77f34b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c4eee07-2wkrt" Nov 01 20:50:45.340725 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:45.340717 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt74q\" (UniqueName: \"kubernetes.io/projected/77f7ce89-0b88-475b-b7b7-a9de5d77f34b-kube-api-access-rt74q\") pod \"maas-api-revoke-sub-8c4eee07-2wkrt\" (UID: \"77f7ce89-0b88-475b-b7b7-a9de5d77f34b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c4eee07-2wkrt" Nov 01 20:50:45.342369 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:45.342351 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78648dee-bptz9" Nov 01 20:50:45.441622 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:45.441521 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/100bce4a-f707-4097-8902-50d8d064c4fd-kube-api-access-gljqm\" (UniqueName: \"kubernetes.io/projected/100bce4a-f707-4097-8902-50d8d064c4fd-kube-api-access-gljqm\") pod \"100bce4a-f707-4097-8902-50d8d064c4fd\" (UID: \"100bce4a-f707-4097-8902-50d8d064c4fd\") " Nov 01 20:50:45.441780 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:45.441673 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/100bce4a-f707-4097-8902-50d8d064c4fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/100bce4a-f707-4097-8902-50d8d064c4fd-maas-api-openshift-ca\") pod \"100bce4a-f707-4097-8902-50d8d064c4fd\" (UID: \"100bce4a-f707-4097-8902-50d8d064c4fd\") " Nov 01 20:50:45.441780 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:45.441742 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/100bce4a-f707-4097-8902-50d8d064c4fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/100bce4a-f707-4097-8902-50d8d064c4fd-maas-api-xks-ca\") pod \"100bce4a-f707-4097-8902-50d8d064c4fd\" (UID: \"100bce4a-f707-4097-8902-50d8d064c4fd\") " Nov 01 20:50:45.441942 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:45.441922 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt74q\" (UniqueName: \"kubernetes.io/projected/77f7ce89-0b88-475b-b7b7-a9de5d77f34b-kube-api-access-rt74q\") pod \"maas-api-revoke-sub-8c4eee07-2wkrt\" (UID: \"77f7ce89-0b88-475b-b7b7-a9de5d77f34b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c4eee07-2wkrt" Nov 01 20:50:45.442230 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:45.442206 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/100bce4a-f707-4097-8902-50d8d064c4fd-maas-api-openshift-ca" pod "100bce4a-f707-4097-8902-50d8d064c4fd" (UID: "100bce4a-f707-4097-8902-50d8d064c4fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:50:45.442333 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:45.442288 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77f7ce89-0b88-475b-b7b7-a9de5d77f34b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c4eee07-2wkrt\" (UID: \"77f7ce89-0b88-475b-b7b7-a9de5d77f34b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c4eee07-2wkrt" Nov 01 20:50:45.442333 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:45.442319 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77f7ce89-0b88-475b-b7b7-a9de5d77f34b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c4eee07-2wkrt\" (UID: \"77f7ce89-0b88-475b-b7b7-a9de5d77f34b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c4eee07-2wkrt" Nov 01 20:50:45.442453 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:45.442443 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/100bce4a-f707-4097-8902-50d8d064c4fd-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:45.443405 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:45.443374 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77f7ce89-0b88-475b-b7b7-a9de5d77f34b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c4eee07-2wkrt\" (UID: \"77f7ce89-0b88-475b-b7b7-a9de5d77f34b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c4eee07-2wkrt" Nov 01 20:50:45.443787 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:45.443759 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/100bce4a-f707-4097-8902-50d8d064c4fd-kube-api-access-gljqm" pod "100bce4a-f707-4097-8902-50d8d064c4fd" (UID: "100bce4a-f707-4097-8902-50d8d064c4fd"). InnerVolumeSpecName "kube-api-access-gljqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:50:45.444359 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:45.444342 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/100bce4a-f707-4097-8902-50d8d064c4fd-maas-api-xks-ca" pod "100bce4a-f707-4097-8902-50d8d064c4fd" (UID: "100bce4a-f707-4097-8902-50d8d064c4fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:50:45.444687 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:45.444670 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77f7ce89-0b88-475b-b7b7-a9de5d77f34b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c4eee07-2wkrt\" (UID: \"77f7ce89-0b88-475b-b7b7-a9de5d77f34b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c4eee07-2wkrt" Nov 01 20:50:45.499307 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:45.499271 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt74q\" (UniqueName: \"kubernetes.io/projected/77f7ce89-0b88-475b-b7b7-a9de5d77f34b-kube-api-access-rt74q\") pod \"maas-api-revoke-sub-8c4eee07-2wkrt\" (UID: \"77f7ce89-0b88-475b-b7b7-a9de5d77f34b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c4eee07-2wkrt" Nov 01 20:50:45.558731 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:45.558695 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c4eee07-2wkrt" Nov 01 20:50:45.559414 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:45.559388 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gljqm\" (UniqueName: \"kubernetes.io/projected/100bce4a-f707-4097-8902-50d8d064c4fd-kube-api-access-gljqm\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:45.559458 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:45.559419 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/100bce4a-f707-4097-8902-50d8d064c4fd-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:45.767257 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:50:45.767212 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 20:50:45.943974 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:50:45.943943 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77f7ce89_0b88_475b_b7b7_a9de5d77f34b.slice/crio-1f194fbb48be04f2d3657aefcf14131ff25b0f2b2ba4abe4db284d632877b414 WatchSource:0}: Error finding container 1f194fbb48be04f2d3657aefcf14131ff25b0f2b2ba4abe4db284d632877b414: Status 404 returned error can't find the container with id 1f194fbb48be04f2d3657aefcf14131ff25b0f2b2ba4abe4db284d632877b414 Nov 01 20:50:45.949982 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:45.946295 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c4eee07-2wkrt"] Nov 01 20:50:46.113609 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:46.113520 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78648dee-bptz9" Nov 01 20:50:46.113609 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:46.113515 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78648dee-bptz9" event={"ID":"100bce4a-f707-4097-8902-50d8d064c4fd","Type":"ContainerDied","Data":"26728c26eccc3273b141a4022d57e74c1ab7aa8fcaa358594fa0e83cd05811ac"} Nov 01 20:50:46.113830 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:46.113694 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26728c26eccc3273b141a4022d57e74c1ab7aa8fcaa358594fa0e83cd05811ac" Nov 01 20:50:46.115040 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:46.115018 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c4eee07-2wkrt" event={"ID":"77f7ce89-0b88-475b-b7b7-a9de5d77f34b","Type":"ContainerStarted","Data":"1f194fbb48be04f2d3657aefcf14131ff25b0f2b2ba4abe4db284d632877b414"} Nov 01 20:50:46.945547 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:46.945510 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:50:47.125487 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:47.125395 4484 generic.go:356] "Generic (PLEG): container finished" podID="77f7ce89-0b88-475b-b7b7-a9de5d77f34b" containerID="03b251dbb3d8691f8b72475b7c30d1f2c8851c87a1835878d0a0a93e27434956" exitCode=0 Nov 01 20:50:47.125487 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:47.125460 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c4eee07-2wkrt" event={"ID":"77f7ce89-0b88-475b-b7b7-a9de5d77f34b","Type":"ContainerDied","Data":"03b251dbb3d8691f8b72475b7c30d1f2c8851c87a1835878d0a0a93e27434956"} Nov 01 20:50:48.302023 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:48.301982 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c4eee07-2wkrt" Nov 01 20:50:48.399361 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:48.399279 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77f7ce89-0b88-475b-b7b7-a9de5d77f34b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77f7ce89-0b88-475b-b7b7-a9de5d77f34b-maas-api-xks-ca\") pod \"77f7ce89-0b88-475b-b7b7-a9de5d77f34b\" (UID: \"77f7ce89-0b88-475b-b7b7-a9de5d77f34b\") " Nov 01 20:50:48.399504 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:48.399438 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77f7ce89-0b88-475b-b7b7-a9de5d77f34b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77f7ce89-0b88-475b-b7b7-a9de5d77f34b-maas-api-openshift-ca\") pod \"77f7ce89-0b88-475b-b7b7-a9de5d77f34b\" (UID: \"77f7ce89-0b88-475b-b7b7-a9de5d77f34b\") " Nov 01 20:50:48.399504 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:48.399496 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77f7ce89-0b88-475b-b7b7-a9de5d77f34b-kube-api-access-rt74q\" (UniqueName: \"kubernetes.io/projected/77f7ce89-0b88-475b-b7b7-a9de5d77f34b-kube-api-access-rt74q\") pod \"77f7ce89-0b88-475b-b7b7-a9de5d77f34b\" (UID: \"77f7ce89-0b88-475b-b7b7-a9de5d77f34b\") " Nov 01 20:50:48.399817 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:48.399785 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77f7ce89-0b88-475b-b7b7-a9de5d77f34b-maas-api-openshift-ca" pod "77f7ce89-0b88-475b-b7b7-a9de5d77f34b" (UID: "77f7ce89-0b88-475b-b7b7-a9de5d77f34b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:50:48.400042 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:48.400022 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77f7ce89-0b88-475b-b7b7-a9de5d77f34b-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:48.401487 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:48.401469 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77f7ce89-0b88-475b-b7b7-a9de5d77f34b-maas-api-xks-ca" pod "77f7ce89-0b88-475b-b7b7-a9de5d77f34b" (UID: "77f7ce89-0b88-475b-b7b7-a9de5d77f34b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:50:48.401576 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:48.401559 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77f7ce89-0b88-475b-b7b7-a9de5d77f34b-kube-api-access-rt74q" pod "77f7ce89-0b88-475b-b7b7-a9de5d77f34b" (UID: "77f7ce89-0b88-475b-b7b7-a9de5d77f34b"). InnerVolumeSpecName "kube-api-access-rt74q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:50:48.501248 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:48.501216 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77f7ce89-0b88-475b-b7b7-a9de5d77f34b-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:48.501248 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:48.501243 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt74q\" (UniqueName: \"kubernetes.io/projected/77f7ce89-0b88-475b-b7b7-a9de5d77f34b-kube-api-access-rt74q\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:49.144254 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:49.144223 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c4eee07-2wkrt" Nov 01 20:50:49.144436 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:49.144253 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c4eee07-2wkrt" event={"ID":"77f7ce89-0b88-475b-b7b7-a9de5d77f34b","Type":"ContainerDied","Data":"1f194fbb48be04f2d3657aefcf14131ff25b0f2b2ba4abe4db284d632877b414"} Nov 01 20:50:49.144436 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:49.144300 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f194fbb48be04f2d3657aefcf14131ff25b0f2b2ba4abe4db284d632877b414" Nov 01 20:50:50.049654 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:50.049618 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-438e9c5e-rt5dz"] Nov 01 20:50:50.100986 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:50.100955 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1184d79b-27xvv"] Nov 01 20:50:50.106213 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:50.106179 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-438e9c5e-rt5dz"] Nov 01 20:50:50.112786 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:50.112759 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1184d79b-27xvv"] Nov 01 20:50:50.384291 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:50.384213 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8aafd12-blzc2"] Nov 01 20:50:50.384864 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:50.384850 4484 cpu_manager.go:414] "containerMap: removing container" podUID="77f7ce89-0b88-475b-b7b7-a9de5d77f34b" containerName="revoke-keys" Nov 01 20:50:50.384902 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:50.384865 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77f7ce89-0b88-475b-b7b7-a9de5d77f34b" containerName="revoke-keys" Nov 01 20:50:50.384936 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:50.384902 4484 cpu_manager.go:414] "containerMap: removing container" podUID="100bce4a-f707-4097-8902-50d8d064c4fd" containerName="revoke-keys" Nov 01 20:50:50.384936 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:50.384908 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="100bce4a-f707-4097-8902-50d8d064c4fd" containerName="revoke-keys" Nov 01 20:50:50.385028 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:50.385017 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="77f7ce89-0b88-475b-b7b7-a9de5d77f34b" containerName="revoke-keys" Nov 01 20:50:50.385075 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:50.385032 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="100bce4a-f707-4097-8902-50d8d064c4fd" containerName="revoke-keys" Nov 01 20:50:50.389906 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:50.389888 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8aafd12-blzc2" Nov 01 20:50:50.425098 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:50.425067 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a861ff5-a3a4-4a99-a555-cc253ca9dbe2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8aafd12-blzc2\" (UID: \"6a861ff5-a3a4-4a99-a555-cc253ca9dbe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8aafd12-blzc2" Nov 01 20:50:50.425297 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:50.425272 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvvz7\" (UniqueName: \"kubernetes.io/projected/6a861ff5-a3a4-4a99-a555-cc253ca9dbe2-kube-api-access-mvvz7\") pod \"maas-api-revoke-sub-e8aafd12-blzc2\" (UID: \"6a861ff5-a3a4-4a99-a555-cc253ca9dbe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8aafd12-blzc2" Nov 01 20:50:50.425388 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:50.425376 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a861ff5-a3a4-4a99-a555-cc253ca9dbe2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8aafd12-blzc2\" (UID: \"6a861ff5-a3a4-4a99-a555-cc253ca9dbe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8aafd12-blzc2" Nov 01 20:50:50.443152 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:50.443119 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:50:50.444184 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:50.444162 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8aafd12-blzc2"] Nov 01 20:50:50.489910 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:50.489878 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:50:50.489910 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:50.489879 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:50:50.490138 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:50.489918 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 20:50:50.526465 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:50.526429 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvvz7\" (UniqueName: \"kubernetes.io/projected/6a861ff5-a3a4-4a99-a555-cc253ca9dbe2-kube-api-access-mvvz7\") pod \"maas-api-revoke-sub-e8aafd12-blzc2\" (UID: \"6a861ff5-a3a4-4a99-a555-cc253ca9dbe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8aafd12-blzc2" Nov 01 20:50:50.526638 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:50.526509 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a861ff5-a3a4-4a99-a555-cc253ca9dbe2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8aafd12-blzc2\" (UID: \"6a861ff5-a3a4-4a99-a555-cc253ca9dbe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8aafd12-blzc2" Nov 01 20:50:50.526638 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:50.526573 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a861ff5-a3a4-4a99-a555-cc253ca9dbe2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8aafd12-blzc2\" (UID: \"6a861ff5-a3a4-4a99-a555-cc253ca9dbe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8aafd12-blzc2" Nov 01 20:50:50.527294 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:50.527262 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a861ff5-a3a4-4a99-a555-cc253ca9dbe2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8aafd12-blzc2\" (UID: \"6a861ff5-a3a4-4a99-a555-cc253ca9dbe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8aafd12-blzc2" Nov 01 20:50:50.528808 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:50.528790 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a861ff5-a3a4-4a99-a555-cc253ca9dbe2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8aafd12-blzc2\" (UID: \"6a861ff5-a3a4-4a99-a555-cc253ca9dbe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8aafd12-blzc2" Nov 01 20:50:50.583818 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:50.583791 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvvz7\" (UniqueName: \"kubernetes.io/projected/6a861ff5-a3a4-4a99-a555-cc253ca9dbe2-kube-api-access-mvvz7\") pod \"maas-api-revoke-sub-e8aafd12-blzc2\" (UID: \"6a861ff5-a3a4-4a99-a555-cc253ca9dbe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8aafd12-blzc2" Nov 01 20:50:50.703807 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:50.703726 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8aafd12-blzc2" Nov 01 20:50:51.047429 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:51.047399 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e72e83b8-8wq9j"] Nov 01 20:50:51.051530 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:51.051501 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e72e83b8-8wq9j"] Nov 01 20:50:51.084468 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:51.084436 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8aafd12-blzc2"] Nov 01 20:50:51.124715 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:50:51.124682 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a861ff5_a3a4_4a99_a555_cc253ca9dbe2.slice/crio-4cbb9fe7a88193902861529efc4e1dee8303e849a38bee03ed3463aac17f0b56 WatchSource:0}: Error finding container 4cbb9fe7a88193902861529efc4e1dee8303e849a38bee03ed3463aac17f0b56: Status 404 returned error can't find the container with id 4cbb9fe7a88193902861529efc4e1dee8303e849a38bee03ed3463aac17f0b56 Nov 01 20:50:51.161301 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:51.161267 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8aafd12-blzc2" event={"ID":"6a861ff5-a3a4-4a99-a555-cc253ca9dbe2","Type":"ContainerStarted","Data":"4cbb9fe7a88193902861529efc4e1dee8303e849a38bee03ed3463aac17f0b56"} Nov 01 20:50:51.376701 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:51.376657 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e0e4ee4-329b-4600-89c5-82e6de5b0479" path="/var/lib/kubelet/pods/5e0e4ee4-329b-4600-89c5-82e6de5b0479/volumes" Nov 01 20:50:51.377367 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:51.377346 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d51ed2a-f4ab-434a-af8b-ebf29ed29198" path="/var/lib/kubelet/pods/9d51ed2a-f4ab-434a-af8b-ebf29ed29198/volumes" Nov 01 20:50:51.377926 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:51.377908 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee3f4ee2-4fa0-4db0-8bc0-19a591f90f1e" path="/var/lib/kubelet/pods/ee3f4ee2-4fa0-4db0-8bc0-19a591f90f1e/volumes" Nov 01 20:50:51.737221 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:51.737130 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:50:52.172448 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:52.172415 4484 generic.go:356] "Generic (PLEG): container finished" podID="6a861ff5-a3a4-4a99-a555-cc253ca9dbe2" containerID="67843f2db6ee32ab4120445cbaa8956485ffd8066102947766c63a9cc268dc20" exitCode=0 Nov 01 20:50:52.172894 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:52.172494 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8aafd12-blzc2" event={"ID":"6a861ff5-a3a4-4a99-a555-cc253ca9dbe2","Type":"ContainerDied","Data":"67843f2db6ee32ab4120445cbaa8956485ffd8066102947766c63a9cc268dc20"} Nov 01 20:50:53.356356 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:53.356331 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8aafd12-blzc2" Nov 01 20:50:53.466724 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:53.466638 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a861ff5-a3a4-4a99-a555-cc253ca9dbe2-kube-api-access-mvvz7\" (UniqueName: \"kubernetes.io/projected/6a861ff5-a3a4-4a99-a555-cc253ca9dbe2-kube-api-access-mvvz7\") pod \"6a861ff5-a3a4-4a99-a555-cc253ca9dbe2\" (UID: \"6a861ff5-a3a4-4a99-a555-cc253ca9dbe2\") " Nov 01 20:50:53.466883 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:53.466738 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a861ff5-a3a4-4a99-a555-cc253ca9dbe2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a861ff5-a3a4-4a99-a555-cc253ca9dbe2-maas-api-openshift-ca\") pod \"6a861ff5-a3a4-4a99-a555-cc253ca9dbe2\" (UID: \"6a861ff5-a3a4-4a99-a555-cc253ca9dbe2\") " Nov 01 20:50:53.466883 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:53.466810 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a861ff5-a3a4-4a99-a555-cc253ca9dbe2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a861ff5-a3a4-4a99-a555-cc253ca9dbe2-maas-api-xks-ca\") pod \"6a861ff5-a3a4-4a99-a555-cc253ca9dbe2\" (UID: \"6a861ff5-a3a4-4a99-a555-cc253ca9dbe2\") " Nov 01 20:50:53.467159 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:53.467137 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a861ff5-a3a4-4a99-a555-cc253ca9dbe2-maas-api-openshift-ca" pod "6a861ff5-a3a4-4a99-a555-cc253ca9dbe2" (UID: "6a861ff5-a3a4-4a99-a555-cc253ca9dbe2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:50:53.467618 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:53.467591 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a861ff5-a3a4-4a99-a555-cc253ca9dbe2-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:53.468956 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:53.468934 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a861ff5-a3a4-4a99-a555-cc253ca9dbe2-maas-api-xks-ca" pod "6a861ff5-a3a4-4a99-a555-cc253ca9dbe2" (UID: "6a861ff5-a3a4-4a99-a555-cc253ca9dbe2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:50:53.469115 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:53.468955 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a861ff5-a3a4-4a99-a555-cc253ca9dbe2-kube-api-access-mvvz7" pod "6a861ff5-a3a4-4a99-a555-cc253ca9dbe2" (UID: "6a861ff5-a3a4-4a99-a555-cc253ca9dbe2"). InnerVolumeSpecName "kube-api-access-mvvz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:50:53.568757 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:53.568710 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a861ff5-a3a4-4a99-a555-cc253ca9dbe2-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:53.568757 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:53.568747 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvvz7\" (UniqueName: \"kubernetes.io/projected/6a861ff5-a3a4-4a99-a555-cc253ca9dbe2-kube-api-access-mvvz7\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:54.188684 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:54.188651 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8aafd12-blzc2" Nov 01 20:50:54.188849 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:54.188681 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8aafd12-blzc2" event={"ID":"6a861ff5-a3a4-4a99-a555-cc253ca9dbe2","Type":"ContainerDied","Data":"4cbb9fe7a88193902861529efc4e1dee8303e849a38bee03ed3463aac17f0b56"} Nov 01 20:50:54.188849 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:54.188739 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4cbb9fe7a88193902861529efc4e1dee8303e849a38bee03ed3463aac17f0b56" Nov 01 20:50:55.532894 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:55.532863 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8"] Nov 01 20:50:55.533552 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:55.533537 4484 cpu_manager.go:414] "containerMap: removing container" podUID="6a861ff5-a3a4-4a99-a555-cc253ca9dbe2" containerName="revoke-keys" Nov 01 20:50:55.533592 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:55.533552 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a861ff5-a3a4-4a99-a555-cc253ca9dbe2" containerName="revoke-keys" Nov 01 20:50:55.533662 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:55.533654 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a861ff5-a3a4-4a99-a555-cc253ca9dbe2" containerName="revoke-keys" Nov 01 20:50:55.537612 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:55.537587 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8" Nov 01 20:50:55.594928 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:55.594891 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b373faf1-ffb2-4a45-b433-0669635a397c-istiod-ca-cert\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8\" (UID: \"b373faf1-ffb2-4a45-b433-0669635a397c\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8" Nov 01 20:50:55.595107 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:55.594950 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b373faf1-ffb2-4a45-b433-0669635a397c-istio-envoy\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8\" (UID: \"b373faf1-ffb2-4a45-b433-0669635a397c\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8" Nov 01 20:50:55.595107 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:55.595086 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b373faf1-ffb2-4a45-b433-0669635a397c-istio-podinfo\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8\" (UID: \"b373faf1-ffb2-4a45-b433-0669635a397c\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8" Nov 01 20:50:55.595208 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:55.595114 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b373faf1-ffb2-4a45-b433-0669635a397c-credential-socket\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8\" (UID: \"b373faf1-ffb2-4a45-b433-0669635a397c\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8" Nov 01 20:50:55.595208 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:55.595173 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b373faf1-ffb2-4a45-b433-0669635a397c-istio-token\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8\" (UID: \"b373faf1-ffb2-4a45-b433-0669635a397c\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8" Nov 01 20:50:55.595314 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:55.595215 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9q4n\" (UniqueName: \"kubernetes.io/projected/b373faf1-ffb2-4a45-b433-0669635a397c-kube-api-access-w9q4n\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8\" (UID: \"b373faf1-ffb2-4a45-b433-0669635a397c\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8" Nov 01 20:50:55.595314 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:55.595296 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b373faf1-ffb2-4a45-b433-0669635a397c-istio-data\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8\" (UID: \"b373faf1-ffb2-4a45-b433-0669635a397c\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8" Nov 01 20:50:55.595383 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:55.595343 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b373faf1-ffb2-4a45-b433-0669635a397c-workload-certs\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8\" (UID: \"b373faf1-ffb2-4a45-b433-0669635a397c\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8" Nov 01 20:50:55.595430 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:55.595403 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b373faf1-ffb2-4a45-b433-0669635a397c-workload-socket\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8\" (UID: \"b373faf1-ffb2-4a45-b433-0669635a397c\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8" Nov 01 20:50:55.646510 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:55.646474 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8"] Nov 01 20:50:55.696469 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:55.696432 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b373faf1-ffb2-4a45-b433-0669635a397c-istio-podinfo\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8\" (UID: \"b373faf1-ffb2-4a45-b433-0669635a397c\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8" Nov 01 20:50:55.696469 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:55.696475 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b373faf1-ffb2-4a45-b433-0669635a397c-credential-socket\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8\" (UID: \"b373faf1-ffb2-4a45-b433-0669635a397c\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8" Nov 01 20:50:55.696718 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:55.696522 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b373faf1-ffb2-4a45-b433-0669635a397c-istio-token\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8\" (UID: \"b373faf1-ffb2-4a45-b433-0669635a397c\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8" Nov 01 20:50:55.696718 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:55.696553 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9q4n\" (UniqueName: \"kubernetes.io/projected/b373faf1-ffb2-4a45-b433-0669635a397c-kube-api-access-w9q4n\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8\" (UID: \"b373faf1-ffb2-4a45-b433-0669635a397c\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8" Nov 01 20:50:55.696718 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:55.696645 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b373faf1-ffb2-4a45-b433-0669635a397c-istio-data\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8\" (UID: \"b373faf1-ffb2-4a45-b433-0669635a397c\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8" Nov 01 20:50:55.696718 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:55.696690 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b373faf1-ffb2-4a45-b433-0669635a397c-workload-certs\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8\" (UID: \"b373faf1-ffb2-4a45-b433-0669635a397c\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8" Nov 01 20:50:55.696924 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:55.696734 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b373faf1-ffb2-4a45-b433-0669635a397c-workload-socket\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8\" (UID: \"b373faf1-ffb2-4a45-b433-0669635a397c\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8" Nov 01 20:50:55.697096 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:55.697031 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b373faf1-ffb2-4a45-b433-0669635a397c-credential-socket\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8\" (UID: \"b373faf1-ffb2-4a45-b433-0669635a397c\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8" Nov 01 20:50:55.697096 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:55.697069 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b373faf1-ffb2-4a45-b433-0669635a397c-istiod-ca-cert\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8\" (UID: \"b373faf1-ffb2-4a45-b433-0669635a397c\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8" Nov 01 20:50:55.697096 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:55.697076 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b373faf1-ffb2-4a45-b433-0669635a397c-istio-data\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8\" (UID: \"b373faf1-ffb2-4a45-b433-0669635a397c\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8" Nov 01 20:50:55.697317 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:55.697102 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b373faf1-ffb2-4a45-b433-0669635a397c-workload-certs\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8\" (UID: \"b373faf1-ffb2-4a45-b433-0669635a397c\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8" Nov 01 20:50:55.697317 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:55.697157 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b373faf1-ffb2-4a45-b433-0669635a397c-workload-socket\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8\" (UID: \"b373faf1-ffb2-4a45-b433-0669635a397c\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8" Nov 01 20:50:55.697317 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:55.697172 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b373faf1-ffb2-4a45-b433-0669635a397c-istio-envoy\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8\" (UID: \"b373faf1-ffb2-4a45-b433-0669635a397c\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8" Nov 01 20:50:55.697537 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:55.697519 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b373faf1-ffb2-4a45-b433-0669635a397c-istiod-ca-cert\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8\" (UID: \"b373faf1-ffb2-4a45-b433-0669635a397c\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8" Nov 01 20:50:55.699058 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:55.699031 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b373faf1-ffb2-4a45-b433-0669635a397c-istio-podinfo\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8\" (UID: \"b373faf1-ffb2-4a45-b433-0669635a397c\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8" Nov 01 20:50:55.699198 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:55.699180 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b373faf1-ffb2-4a45-b433-0669635a397c-istio-envoy\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8\" (UID: \"b373faf1-ffb2-4a45-b433-0669635a397c\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8" Nov 01 20:50:55.754136 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:55.754101 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b373faf1-ffb2-4a45-b433-0669635a397c-istio-token\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8\" (UID: \"b373faf1-ffb2-4a45-b433-0669635a397c\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8" Nov 01 20:50:55.754323 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:55.754241 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9q4n\" (UniqueName: \"kubernetes.io/projected/b373faf1-ffb2-4a45-b433-0669635a397c-kube-api-access-w9q4n\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8\" (UID: \"b373faf1-ffb2-4a45-b433-0669635a397c\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8" Nov 01 20:50:55.851335 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:55.851254 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8" Nov 01 20:50:56.281236 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:50:56.281208 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb373faf1_ffb2_4a45_b433_0669635a397c.slice/crio-df29c92cb8534650ae52a18d1ba0d842a9cb9aad1a175c332b06c54ae1f83ab1 WatchSource:0}: Error finding container df29c92cb8534650ae52a18d1ba0d842a9cb9aad1a175c332b06c54ae1f83ab1: Status 404 returned error can't find the container with id df29c92cb8534650ae52a18d1ba0d842a9cb9aad1a175c332b06c54ae1f83ab1 Nov 01 20:50:56.281975 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:56.281949 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8"] Nov 01 20:50:56.283913 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:56.283880 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:50:56.284029 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:56.283944 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:50:56.284029 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:56.283976 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:50:56.552813 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:56.552723 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:50:57.140571 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:57.140541 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87582d8d-n8b29"] Nov 01 20:50:57.144737 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:57.144716 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87582d8d-n8b29" Nov 01 20:50:57.198082 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:57.198047 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:50:57.198082 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:57.198061 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:50:57.238085 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:57.238047 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c1e21b4-946b-4351-92b3-948b442bdac3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87582d8d-n8b29\" (UID: \"9c1e21b4-946b-4351-92b3-948b442bdac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87582d8d-n8b29" Nov 01 20:50:57.238296 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:57.238182 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28fsm\" (UniqueName: \"kubernetes.io/projected/9c1e21b4-946b-4351-92b3-948b442bdac3-kube-api-access-28fsm\") pod \"maas-api-revoke-sub-87582d8d-n8b29\" (UID: \"9c1e21b4-946b-4351-92b3-948b442bdac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87582d8d-n8b29" Nov 01 20:50:57.238388 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:57.238370 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c1e21b4-946b-4351-92b3-948b442bdac3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87582d8d-n8b29\" (UID: \"9c1e21b4-946b-4351-92b3-948b442bdac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87582d8d-n8b29" Nov 01 20:50:57.243625 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:57.243590 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8" event={"ID":"b373faf1-ffb2-4a45-b433-0669635a397c","Type":"ContainerStarted","Data":"b6541f7eefe7ba24ae375b42001c3d8d5be218cac65c3c82b0736e0ce422fffe"} Nov 01 20:50:57.243779 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:57.243650 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8" event={"ID":"b373faf1-ffb2-4a45-b433-0669635a397c","Type":"ContainerStarted","Data":"df29c92cb8534650ae52a18d1ba0d842a9cb9aad1a175c332b06c54ae1f83ab1"} Nov 01 20:50:57.248681 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:57.248656 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:50:57.248827 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:57.248724 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 20:50:57.255911 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:57.255889 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87582d8d-n8b29"] Nov 01 20:50:57.338819 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:57.338780 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a49a9a09-p2vd8"] Nov 01 20:50:57.339579 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:57.339554 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c1e21b4-946b-4351-92b3-948b442bdac3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87582d8d-n8b29\" (UID: \"9c1e21b4-946b-4351-92b3-948b442bdac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87582d8d-n8b29" Nov 01 20:50:57.339715 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:57.339674 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28fsm\" (UniqueName: \"kubernetes.io/projected/9c1e21b4-946b-4351-92b3-948b442bdac3-kube-api-access-28fsm\") pod \"maas-api-revoke-sub-87582d8d-n8b29\" (UID: \"9c1e21b4-946b-4351-92b3-948b442bdac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87582d8d-n8b29" Nov 01 20:50:57.339864 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:57.339833 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c1e21b4-946b-4351-92b3-948b442bdac3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87582d8d-n8b29\" (UID: \"9c1e21b4-946b-4351-92b3-948b442bdac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87582d8d-n8b29" Nov 01 20:50:57.340727 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:57.340632 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c1e21b4-946b-4351-92b3-948b442bdac3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87582d8d-n8b29\" (UID: \"9c1e21b4-946b-4351-92b3-948b442bdac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87582d8d-n8b29" Nov 01 20:50:57.342306 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:57.342281 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c1e21b4-946b-4351-92b3-948b442bdac3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87582d8d-n8b29\" (UID: \"9c1e21b4-946b-4351-92b3-948b442bdac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87582d8d-n8b29" Nov 01 20:50:57.355326 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:57.355306 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49a9a09-p2vd8" Nov 01 20:50:57.375624 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:57.375575 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8" podStartSLOduration=2.375561824 podStartE2EDuration="2.375561824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:50:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:50:57.374912801 +0000 UTC m=+1208.152552937" watchObservedRunningTime="2026-11-01 20:50:57.375561824 +0000 UTC m=+1208.153201958" Nov 01 20:50:57.398651 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:57.398576 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a49a9a09-p2vd8"] Nov 01 20:50:57.399810 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:57.399787 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28fsm\" (UniqueName: \"kubernetes.io/projected/9c1e21b4-946b-4351-92b3-948b442bdac3-kube-api-access-28fsm\") pod \"maas-api-revoke-sub-87582d8d-n8b29\" (UID: \"9c1e21b4-946b-4351-92b3-948b442bdac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87582d8d-n8b29" Nov 01 20:50:57.441529 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:57.441479 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fbaa918-a4bc-4157-8c63-5181f2e0ac86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a49a9a09-p2vd8\" (UID: \"2fbaa918-a4bc-4157-8c63-5181f2e0ac86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49a9a09-p2vd8" Nov 01 20:50:57.441845 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:57.441811 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fbaa918-a4bc-4157-8c63-5181f2e0ac86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a49a9a09-p2vd8\" (UID: \"2fbaa918-a4bc-4157-8c63-5181f2e0ac86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49a9a09-p2vd8" Nov 01 20:50:57.441968 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:57.441920 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnw2r\" (UniqueName: \"kubernetes.io/projected/2fbaa918-a4bc-4157-8c63-5181f2e0ac86-kube-api-access-pnw2r\") pod \"maas-api-revoke-sub-a49a9a09-p2vd8\" (UID: \"2fbaa918-a4bc-4157-8c63-5181f2e0ac86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49a9a09-p2vd8" Nov 01 20:50:57.461063 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:57.461031 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87582d8d-n8b29" Nov 01 20:50:57.543959 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:57.543927 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fbaa918-a4bc-4157-8c63-5181f2e0ac86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a49a9a09-p2vd8\" (UID: \"2fbaa918-a4bc-4157-8c63-5181f2e0ac86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49a9a09-p2vd8" Nov 01 20:50:57.544116 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:57.543985 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnw2r\" (UniqueName: \"kubernetes.io/projected/2fbaa918-a4bc-4157-8c63-5181f2e0ac86-kube-api-access-pnw2r\") pod \"maas-api-revoke-sub-a49a9a09-p2vd8\" (UID: \"2fbaa918-a4bc-4157-8c63-5181f2e0ac86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49a9a09-p2vd8" Nov 01 20:50:57.544169 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:57.544147 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fbaa918-a4bc-4157-8c63-5181f2e0ac86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a49a9a09-p2vd8\" (UID: \"2fbaa918-a4bc-4157-8c63-5181f2e0ac86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49a9a09-p2vd8" Nov 01 20:50:57.544958 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:57.544921 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fbaa918-a4bc-4157-8c63-5181f2e0ac86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a49a9a09-p2vd8\" (UID: \"2fbaa918-a4bc-4157-8c63-5181f2e0ac86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49a9a09-p2vd8" Nov 01 20:50:57.547075 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:57.547049 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fbaa918-a4bc-4157-8c63-5181f2e0ac86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a49a9a09-p2vd8\" (UID: \"2fbaa918-a4bc-4157-8c63-5181f2e0ac86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49a9a09-p2vd8" Nov 01 20:50:57.602789 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:57.602745 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnw2r\" (UniqueName: \"kubernetes.io/projected/2fbaa918-a4bc-4157-8c63-5181f2e0ac86-kube-api-access-pnw2r\") pod \"maas-api-revoke-sub-a49a9a09-p2vd8\" (UID: \"2fbaa918-a4bc-4157-8c63-5181f2e0ac86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49a9a09-p2vd8" Nov 01 20:50:57.670465 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:57.670380 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49a9a09-p2vd8" Nov 01 20:50:57.851858 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:57.851818 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8" Nov 01 20:50:57.852978 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:57.852945 4484 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 01 20:50:57.853178 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:57.853033 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8" podUID="b373faf1-ffb2-4a45-b433-0669635a397c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 01 20:50:57.876709 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:57.876669 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87582d8d-n8b29"] Nov 01 20:50:57.881294 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:50:57.881265 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c1e21b4_946b_4351_92b3_948b442bdac3.slice/crio-ffa30ae757c0cab039b6e44e9120e418226083062df6d60a93166767f46696ce WatchSource:0}: Error finding container ffa30ae757c0cab039b6e44e9120e418226083062df6d60a93166767f46696ce: Status 404 returned error can't find the container with id ffa30ae757c0cab039b6e44e9120e418226083062df6d60a93166767f46696ce Nov 01 20:50:58.075571 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:58.075544 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a49a9a09-p2vd8"] Nov 01 20:50:58.256978 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:58.256936 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87582d8d-n8b29" event={"ID":"9c1e21b4-946b-4351-92b3-948b442bdac3","Type":"ContainerStarted","Data":"ffa30ae757c0cab039b6e44e9120e418226083062df6d60a93166767f46696ce"} Nov 01 20:50:58.260148 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:58.260116 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49a9a09-p2vd8" event={"ID":"2fbaa918-a4bc-4157-8c63-5181f2e0ac86","Type":"ContainerStarted","Data":"44cb8619bb304356be0f24b01935bfd4370cce0389571796b21c646044ab662b"} Nov 01 20:50:58.852100 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:58.851980 4484 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 01 20:50:58.852100 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:58.852066 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8" podUID="b373faf1-ffb2-4a45-b433-0669635a397c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 01 20:50:59.267411 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:59.267374 4484 generic.go:356] "Generic (PLEG): container finished" podID="9c1e21b4-946b-4351-92b3-948b442bdac3" containerID="243056b64df75f2a68af1fbcd9216366c182dad3c09fa6f3eb8426326c34239a" exitCode=0 Nov 01 20:50:59.267605 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:59.267458 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87582d8d-n8b29" event={"ID":"9c1e21b4-946b-4351-92b3-948b442bdac3","Type":"ContainerDied","Data":"243056b64df75f2a68af1fbcd9216366c182dad3c09fa6f3eb8426326c34239a"} Nov 01 20:50:59.269362 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:59.269293 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49a9a09-p2vd8" event={"ID":"2fbaa918-a4bc-4157-8c63-5181f2e0ac86","Type":"ContainerStarted","Data":"9d4acc7b681b2f003986d233869841a14bafd33d8707a5a533c9401b0b9a6a1e"} Nov 01 20:50:59.533079 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:59.533027 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49a9a09-p2vd8" podStartSLOduration=2.533011079 podStartE2EDuration="2.533011079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:50:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:50:59.529267329 +0000 UTC m=+1210.306907466" watchObservedRunningTime="2026-11-01 20:50:59.533011079 +0000 UTC m=+1210.310651250" Nov 01 20:50:59.852676 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:59.852601 4484 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 01 20:50:59.852676 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:50:59.852656 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8" podUID="b373faf1-ffb2-4a45-b433-0669635a397c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 01 20:51:00.278687 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:00.278653 4484 generic.go:356] "Generic (PLEG): container finished" podID="2fbaa918-a4bc-4157-8c63-5181f2e0ac86" containerID="9d4acc7b681b2f003986d233869841a14bafd33d8707a5a533c9401b0b9a6a1e" exitCode=0 Nov 01 20:51:00.278861 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:00.278744 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49a9a09-p2vd8" event={"ID":"2fbaa918-a4bc-4157-8c63-5181f2e0ac86","Type":"ContainerDied","Data":"9d4acc7b681b2f003986d233869841a14bafd33d8707a5a533c9401b0b9a6a1e"} Nov 01 20:51:00.497226 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:00.497199 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87582d8d-n8b29" Nov 01 20:51:00.589438 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:00.589348 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c1e21b4-946b-4351-92b3-948b442bdac3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c1e21b4-946b-4351-92b3-948b442bdac3-maas-api-xks-ca\") pod \"9c1e21b4-946b-4351-92b3-948b442bdac3\" (UID: \"9c1e21b4-946b-4351-92b3-948b442bdac3\") " Nov 01 20:51:00.589581 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:00.589475 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c1e21b4-946b-4351-92b3-948b442bdac3-kube-api-access-28fsm\" (UniqueName: \"kubernetes.io/projected/9c1e21b4-946b-4351-92b3-948b442bdac3-kube-api-access-28fsm\") pod \"9c1e21b4-946b-4351-92b3-948b442bdac3\" (UID: \"9c1e21b4-946b-4351-92b3-948b442bdac3\") " Nov 01 20:51:00.589636 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:00.589630 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c1e21b4-946b-4351-92b3-948b442bdac3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c1e21b4-946b-4351-92b3-948b442bdac3-maas-api-openshift-ca\") pod \"9c1e21b4-946b-4351-92b3-948b442bdac3\" (UID: \"9c1e21b4-946b-4351-92b3-948b442bdac3\") " Nov 01 20:51:00.590053 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:00.590025 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c1e21b4-946b-4351-92b3-948b442bdac3-maas-api-openshift-ca" pod "9c1e21b4-946b-4351-92b3-948b442bdac3" (UID: "9c1e21b4-946b-4351-92b3-948b442bdac3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:51:00.591586 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:00.591564 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c1e21b4-946b-4351-92b3-948b442bdac3-kube-api-access-28fsm" pod "9c1e21b4-946b-4351-92b3-948b442bdac3" (UID: "9c1e21b4-946b-4351-92b3-948b442bdac3"). InnerVolumeSpecName "kube-api-access-28fsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:51:00.591650 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:00.591630 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c1e21b4-946b-4351-92b3-948b442bdac3-maas-api-xks-ca" pod "9c1e21b4-946b-4351-92b3-948b442bdac3" (UID: "9c1e21b4-946b-4351-92b3-948b442bdac3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:51:00.691407 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:00.691376 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28fsm\" (UniqueName: \"kubernetes.io/projected/9c1e21b4-946b-4351-92b3-948b442bdac3-kube-api-access-28fsm\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:00.691407 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:00.691403 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c1e21b4-946b-4351-92b3-948b442bdac3-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:00.691407 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:00.691416 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c1e21b4-946b-4351-92b3-948b442bdac3-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:00.851924 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:00.851820 4484 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 01 20:51:00.851924 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:00.851896 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8" podUID="b373faf1-ffb2-4a45-b433-0669635a397c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 01 20:51:01.290946 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:01.290914 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87582d8d-n8b29" Nov 01 20:51:01.291499 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:01.290963 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87582d8d-n8b29" event={"ID":"9c1e21b4-946b-4351-92b3-948b442bdac3","Type":"ContainerDied","Data":"ffa30ae757c0cab039b6e44e9120e418226083062df6d60a93166767f46696ce"} Nov 01 20:51:01.291499 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:01.291032 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffa30ae757c0cab039b6e44e9120e418226083062df6d60a93166767f46696ce" Nov 01 20:51:01.345368 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:01.345337 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:51:01.639543 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:01.639515 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49a9a09-p2vd8" Nov 01 20:51:01.729660 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:01.729622 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fbaa918-a4bc-4157-8c63-5181f2e0ac86-kube-api-access-pnw2r\" (UniqueName: \"kubernetes.io/projected/2fbaa918-a4bc-4157-8c63-5181f2e0ac86-kube-api-access-pnw2r\") pod \"2fbaa918-a4bc-4157-8c63-5181f2e0ac86\" (UID: \"2fbaa918-a4bc-4157-8c63-5181f2e0ac86\") " Nov 01 20:51:01.729845 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:01.729797 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fbaa918-a4bc-4157-8c63-5181f2e0ac86-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fbaa918-a4bc-4157-8c63-5181f2e0ac86-maas-api-openshift-ca\") pod \"2fbaa918-a4bc-4157-8c63-5181f2e0ac86\" (UID: \"2fbaa918-a4bc-4157-8c63-5181f2e0ac86\") " Nov 01 20:51:01.729888 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:01.729848 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fbaa918-a4bc-4157-8c63-5181f2e0ac86-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fbaa918-a4bc-4157-8c63-5181f2e0ac86-maas-api-xks-ca\") pod \"2fbaa918-a4bc-4157-8c63-5181f2e0ac86\" (UID: \"2fbaa918-a4bc-4157-8c63-5181f2e0ac86\") " Nov 01 20:51:01.730266 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:01.730237 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fbaa918-a4bc-4157-8c63-5181f2e0ac86-maas-api-openshift-ca" pod "2fbaa918-a4bc-4157-8c63-5181f2e0ac86" (UID: "2fbaa918-a4bc-4157-8c63-5181f2e0ac86"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:51:01.730452 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:01.730435 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fbaa918-a4bc-4157-8c63-5181f2e0ac86-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:01.731866 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:01.731841 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fbaa918-a4bc-4157-8c63-5181f2e0ac86-kube-api-access-pnw2r" pod "2fbaa918-a4bc-4157-8c63-5181f2e0ac86" (UID: "2fbaa918-a4bc-4157-8c63-5181f2e0ac86"). InnerVolumeSpecName "kube-api-access-pnw2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:51:01.732193 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:01.732162 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fbaa918-a4bc-4157-8c63-5181f2e0ac86-maas-api-xks-ca" pod "2fbaa918-a4bc-4157-8c63-5181f2e0ac86" (UID: "2fbaa918-a4bc-4157-8c63-5181f2e0ac86"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:51:01.831942 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:01.831892 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fbaa918-a4bc-4157-8c63-5181f2e0ac86-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:01.831942 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:01.831934 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnw2r\" (UniqueName: \"kubernetes.io/projected/2fbaa918-a4bc-4157-8c63-5181f2e0ac86-kube-api-access-pnw2r\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:01.851946 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:01.851867 4484 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 01 20:51:01.851946 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:01.851922 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8" podUID="b373faf1-ffb2-4a45-b433-0669635a397c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 01 20:51:02.301108 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:02.301065 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49a9a09-p2vd8" event={"ID":"2fbaa918-a4bc-4157-8c63-5181f2e0ac86","Type":"ContainerDied","Data":"44cb8619bb304356be0f24b01935bfd4370cce0389571796b21c646044ab662b"} Nov 01 20:51:02.301108 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:02.301090 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49a9a09-p2vd8" Nov 01 20:51:02.301530 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:02.301141 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44cb8619bb304356be0f24b01935bfd4370cce0389571796b21c646044ab662b" Nov 01 20:51:02.591313 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:02.591224 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad59ce44-2g6wh"] Nov 01 20:51:02.591963 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:02.591944 4484 cpu_manager.go:414] "containerMap: removing container" podUID="2fbaa918-a4bc-4157-8c63-5181f2e0ac86" containerName="revoke-keys" Nov 01 20:51:02.592111 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:02.591965 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fbaa918-a4bc-4157-8c63-5181f2e0ac86" containerName="revoke-keys" Nov 01 20:51:02.592111 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:02.592064 4484 cpu_manager.go:414] "containerMap: removing container" podUID="9c1e21b4-946b-4351-92b3-948b442bdac3" containerName="revoke-keys" Nov 01 20:51:02.592111 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:02.592075 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c1e21b4-946b-4351-92b3-948b442bdac3" containerName="revoke-keys" Nov 01 20:51:02.592277 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:02.592237 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c1e21b4-946b-4351-92b3-948b442bdac3" containerName="revoke-keys" Nov 01 20:51:02.592277 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:02.592257 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fbaa918-a4bc-4157-8c63-5181f2e0ac86" containerName="revoke-keys" Nov 01 20:51:02.596234 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:02.596212 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad59ce44-2g6wh" Nov 01 20:51:02.649708 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:02.649652 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:51:02.650060 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:02.650029 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:51:02.652742 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:02.652717 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad59ce44-2g6wh"] Nov 01 20:51:02.700772 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:02.700739 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:51:02.701024 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:02.700849 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 20:51:02.747915 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:02.747877 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbb6aaa5-b66d-421d-91ea-1f65e593b8f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad59ce44-2g6wh\" (UID: \"dbb6aaa5-b66d-421d-91ea-1f65e593b8f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad59ce44-2g6wh" Nov 01 20:51:02.748165 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:02.748143 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbb6aaa5-b66d-421d-91ea-1f65e593b8f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad59ce44-2g6wh\" (UID: \"dbb6aaa5-b66d-421d-91ea-1f65e593b8f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad59ce44-2g6wh" Nov 01 20:51:02.748273 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:02.748253 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vpvh\" (UniqueName: \"kubernetes.io/projected/dbb6aaa5-b66d-421d-91ea-1f65e593b8f5-kube-api-access-8vpvh\") pod \"maas-api-revoke-sub-ad59ce44-2g6wh\" (UID: \"dbb6aaa5-b66d-421d-91ea-1f65e593b8f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad59ce44-2g6wh" Nov 01 20:51:02.850218 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:02.850134 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vpvh\" (UniqueName: \"kubernetes.io/projected/dbb6aaa5-b66d-421d-91ea-1f65e593b8f5-kube-api-access-8vpvh\") pod \"maas-api-revoke-sub-ad59ce44-2g6wh\" (UID: \"dbb6aaa5-b66d-421d-91ea-1f65e593b8f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad59ce44-2g6wh" Nov 01 20:51:02.850375 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:02.850257 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbb6aaa5-b66d-421d-91ea-1f65e593b8f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad59ce44-2g6wh\" (UID: \"dbb6aaa5-b66d-421d-91ea-1f65e593b8f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad59ce44-2g6wh" Nov 01 20:51:02.850375 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:02.850353 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbb6aaa5-b66d-421d-91ea-1f65e593b8f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad59ce44-2g6wh\" (UID: \"dbb6aaa5-b66d-421d-91ea-1f65e593b8f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad59ce44-2g6wh" Nov 01 20:51:02.851182 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:02.851158 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbb6aaa5-b66d-421d-91ea-1f65e593b8f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad59ce44-2g6wh\" (UID: \"dbb6aaa5-b66d-421d-91ea-1f65e593b8f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad59ce44-2g6wh" Nov 01 20:51:02.852177 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:02.852153 4484 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 01 20:51:02.852288 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:02.852204 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8" podUID="b373faf1-ffb2-4a45-b433-0669635a397c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 01 20:51:02.852797 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:02.852777 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbb6aaa5-b66d-421d-91ea-1f65e593b8f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad59ce44-2g6wh\" (UID: \"dbb6aaa5-b66d-421d-91ea-1f65e593b8f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad59ce44-2g6wh" Nov 01 20:51:02.907532 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:02.907498 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vpvh\" (UniqueName: \"kubernetes.io/projected/dbb6aaa5-b66d-421d-91ea-1f65e593b8f5-kube-api-access-8vpvh\") pod \"maas-api-revoke-sub-ad59ce44-2g6wh\" (UID: \"dbb6aaa5-b66d-421d-91ea-1f65e593b8f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad59ce44-2g6wh" Nov 01 20:51:02.912378 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:02.912354 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad59ce44-2g6wh" Nov 01 20:51:03.048324 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:03.048272 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b5e8b28-mdkfk"] Nov 01 20:51:03.053763 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:03.053228 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b5e8b28-mdkfk"] Nov 01 20:51:03.301534 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:51:03.301496 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbb6aaa5_b66d_421d_91ea_1f65e593b8f5.slice/crio-cdd53ed983dd6c48e97092b9b140113b72b147df638f088ad2f9442755cd3c28 WatchSource:0}: Error finding container cdd53ed983dd6c48e97092b9b140113b72b147df638f088ad2f9442755cd3c28: Status 404 returned error can't find the container with id cdd53ed983dd6c48e97092b9b140113b72b147df638f088ad2f9442755cd3c28 Nov 01 20:51:03.301964 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:03.301580 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad59ce44-2g6wh"] Nov 01 20:51:03.374640 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:03.374597 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df220463-0233-42b1-a927-496af334cd6e" path="/var/lib/kubelet/pods/df220463-0233-42b1-a927-496af334cd6e/volumes" Nov 01 20:51:03.856458 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:03.856421 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8" Nov 01 20:51:03.856857 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:03.856839 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8" Nov 01 20:51:03.857548 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:03.857524 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8" Nov 01 20:51:04.047691 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:04.047656 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-306bdb4a-v6w48"] Nov 01 20:51:04.052439 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:04.052413 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-306bdb4a-v6w48"] Nov 01 20:51:04.320157 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:04.320063 4484 generic.go:356] "Generic (PLEG): container finished" podID="dbb6aaa5-b66d-421d-91ea-1f65e593b8f5" containerID="cfa800d7f517ac0a0794e6e2fb7e4b0c6b5eadf581dcaa3d2f093bf066aa4e8f" exitCode=0 Nov 01 20:51:04.320157 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:04.320141 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad59ce44-2g6wh" event={"ID":"dbb6aaa5-b66d-421d-91ea-1f65e593b8f5","Type":"ContainerDied","Data":"cfa800d7f517ac0a0794e6e2fb7e4b0c6b5eadf581dcaa3d2f093bf066aa4e8f"} Nov 01 20:51:04.320552 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:04.320191 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad59ce44-2g6wh" event={"ID":"dbb6aaa5-b66d-421d-91ea-1f65e593b8f5","Type":"ContainerStarted","Data":"cdd53ed983dd6c48e97092b9b140113b72b147df638f088ad2f9442755cd3c28"} Nov 01 20:51:05.046928 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:05.046893 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dcd387e-spkmz"] Nov 01 20:51:05.100781 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:05.100742 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dcd387e-spkmz"] Nov 01 20:51:05.374829 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:05.374799 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33f110a1-18ca-49a7-b518-14f75737f0da" path="/var/lib/kubelet/pods/33f110a1-18ca-49a7-b518-14f75737f0da/volumes" Nov 01 20:51:05.375356 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:05.375341 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8624507-afd3-4878-bbd7-dc386e2cd3ab" path="/var/lib/kubelet/pods/f8624507-afd3-4878-bbd7-dc386e2cd3ab/volumes" Nov 01 20:51:05.492798 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:05.492769 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad59ce44-2g6wh" Nov 01 20:51:05.591290 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:05.591191 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbb6aaa5-b66d-421d-91ea-1f65e593b8f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbb6aaa5-b66d-421d-91ea-1f65e593b8f5-maas-api-xks-ca\") pod \"dbb6aaa5-b66d-421d-91ea-1f65e593b8f5\" (UID: \"dbb6aaa5-b66d-421d-91ea-1f65e593b8f5\") " Nov 01 20:51:05.591290 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:05.591282 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbb6aaa5-b66d-421d-91ea-1f65e593b8f5-kube-api-access-8vpvh\" (UniqueName: \"kubernetes.io/projected/dbb6aaa5-b66d-421d-91ea-1f65e593b8f5-kube-api-access-8vpvh\") pod \"dbb6aaa5-b66d-421d-91ea-1f65e593b8f5\" (UID: \"dbb6aaa5-b66d-421d-91ea-1f65e593b8f5\") " Nov 01 20:51:05.591511 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:05.591335 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbb6aaa5-b66d-421d-91ea-1f65e593b8f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbb6aaa5-b66d-421d-91ea-1f65e593b8f5-maas-api-openshift-ca\") pod \"dbb6aaa5-b66d-421d-91ea-1f65e593b8f5\" (UID: \"dbb6aaa5-b66d-421d-91ea-1f65e593b8f5\") " Nov 01 20:51:05.591729 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:05.591703 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbb6aaa5-b66d-421d-91ea-1f65e593b8f5-maas-api-openshift-ca" pod "dbb6aaa5-b66d-421d-91ea-1f65e593b8f5" (UID: "dbb6aaa5-b66d-421d-91ea-1f65e593b8f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:51:05.591870 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:05.591850 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbb6aaa5-b66d-421d-91ea-1f65e593b8f5-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:05.593506 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:05.593483 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbb6aaa5-b66d-421d-91ea-1f65e593b8f5-kube-api-access-8vpvh" pod "dbb6aaa5-b66d-421d-91ea-1f65e593b8f5" (UID: "dbb6aaa5-b66d-421d-91ea-1f65e593b8f5"). InnerVolumeSpecName "kube-api-access-8vpvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:51:05.593640 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:05.593626 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbb6aaa5-b66d-421d-91ea-1f65e593b8f5-maas-api-xks-ca" pod "dbb6aaa5-b66d-421d-91ea-1f65e593b8f5" (UID: "dbb6aaa5-b66d-421d-91ea-1f65e593b8f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:51:05.693151 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:05.693115 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbb6aaa5-b66d-421d-91ea-1f65e593b8f5-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:05.693151 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:05.693144 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vpvh\" (UniqueName: \"kubernetes.io/projected/dbb6aaa5-b66d-421d-91ea-1f65e593b8f5-kube-api-access-8vpvh\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:06.221607 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:06.221486 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:51:06.337431 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:06.337398 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad59ce44-2g6wh" Nov 01 20:51:06.337431 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:06.337424 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad59ce44-2g6wh" event={"ID":"dbb6aaa5-b66d-421d-91ea-1f65e593b8f5","Type":"ContainerDied","Data":"cdd53ed983dd6c48e97092b9b140113b72b147df638f088ad2f9442755cd3c28"} Nov 01 20:51:06.337689 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:06.337472 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdd53ed983dd6c48e97092b9b140113b72b147df638f088ad2f9442755cd3c28" Nov 01 20:51:06.640375 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:06.640338 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa38f47b-45shn"] Nov 01 20:51:06.640983 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:06.640967 4484 cpu_manager.go:414] "containerMap: removing container" podUID="dbb6aaa5-b66d-421d-91ea-1f65e593b8f5" containerName="revoke-keys" Nov 01 20:51:06.640983 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:06.640982 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbb6aaa5-b66d-421d-91ea-1f65e593b8f5" containerName="revoke-keys" Nov 01 20:51:06.641158 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:06.641147 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbb6aaa5-b66d-421d-91ea-1f65e593b8f5" containerName="revoke-keys" Nov 01 20:51:06.645082 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:06.645043 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa38f47b-45shn" Nov 01 20:51:06.698432 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:06.698399 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:51:06.698432 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:06.698433 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:51:06.698701 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:06.698514 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:51:06.698701 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:06.698590 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 20:51:06.712487 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:06.712461 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e3f93a5-1229-4a1f-994a-e0698add77e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa38f47b-45shn\" (UID: \"6e3f93a5-1229-4a1f-994a-e0698add77e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa38f47b-45shn" Nov 01 20:51:06.712616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:06.712576 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e3f93a5-1229-4a1f-994a-e0698add77e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa38f47b-45shn\" (UID: \"6e3f93a5-1229-4a1f-994a-e0698add77e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa38f47b-45shn" Nov 01 20:51:06.712685 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:06.712669 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9549r\" (UniqueName: \"kubernetes.io/projected/6e3f93a5-1229-4a1f-994a-e0698add77e9-kube-api-access-9549r\") pod \"maas-api-revoke-sub-aa38f47b-45shn\" (UID: \"6e3f93a5-1229-4a1f-994a-e0698add77e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa38f47b-45shn" Nov 01 20:51:06.740592 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:06.740560 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa38f47b-45shn"] Nov 01 20:51:06.813808 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:06.813773 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e3f93a5-1229-4a1f-994a-e0698add77e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa38f47b-45shn\" (UID: \"6e3f93a5-1229-4a1f-994a-e0698add77e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa38f47b-45shn" Nov 01 20:51:06.813985 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:06.813869 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9549r\" (UniqueName: \"kubernetes.io/projected/6e3f93a5-1229-4a1f-994a-e0698add77e9-kube-api-access-9549r\") pod \"maas-api-revoke-sub-aa38f47b-45shn\" (UID: \"6e3f93a5-1229-4a1f-994a-e0698add77e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa38f47b-45shn" Nov 01 20:51:06.813985 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:06.813978 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e3f93a5-1229-4a1f-994a-e0698add77e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa38f47b-45shn\" (UID: \"6e3f93a5-1229-4a1f-994a-e0698add77e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa38f47b-45shn" Nov 01 20:51:06.814550 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:06.814524 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e3f93a5-1229-4a1f-994a-e0698add77e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa38f47b-45shn\" (UID: \"6e3f93a5-1229-4a1f-994a-e0698add77e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa38f47b-45shn" Nov 01 20:51:06.816282 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:06.816260 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e3f93a5-1229-4a1f-994a-e0698add77e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa38f47b-45shn\" (UID: \"6e3f93a5-1229-4a1f-994a-e0698add77e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa38f47b-45shn" Nov 01 20:51:06.871359 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:06.871326 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9549r\" (UniqueName: \"kubernetes.io/projected/6e3f93a5-1229-4a1f-994a-e0698add77e9-kube-api-access-9549r\") pod \"maas-api-revoke-sub-aa38f47b-45shn\" (UID: \"6e3f93a5-1229-4a1f-994a-e0698add77e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa38f47b-45shn" Nov 01 20:51:06.959608 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:06.959523 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa38f47b-45shn" Nov 01 20:51:07.336764 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:51:07.336726 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e3f93a5_1229_4a1f_994a_e0698add77e9.slice/crio-0c0204c9c443ad5514ed45304dc8cb7cf297f418982cb036d3d923dc3f71887a WatchSource:0}: Error finding container 0c0204c9c443ad5514ed45304dc8cb7cf297f418982cb036d3d923dc3f71887a: Status 404 returned error can't find the container with id 0c0204c9c443ad5514ed45304dc8cb7cf297f418982cb036d3d923dc3f71887a Nov 01 20:51:07.337312 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:07.337293 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa38f47b-45shn"] Nov 01 20:51:07.346228 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:07.346195 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa38f47b-45shn" event={"ID":"6e3f93a5-1229-4a1f-994a-e0698add77e9","Type":"ContainerStarted","Data":"0c0204c9c443ad5514ed45304dc8cb7cf297f418982cb036d3d923dc3f71887a"} Nov 01 20:51:08.055148 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:08.055109 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c05c922a-cgffx"] Nov 01 20:51:08.059476 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:08.059443 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c05c922a-cgffx"] Nov 01 20:51:08.356471 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:08.356427 4484 generic.go:356] "Generic (PLEG): container finished" podID="6e3f93a5-1229-4a1f-994a-e0698add77e9" containerID="1db32fba5019b9cfc7f0c5ba1cf95832bd3d6331f9ddac7a642000d0c99958c9" exitCode=0 Nov 01 20:51:08.356647 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:08.356469 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa38f47b-45shn" event={"ID":"6e3f93a5-1229-4a1f-994a-e0698add77e9","Type":"ContainerDied","Data":"1db32fba5019b9cfc7f0c5ba1cf95832bd3d6331f9ddac7a642000d0c99958c9"} Nov 01 20:51:08.995156 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:08.995119 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c325b215-sf5cd"] Nov 01 20:51:08.999584 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:08.999567 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c325b215-sf5cd" Nov 01 20:51:09.047565 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.047526 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4507914c-adb9-40c8-8b1e-7494f6cfde91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c325b215-sf5cd\" (UID: \"4507914c-adb9-40c8-8b1e-7494f6cfde91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c325b215-sf5cd" Nov 01 20:51:09.047736 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.047664 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp6zc\" (UniqueName: \"kubernetes.io/projected/4507914c-adb9-40c8-8b1e-7494f6cfde91-kube-api-access-pp6zc\") pod \"maas-api-revoke-sub-c325b215-sf5cd\" (UID: \"4507914c-adb9-40c8-8b1e-7494f6cfde91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c325b215-sf5cd" Nov 01 20:51:09.047736 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.047720 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4507914c-adb9-40c8-8b1e-7494f6cfde91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c325b215-sf5cd\" (UID: \"4507914c-adb9-40c8-8b1e-7494f6cfde91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c325b215-sf5cd" Nov 01 20:51:09.056613 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.056581 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c325b215-sf5cd"] Nov 01 20:51:09.149345 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.149303 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4507914c-adb9-40c8-8b1e-7494f6cfde91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c325b215-sf5cd\" (UID: \"4507914c-adb9-40c8-8b1e-7494f6cfde91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c325b215-sf5cd" Nov 01 20:51:09.149535 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.149409 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp6zc\" (UniqueName: \"kubernetes.io/projected/4507914c-adb9-40c8-8b1e-7494f6cfde91-kube-api-access-pp6zc\") pod \"maas-api-revoke-sub-c325b215-sf5cd\" (UID: \"4507914c-adb9-40c8-8b1e-7494f6cfde91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c325b215-sf5cd" Nov 01 20:51:09.149535 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.149447 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4507914c-adb9-40c8-8b1e-7494f6cfde91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c325b215-sf5cd\" (UID: \"4507914c-adb9-40c8-8b1e-7494f6cfde91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c325b215-sf5cd" Nov 01 20:51:09.150171 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.150142 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4507914c-adb9-40c8-8b1e-7494f6cfde91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c325b215-sf5cd\" (UID: \"4507914c-adb9-40c8-8b1e-7494f6cfde91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c325b215-sf5cd" Nov 01 20:51:09.151961 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.151934 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4507914c-adb9-40c8-8b1e-7494f6cfde91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c325b215-sf5cd\" (UID: \"4507914c-adb9-40c8-8b1e-7494f6cfde91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c325b215-sf5cd" Nov 01 20:51:09.206893 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.206857 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp6zc\" (UniqueName: \"kubernetes.io/projected/4507914c-adb9-40c8-8b1e-7494f6cfde91-kube-api-access-pp6zc\") pod \"maas-api-revoke-sub-c325b215-sf5cd\" (UID: \"4507914c-adb9-40c8-8b1e-7494f6cfde91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c325b215-sf5cd" Nov 01 20:51:09.313473 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.313384 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c325b215-sf5cd" Nov 01 20:51:09.375883 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.375844 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46c9582c-b073-4fb8-9b12-705307d2f82a" path="/var/lib/kubelet/pods/46c9582c-b073-4fb8-9b12-705307d2f82a/volumes" Nov 01 20:51:09.409858 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.409809 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw"] Nov 01 20:51:09.419258 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.418228 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw" Nov 01 20:51:09.454747 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.454712 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4b1073f-60a3-444a-8746-a3bbba04ee86-istio-data\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw\" (UID: \"f4b1073f-60a3-444a-8746-a3bbba04ee86\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw" Nov 01 20:51:09.454910 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.454756 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4b1073f-60a3-444a-8746-a3bbba04ee86-istio-token\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw\" (UID: \"f4b1073f-60a3-444a-8746-a3bbba04ee86\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw" Nov 01 20:51:09.454910 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.454814 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4b1073f-60a3-444a-8746-a3bbba04ee86-credential-socket\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw\" (UID: \"f4b1073f-60a3-444a-8746-a3bbba04ee86\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw" Nov 01 20:51:09.454910 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.454894 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4b1073f-60a3-444a-8746-a3bbba04ee86-istio-podinfo\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw\" (UID: \"f4b1073f-60a3-444a-8746-a3bbba04ee86\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw" Nov 01 20:51:09.455136 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.455015 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92n2s\" (UniqueName: \"kubernetes.io/projected/f4b1073f-60a3-444a-8746-a3bbba04ee86-kube-api-access-92n2s\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw\" (UID: \"f4b1073f-60a3-444a-8746-a3bbba04ee86\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw" Nov 01 20:51:09.455136 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.455068 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4b1073f-60a3-444a-8746-a3bbba04ee86-workload-socket\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw\" (UID: \"f4b1073f-60a3-444a-8746-a3bbba04ee86\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw" Nov 01 20:51:09.455136 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.455113 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4b1073f-60a3-444a-8746-a3bbba04ee86-istiod-ca-cert\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw\" (UID: \"f4b1073f-60a3-444a-8746-a3bbba04ee86\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw" Nov 01 20:51:09.455287 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.455197 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4b1073f-60a3-444a-8746-a3bbba04ee86-istio-envoy\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw\" (UID: \"f4b1073f-60a3-444a-8746-a3bbba04ee86\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw" Nov 01 20:51:09.455287 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.455240 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4b1073f-60a3-444a-8746-a3bbba04ee86-workload-certs\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw\" (UID: \"f4b1073f-60a3-444a-8746-a3bbba04ee86\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw" Nov 01 20:51:09.514017 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.513536 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw"] Nov 01 20:51:09.556719 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.556687 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4b1073f-60a3-444a-8746-a3bbba04ee86-istio-data\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw\" (UID: \"f4b1073f-60a3-444a-8746-a3bbba04ee86\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw" Nov 01 20:51:09.556889 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.556736 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4b1073f-60a3-444a-8746-a3bbba04ee86-istio-token\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw\" (UID: \"f4b1073f-60a3-444a-8746-a3bbba04ee86\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw" Nov 01 20:51:09.556889 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.556784 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4b1073f-60a3-444a-8746-a3bbba04ee86-credential-socket\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw\" (UID: \"f4b1073f-60a3-444a-8746-a3bbba04ee86\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw" Nov 01 20:51:09.556889 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.556855 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4b1073f-60a3-444a-8746-a3bbba04ee86-istio-podinfo\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw\" (UID: \"f4b1073f-60a3-444a-8746-a3bbba04ee86\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw" Nov 01 20:51:09.557064 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.556952 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92n2s\" (UniqueName: \"kubernetes.io/projected/f4b1073f-60a3-444a-8746-a3bbba04ee86-kube-api-access-92n2s\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw\" (UID: \"f4b1073f-60a3-444a-8746-a3bbba04ee86\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw" Nov 01 20:51:09.557064 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.557019 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4b1073f-60a3-444a-8746-a3bbba04ee86-workload-socket\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw\" (UID: \"f4b1073f-60a3-444a-8746-a3bbba04ee86\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw" Nov 01 20:51:09.557064 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.557051 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4b1073f-60a3-444a-8746-a3bbba04ee86-istiod-ca-cert\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw\" (UID: \"f4b1073f-60a3-444a-8746-a3bbba04ee86\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw" Nov 01 20:51:09.557226 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.557152 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4b1073f-60a3-444a-8746-a3bbba04ee86-istio-envoy\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw\" (UID: \"f4b1073f-60a3-444a-8746-a3bbba04ee86\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw" Nov 01 20:51:09.557226 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.557194 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4b1073f-60a3-444a-8746-a3bbba04ee86-workload-certs\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw\" (UID: \"f4b1073f-60a3-444a-8746-a3bbba04ee86\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw" Nov 01 20:51:09.557611 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.557587 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4b1073f-60a3-444a-8746-a3bbba04ee86-workload-certs\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw\" (UID: \"f4b1073f-60a3-444a-8746-a3bbba04ee86\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw" Nov 01 20:51:09.557820 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.557802 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4b1073f-60a3-444a-8746-a3bbba04ee86-istio-data\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw\" (UID: \"f4b1073f-60a3-444a-8746-a3bbba04ee86\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw" Nov 01 20:51:09.558318 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.558296 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4b1073f-60a3-444a-8746-a3bbba04ee86-credential-socket\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw\" (UID: \"f4b1073f-60a3-444a-8746-a3bbba04ee86\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw" Nov 01 20:51:09.558846 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.558820 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4b1073f-60a3-444a-8746-a3bbba04ee86-workload-socket\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw\" (UID: \"f4b1073f-60a3-444a-8746-a3bbba04ee86\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw" Nov 01 20:51:09.559636 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.559610 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4b1073f-60a3-444a-8746-a3bbba04ee86-istiod-ca-cert\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw\" (UID: \"f4b1073f-60a3-444a-8746-a3bbba04ee86\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw" Nov 01 20:51:09.560737 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.560719 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4b1073f-60a3-444a-8746-a3bbba04ee86-istio-podinfo\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw\" (UID: \"f4b1073f-60a3-444a-8746-a3bbba04ee86\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw" Nov 01 20:51:09.561516 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.561499 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4b1073f-60a3-444a-8746-a3bbba04ee86-istio-envoy\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw\" (UID: \"f4b1073f-60a3-444a-8746-a3bbba04ee86\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw" Nov 01 20:51:09.577562 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.577533 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa38f47b-45shn" Nov 01 20:51:09.615642 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.615608 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4b1073f-60a3-444a-8746-a3bbba04ee86-istio-token\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw\" (UID: \"f4b1073f-60a3-444a-8746-a3bbba04ee86\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw" Nov 01 20:51:09.616172 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.616155 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92n2s\" (UniqueName: \"kubernetes.io/projected/f4b1073f-60a3-444a-8746-a3bbba04ee86-kube-api-access-92n2s\") pod \"e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw\" (UID: \"f4b1073f-60a3-444a-8746-a3bbba04ee86\") " pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw" Nov 01 20:51:09.677020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.671614 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e3f93a5-1229-4a1f-994a-e0698add77e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e3f93a5-1229-4a1f-994a-e0698add77e9-maas-api-openshift-ca\") pod \"6e3f93a5-1229-4a1f-994a-e0698add77e9\" (UID: \"6e3f93a5-1229-4a1f-994a-e0698add77e9\") " Nov 01 20:51:09.677020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.671677 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e3f93a5-1229-4a1f-994a-e0698add77e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e3f93a5-1229-4a1f-994a-e0698add77e9-maas-api-xks-ca\") pod \"6e3f93a5-1229-4a1f-994a-e0698add77e9\" (UID: \"6e3f93a5-1229-4a1f-994a-e0698add77e9\") " Nov 01 20:51:09.677020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.671824 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e3f93a5-1229-4a1f-994a-e0698add77e9-kube-api-access-9549r\" (UniqueName: \"kubernetes.io/projected/6e3f93a5-1229-4a1f-994a-e0698add77e9-kube-api-access-9549r\") pod \"6e3f93a5-1229-4a1f-994a-e0698add77e9\" (UID: \"6e3f93a5-1229-4a1f-994a-e0698add77e9\") " Nov 01 20:51:09.741031 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.717227 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e3f93a5-1229-4a1f-994a-e0698add77e9-maas-api-openshift-ca" pod "6e3f93a5-1229-4a1f-994a-e0698add77e9" (UID: "6e3f93a5-1229-4a1f-994a-e0698add77e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:51:09.758369 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.757314 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw" Nov 01 20:51:09.795769 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.788957 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e3f93a5-1229-4a1f-994a-e0698add77e9-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:09.846048 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.828087 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e3f93a5-1229-4a1f-994a-e0698add77e9-kube-api-access-9549r" pod "6e3f93a5-1229-4a1f-994a-e0698add77e9" (UID: "6e3f93a5-1229-4a1f-994a-e0698add77e9"). InnerVolumeSpecName "kube-api-access-9549r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:51:09.846048 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.843084 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv"] Nov 01 20:51:09.846048 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.844265 4484 cpu_manager.go:414] "containerMap: removing container" podUID="6e3f93a5-1229-4a1f-994a-e0698add77e9" containerName="revoke-keys" Nov 01 20:51:09.846048 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.844290 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e3f93a5-1229-4a1f-994a-e0698add77e9" containerName="revoke-keys" Nov 01 20:51:09.846048 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.844464 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e3f93a5-1229-4a1f-994a-e0698add77e9" containerName="revoke-keys" Nov 01 20:51:09.853544 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.852842 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv" Nov 01 20:51:09.895250 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.891584 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-istiod-ca-cert\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv\" (UID: \"b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv" Nov 01 20:51:09.895250 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.891648 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-workload-socket\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv\" (UID: \"b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv" Nov 01 20:51:09.895250 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.891729 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-istio-token\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv\" (UID: \"b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv" Nov 01 20:51:09.895250 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.891758 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-istio-envoy\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv\" (UID: \"b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv" Nov 01 20:51:09.895250 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.891792 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgn4w\" (UniqueName: \"kubernetes.io/projected/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-kube-api-access-zgn4w\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv\" (UID: \"b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv" Nov 01 20:51:09.895250 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.891838 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-istio-data\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv\" (UID: \"b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv" Nov 01 20:51:09.895250 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.891882 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-istio-podinfo\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv\" (UID: \"b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv" Nov 01 20:51:09.895250 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.891913 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-workload-certs\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv\" (UID: \"b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv" Nov 01 20:51:09.895250 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.892053 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-credential-socket\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv\" (UID: \"b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv" Nov 01 20:51:09.895250 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.892189 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9549r\" (UniqueName: \"kubernetes.io/projected/6e3f93a5-1229-4a1f-994a-e0698add77e9-kube-api-access-9549r\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:09.942478 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.942438 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e3f93a5-1229-4a1f-994a-e0698add77e9-maas-api-xks-ca" pod "6e3f93a5-1229-4a1f-994a-e0698add77e9" (UID: "6e3f93a5-1229-4a1f-994a-e0698add77e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:51:09.960178 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.958360 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c325b215-sf5cd"] Nov 01 20:51:09.982112 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.969864 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz"] Nov 01 20:51:09.990539 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.987133 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv"] Nov 01 20:51:09.996774 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.994655 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz"] Nov 01 20:51:09.997291 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.988306 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz" Nov 01 20:51:09.999162 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.996922 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-credential-socket\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv\" (UID: \"b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv" Nov 01 20:51:09.999162 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.997966 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-istiod-ca-cert\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv\" (UID: \"b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv" Nov 01 20:51:09.999162 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.998040 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-workload-socket\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv\" (UID: \"b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv" Nov 01 20:51:09.999162 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.998130 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-istio-token\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv\" (UID: \"b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv" Nov 01 20:51:09.999162 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.998159 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-istio-envoy\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv\" (UID: \"b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv" Nov 01 20:51:09.999162 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.998190 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgn4w\" (UniqueName: \"kubernetes.io/projected/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-kube-api-access-zgn4w\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv\" (UID: \"b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv" Nov 01 20:51:09.999162 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.998240 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-istio-data\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv\" (UID: \"b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv" Nov 01 20:51:09.999162 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.998284 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-istio-podinfo\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv\" (UID: \"b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv" Nov 01 20:51:09.999162 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.998317 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-workload-certs\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv\" (UID: \"b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv" Nov 01 20:51:09.999162 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.998405 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e3f93a5-1229-4a1f-994a-e0698add77e9-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:09.999162 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.998747 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-workload-certs\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv\" (UID: \"b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv" Nov 01 20:51:09.999162 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:09.997374 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-credential-socket\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv\" (UID: \"b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv" Nov 01 20:51:10.001211 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.000269 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-istiod-ca-cert\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv\" (UID: \"b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv" Nov 01 20:51:10.001776 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.001723 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-istio-data\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv\" (UID: \"b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv" Nov 01 20:51:10.004355 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.004332 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-workload-socket\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv\" (UID: \"b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv" Nov 01 20:51:10.005841 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.005782 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-istio-envoy\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv\" (UID: \"b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv" Nov 01 20:51:10.046746 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.046677 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-istio-podinfo\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv\" (UID: \"b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv" Nov 01 20:51:10.116052 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.114500 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-istio-data\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz\" (UID: \"e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz" Nov 01 20:51:10.116052 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.114561 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-istio-podinfo\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz\" (UID: \"e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz" Nov 01 20:51:10.116052 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.114598 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-istio-envoy\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz\" (UID: \"e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz" Nov 01 20:51:10.116052 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.114729 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-credential-socket\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz\" (UID: \"e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz" Nov 01 20:51:10.116052 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.114767 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-istio-token\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz\" (UID: \"e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz" Nov 01 20:51:10.116052 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.114856 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-workload-socket\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz\" (UID: \"e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz" Nov 01 20:51:10.116052 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.114934 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-workload-certs\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz\" (UID: \"e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz" Nov 01 20:51:10.116052 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.115057 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-istiod-ca-cert\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz\" (UID: \"e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz" Nov 01 20:51:10.116052 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.115090 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx5np\" (UniqueName: \"kubernetes.io/projected/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-kube-api-access-gx5np\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz\" (UID: \"e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz" Nov 01 20:51:10.189043 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.155624 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-istio-token\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv\" (UID: \"b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv" Nov 01 20:51:10.242826 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.226283 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-credential-socket\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz\" (UID: \"e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz" Nov 01 20:51:10.242826 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.226347 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-istio-token\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz\" (UID: \"e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz" Nov 01 20:51:10.242826 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.235971 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-credential-socket\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz\" (UID: \"e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz" Nov 01 20:51:10.257014 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.249233 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-workload-socket\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz\" (UID: \"e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz" Nov 01 20:51:10.257014 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.249362 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-workload-certs\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz\" (UID: \"e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz" Nov 01 20:51:10.257014 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.249494 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-istiod-ca-cert\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz\" (UID: \"e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz" Nov 01 20:51:10.257014 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.249524 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx5np\" (UniqueName: \"kubernetes.io/projected/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-kube-api-access-gx5np\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz\" (UID: \"e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz" Nov 01 20:51:10.257014 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.249572 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-istio-data\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz\" (UID: \"e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz" Nov 01 20:51:10.257014 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.249620 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-istio-podinfo\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz\" (UID: \"e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz" Nov 01 20:51:10.257014 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.249667 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-istio-envoy\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz\" (UID: \"e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz" Nov 01 20:51:10.257014 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.250516 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-workload-certs\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz\" (UID: \"e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz" Nov 01 20:51:10.257014 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.251651 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-workload-socket\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz\" (UID: \"e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz" Nov 01 20:51:10.257014 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.252390 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-istio-data\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz\" (UID: \"e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz" Nov 01 20:51:10.273076 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.259641 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-istiod-ca-cert\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz\" (UID: \"e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz" Nov 01 20:51:10.347153 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.329102 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-istio-envoy\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz\" (UID: \"e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz" Nov 01 20:51:10.347153 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.337835 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgn4w\" (UniqueName: \"kubernetes.io/projected/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-kube-api-access-zgn4w\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv\" (UID: \"b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv" Nov 01 20:51:10.379271 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.354160 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-istio-podinfo\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz\" (UID: \"e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz" Nov 01 20:51:10.379271 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.374480 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf"] Nov 01 20:51:10.381889 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.381757 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf" Nov 01 20:51:10.396122 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.396090 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx5np\" (UniqueName: \"kubernetes.io/projected/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-kube-api-access-gx5np\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz\" (UID: \"e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz" Nov 01 20:51:10.420049 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.399548 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-istio-token\") pod \"e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz\" (UID: \"e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed\") " pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz" Nov 01 20:51:10.420049 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.400042 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz" Nov 01 20:51:10.420049 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.410419 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:51:10.420049 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.410563 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:51:10.420049 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.410672 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:51:10.421788 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.421750 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c325b215-sf5cd" event={"ID":"4507914c-adb9-40c8-8b1e-7494f6cfde91","Type":"ContainerStarted","Data":"f37dc3fda5b2c0e1d3847c98b12234eced0da183b3f53b1297ec753aff93c592"} Nov 01 20:51:10.459925 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.459895 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw"] Nov 01 20:51:10.472093 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.472056 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf"] Nov 01 20:51:10.486488 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.486441 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa38f47b-45shn" event={"ID":"6e3f93a5-1229-4a1f-994a-e0698add77e9","Type":"ContainerDied","Data":"0c0204c9c443ad5514ed45304dc8cb7cf297f418982cb036d3d923dc3f71887a"} Nov 01 20:51:10.487116 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.487095 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c0204c9c443ad5514ed45304dc8cb7cf297f418982cb036d3d923dc3f71887a" Nov 01 20:51:10.491328 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.467360 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhvhs\" (UniqueName: \"kubernetes.io/projected/ef2f968c-433d-49f8-8fc4-7c8551d647fb-kube-api-access-jhvhs\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf\" (UID: \"ef2f968c-433d-49f8-8fc4-7c8551d647fb\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf" Nov 01 20:51:10.491621 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.491596 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef2f968c-433d-49f8-8fc4-7c8551d647fb-workload-socket\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf\" (UID: \"ef2f968c-433d-49f8-8fc4-7c8551d647fb\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf" Nov 01 20:51:10.491951 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.491932 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef2f968c-433d-49f8-8fc4-7c8551d647fb-istio-token\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf\" (UID: \"ef2f968c-433d-49f8-8fc4-7c8551d647fb\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf" Nov 01 20:51:10.493012 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.492976 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef2f968c-433d-49f8-8fc4-7c8551d647fb-workload-certs\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf\" (UID: \"ef2f968c-433d-49f8-8fc4-7c8551d647fb\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf" Nov 01 20:51:10.493262 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.475509 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa38f47b-45shn" Nov 01 20:51:10.507731 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.507627 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv" Nov 01 20:51:10.508199 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.493233 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef2f968c-433d-49f8-8fc4-7c8551d647fb-istio-data\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf\" (UID: \"ef2f968c-433d-49f8-8fc4-7c8551d647fb\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf" Nov 01 20:51:10.508512 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.508492 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef2f968c-433d-49f8-8fc4-7c8551d647fb-istio-envoy\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf\" (UID: \"ef2f968c-433d-49f8-8fc4-7c8551d647fb\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf" Nov 01 20:51:10.508988 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.508965 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef2f968c-433d-49f8-8fc4-7c8551d647fb-istio-podinfo\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf\" (UID: \"ef2f968c-433d-49f8-8fc4-7c8551d647fb\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf" Nov 01 20:51:10.509291 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.509245 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef2f968c-433d-49f8-8fc4-7c8551d647fb-credential-socket\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf\" (UID: \"ef2f968c-433d-49f8-8fc4-7c8551d647fb\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf" Nov 01 20:51:10.509443 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.509423 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef2f968c-433d-49f8-8fc4-7c8551d647fb-istiod-ca-cert\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf\" (UID: \"ef2f968c-433d-49f8-8fc4-7c8551d647fb\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf" Nov 01 20:51:10.643079 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.616493 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef2f968c-433d-49f8-8fc4-7c8551d647fb-istio-token\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf\" (UID: \"ef2f968c-433d-49f8-8fc4-7c8551d647fb\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf" Nov 01 20:51:10.643079 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.616556 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef2f968c-433d-49f8-8fc4-7c8551d647fb-workload-certs\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf\" (UID: \"ef2f968c-433d-49f8-8fc4-7c8551d647fb\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf" Nov 01 20:51:10.643079 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.616596 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef2f968c-433d-49f8-8fc4-7c8551d647fb-istio-data\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf\" (UID: \"ef2f968c-433d-49f8-8fc4-7c8551d647fb\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf" Nov 01 20:51:10.643079 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.616660 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef2f968c-433d-49f8-8fc4-7c8551d647fb-istio-envoy\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf\" (UID: \"ef2f968c-433d-49f8-8fc4-7c8551d647fb\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf" Nov 01 20:51:10.643079 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.616858 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef2f968c-433d-49f8-8fc4-7c8551d647fb-istio-podinfo\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf\" (UID: \"ef2f968c-433d-49f8-8fc4-7c8551d647fb\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf" Nov 01 20:51:10.643079 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.616905 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef2f968c-433d-49f8-8fc4-7c8551d647fb-credential-socket\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf\" (UID: \"ef2f968c-433d-49f8-8fc4-7c8551d647fb\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf" Nov 01 20:51:10.643079 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.616935 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef2f968c-433d-49f8-8fc4-7c8551d647fb-istiod-ca-cert\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf\" (UID: \"ef2f968c-433d-49f8-8fc4-7c8551d647fb\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf" Nov 01 20:51:10.682021 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.656449 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhvhs\" (UniqueName: \"kubernetes.io/projected/ef2f968c-433d-49f8-8fc4-7c8551d647fb-kube-api-access-jhvhs\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf\" (UID: \"ef2f968c-433d-49f8-8fc4-7c8551d647fb\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf" Nov 01 20:51:10.690177 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.690142 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef2f968c-433d-49f8-8fc4-7c8551d647fb-workload-socket\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf\" (UID: \"ef2f968c-433d-49f8-8fc4-7c8551d647fb\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf" Nov 01 20:51:10.709175 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.671564 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef2f968c-433d-49f8-8fc4-7c8551d647fb-istio-data\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf\" (UID: \"ef2f968c-433d-49f8-8fc4-7c8551d647fb\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf" Nov 01 20:51:10.709483 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.685584 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef2f968c-433d-49f8-8fc4-7c8551d647fb-istiod-ca-cert\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf\" (UID: \"ef2f968c-433d-49f8-8fc4-7c8551d647fb\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf" Nov 01 20:51:10.709610 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.692881 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef2f968c-433d-49f8-8fc4-7c8551d647fb-istio-envoy\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf\" (UID: \"ef2f968c-433d-49f8-8fc4-7c8551d647fb\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf" Nov 01 20:51:10.709737 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.693157 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef2f968c-433d-49f8-8fc4-7c8551d647fb-credential-socket\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf\" (UID: \"ef2f968c-433d-49f8-8fc4-7c8551d647fb\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf" Nov 01 20:51:10.709824 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.671285 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef2f968c-433d-49f8-8fc4-7c8551d647fb-workload-certs\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf\" (UID: \"ef2f968c-433d-49f8-8fc4-7c8551d647fb\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf" Nov 01 20:51:10.709905 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.694900 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef2f968c-433d-49f8-8fc4-7c8551d647fb-workload-socket\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf\" (UID: \"ef2f968c-433d-49f8-8fc4-7c8551d647fb\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf" Nov 01 20:51:10.750118 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.750062 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef2f968c-433d-49f8-8fc4-7c8551d647fb-istio-podinfo\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf\" (UID: \"ef2f968c-433d-49f8-8fc4-7c8551d647fb\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf" Nov 01 20:51:10.795195 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.795092 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef2f968c-433d-49f8-8fc4-7c8551d647fb-istio-token\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf\" (UID: \"ef2f968c-433d-49f8-8fc4-7c8551d647fb\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf" Nov 01 20:51:10.812169 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.812125 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhvhs\" (UniqueName: \"kubernetes.io/projected/ef2f968c-433d-49f8-8fc4-7c8551d647fb-kube-api-access-jhvhs\") pod \"e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf\" (UID: \"ef2f968c-433d-49f8-8fc4-7c8551d647fb\") " pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf" Nov 01 20:51:10.958506 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:10.958202 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf" Nov 01 20:51:12.048605 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:12.048565 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:51:12.404020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:12.392374 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" podUID="9f19be29-2c84-4ae4-9674-acca4905f4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:12.404020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:12.392812 4484 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" podUID="70b4dacf-b3ac-46ec-99e8-a44f05587e7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:12.439090 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:12.422223 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" podUID="a65d9fd8-6193-4d03-9a3c-123f48b5dcdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:12.479439 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:51:12.479402 4484 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.102s" Nov 01 20:51:12.479656 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:12.479631 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c325b215-sf5cd" event={"ID":"4507914c-adb9-40c8-8b1e-7494f6cfde91","Type":"ContainerStarted","Data":"add0f520a069756283e725df7571c069a0d73b03c578da0b842cc67b4c39158e"} Nov 01 20:51:12.479823 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:12.479807 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw" event={"ID":"f4b1073f-60a3-444a-8746-a3bbba04ee86","Type":"ContainerStarted","Data":"a2b5fbfbd2122488f2433f61d5e30cc2e6450652a7b82943134887eacc4b4e61"} Nov 01 20:51:12.479940 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:12.479926 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw" event={"ID":"f4b1073f-60a3-444a-8746-a3bbba04ee86","Type":"ContainerStarted","Data":"7c99a32adf6a596e6da8019c850c8fd35926991890e09e41ff3c0bef84a00162"} Nov 01 20:51:12.480102 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:12.480084 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz"] Nov 01 20:51:12.480240 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:12.480227 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv"] Nov 01 20:51:12.623746 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:12.595081 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:51:12.623746 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:12.595207 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:51:12.623746 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:12.595255 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:51:12.819023 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:12.769603 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw" Nov 01 20:51:12.819023 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:12.769814 4484 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 01 20:51:12.819023 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:12.769893 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw" podUID="f4b1073f-60a3-444a-8746-a3bbba04ee86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 01 20:51:12.819023 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:12.818808 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c325b215-sf5cd" podStartSLOduration=4.795850028 podStartE2EDuration="4.795850028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:51:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:51:12.49455256 +0000 UTC m=+1223.272192696" watchObservedRunningTime="2026-11-01 20:51:12.795850028 +0000 UTC m=+1223.573490165" Nov 01 20:51:13.149077 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:13.148162 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" podUID="70b4dacf-b3ac-46ec-99e8-a44f05587e7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 01 20:51:13.513580 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:13.509315 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:51:13.513580 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:13.509430 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:51:13.513580 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:13.509473 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:51:13.611058 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:51:13.583407 4484 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4507914c_adb9_40c8_8b1e_7494f6cfde91.slice/crio-conmon-add0f520a069756283e725df7571c069a0d73b03c578da0b842cc67b4c39158e.scope\": RecentStats: unable to find data in memory cache]" Nov 01 20:51:13.689539 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:13.689486 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw" podStartSLOduration=4.68946708 podStartE2EDuration="4.68946708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:51:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:51:12.661910031 +0000 UTC m=+1223.439550167" watchObservedRunningTime="2026-11-01 20:51:13.68946708 +0000 UTC m=+1224.467107217" Nov 01 20:51:13.707069 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:13.700218 4484 generic.go:356] "Generic (PLEG): container finished" podID="4507914c-adb9-40c8-8b1e-7494f6cfde91" containerID="add0f520a069756283e725df7571c069a0d73b03c578da0b842cc67b4c39158e" exitCode=0 Nov 01 20:51:13.752342 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:13.752298 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:51:13.752721 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:13.752681 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:51:13.752880 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:13.752861 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:51:13.779422 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:13.779389 4484 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 01 20:51:13.779661 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:13.779633 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw" podUID="f4b1073f-60a3-444a-8746-a3bbba04ee86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 01 20:51:13.792097 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:13.792045 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv" event={"ID":"b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2","Type":"ContainerStarted","Data":"6081548bc3c04db34cc47a2118604924edd900bd104a412465b8c6d44b1832d6"} Nov 01 20:51:13.792395 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:13.792360 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf"] Nov 01 20:51:13.792563 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:13.792543 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz" event={"ID":"e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed","Type":"ContainerStarted","Data":"82293654e03aff0b5a607849770eb46daf3ed64eaddef271fa6351272cfea9e9"} Nov 01 20:51:13.792863 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:13.792818 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c325b215-sf5cd" event={"ID":"4507914c-adb9-40c8-8b1e-7494f6cfde91","Type":"ContainerDied","Data":"add0f520a069756283e725df7571c069a0d73b03c578da0b842cc67b4c39158e"} Nov 01 20:51:14.189119 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:14.189059 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49fe0f19-rpvxc"] Nov 01 20:51:14.195030 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:14.194986 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49fe0f19-rpvxc" Nov 01 20:51:14.249194 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:14.249147 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49fe0f19-rpvxc"] Nov 01 20:51:14.352593 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:14.337270 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c3df29e-158c-4dc1-aef8-849271bdf172-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49fe0f19-rpvxc\" (UID: \"0c3df29e-158c-4dc1-aef8-849271bdf172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49fe0f19-rpvxc" Nov 01 20:51:14.353083 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:14.353046 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c3df29e-158c-4dc1-aef8-849271bdf172-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49fe0f19-rpvxc\" (UID: \"0c3df29e-158c-4dc1-aef8-849271bdf172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49fe0f19-rpvxc" Nov 01 20:51:14.354638 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:14.354601 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkwxx\" (UniqueName: \"kubernetes.io/projected/0c3df29e-158c-4dc1-aef8-849271bdf172-kube-api-access-hkwxx\") pod \"maas-api-revoke-sub-49fe0f19-rpvxc\" (UID: \"0c3df29e-158c-4dc1-aef8-849271bdf172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49fe0f19-rpvxc" Nov 01 20:51:14.461340 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:14.461204 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkwxx\" (UniqueName: \"kubernetes.io/projected/0c3df29e-158c-4dc1-aef8-849271bdf172-kube-api-access-hkwxx\") pod \"maas-api-revoke-sub-49fe0f19-rpvxc\" (UID: \"0c3df29e-158c-4dc1-aef8-849271bdf172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49fe0f19-rpvxc" Nov 01 20:51:14.461514 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:14.461492 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c3df29e-158c-4dc1-aef8-849271bdf172-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49fe0f19-rpvxc\" (UID: \"0c3df29e-158c-4dc1-aef8-849271bdf172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49fe0f19-rpvxc" Nov 01 20:51:14.461818 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:14.461775 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c3df29e-158c-4dc1-aef8-849271bdf172-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49fe0f19-rpvxc\" (UID: \"0c3df29e-158c-4dc1-aef8-849271bdf172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49fe0f19-rpvxc" Nov 01 20:51:14.463449 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:14.463423 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c3df29e-158c-4dc1-aef8-849271bdf172-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49fe0f19-rpvxc\" (UID: \"0c3df29e-158c-4dc1-aef8-849271bdf172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49fe0f19-rpvxc" Nov 01 20:51:14.464627 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:14.464599 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c3df29e-158c-4dc1-aef8-849271bdf172-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49fe0f19-rpvxc\" (UID: \"0c3df29e-158c-4dc1-aef8-849271bdf172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49fe0f19-rpvxc" Nov 01 20:51:14.536030 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:14.534837 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkwxx\" (UniqueName: \"kubernetes.io/projected/0c3df29e-158c-4dc1-aef8-849271bdf172-kube-api-access-hkwxx\") pod \"maas-api-revoke-sub-49fe0f19-rpvxc\" (UID: \"0c3df29e-158c-4dc1-aef8-849271bdf172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49fe0f19-rpvxc" Nov 01 20:51:14.557472 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:14.556265 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49fe0f19-rpvxc" Nov 01 20:51:14.759947 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:14.759907 4484 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 01 20:51:14.760254 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:14.760226 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw" podUID="f4b1073f-60a3-444a-8746-a3bbba04ee86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 01 20:51:14.781536 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:14.781490 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv" event={"ID":"b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2","Type":"ContainerStarted","Data":"44b9d35619a9b318ba03543f0a159f06277941169a511aa5f57f7194abf9ddf4"} Nov 01 20:51:14.807104 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:14.797217 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf" event={"ID":"ef2f968c-433d-49f8-8fc4-7c8551d647fb","Type":"ContainerStarted","Data":"8eb9816d4eca1ec89f3adadcb08ce4caaa9cc08431c2dbe91e021610b5ae2efb"} Nov 01 20:51:14.807104 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:14.797287 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf" event={"ID":"ef2f968c-433d-49f8-8fc4-7c8551d647fb","Type":"ContainerStarted","Data":"05027bfb9625e224cf26f0a39f8b19ede4a13ceb3e6b012f6d53346a6a82c2e3"} Nov 01 20:51:14.807104 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:14.802621 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz" event={"ID":"e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed","Type":"ContainerStarted","Data":"dadc7b561815a1fdd3d45629dff96f4e18bc88a802cd7de7df5a693be1537f86"} Nov 01 20:51:14.956316 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:14.954170 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv" podStartSLOduration=5.954147912 podStartE2EDuration="5.954147912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:51:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:51:14.912331501 +0000 UTC m=+1225.689971637" watchObservedRunningTime="2026-11-01 20:51:14.954147912 +0000 UTC m=+1225.731788072" Nov 01 20:51:15.058147 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:14.979395 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf" Nov 01 20:51:15.442081 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:15.335083 4484 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 01 20:51:15.442081 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:15.335162 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf" podUID="ef2f968c-433d-49f8-8fc4-7c8551d647fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 01 20:51:15.442081 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:15.407231 4484 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 01 20:51:15.442081 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:15.407286 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz" podUID="e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 01 20:51:15.566968 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:15.566934 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bcd2422-wr76f"] Nov 01 20:51:15.567274 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:15.567247 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bcd2422-wr76f"] Nov 01 20:51:15.567431 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:15.567417 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz" Nov 01 20:51:15.567564 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:15.567552 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv" Nov 01 20:51:15.567759 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:15.567737 4484 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Nov 01 20:51:15.567893 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:15.567872 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv" podUID="b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Nov 01 20:51:15.722569 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:15.722444 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf" podStartSLOduration=5.722422302 podStartE2EDuration="5.722422302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:51:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:51:15.360648231 +0000 UTC m=+1226.138288366" watchObservedRunningTime="2026-11-01 20:51:15.722422302 +0000 UTC m=+1226.500062443" Nov 01 20:51:15.775116 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:15.774966 4484 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 01 20:51:15.775116 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:15.775046 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw" podUID="f4b1073f-60a3-444a-8746-a3bbba04ee86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 01 20:51:15.776198 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:15.776140 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz" podStartSLOduration=6.776118026 podStartE2EDuration="6.776118026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:51:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:51:15.500074119 +0000 UTC m=+1226.277714256" watchObservedRunningTime="2026-11-01 20:51:15.776118026 +0000 UTC m=+1226.553758162" Nov 01 20:51:16.012234 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:16.012183 4484 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 01 20:51:16.012498 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:16.012466 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf" podUID="ef2f968c-433d-49f8-8fc4-7c8551d647fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 01 20:51:16.036392 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:16.034140 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" podUID="fe8307db-264d-4b8a-b097-f180bc255a9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:16.054211 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:16.054173 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49fe0f19-rpvxc"] Nov 01 20:51:16.237024 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:51:16.227218 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 20:51:16.237024 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:16.227314 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" podUID="311f9699-6585-4a2c-8c20-e6b696b2f857" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:16.266105 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:16.257342 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" podUID="70b4dacf-b3ac-46ec-99e8-a44f05587e7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 01 20:51:16.266105 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:16.265333 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" podUID="17a9ac24-19cb-48c4-99b5-d74ce8aac0a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:16.377017 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:16.376120 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" podUID="a65d9fd8-6193-4d03-9a3c-123f48b5dcdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Nov 01 20:51:16.392024 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:16.382304 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" podUID="9f19be29-2c84-4ae4-9674-acca4905f4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:16.444022 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:16.427185 4484 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 01 20:51:16.444022 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:16.427254 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz" podUID="e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 01 20:51:16.517090 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:16.509776 4484 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Nov 01 20:51:16.517090 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:16.509843 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv" podUID="b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Nov 01 20:51:16.956065 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:16.938360 4484 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 01 20:51:16.956065 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:16.938428 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw" podUID="f4b1073f-60a3-444a-8746-a3bbba04ee86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 01 20:51:17.152789 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:17.019032 4484 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 01 20:51:17.152789 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:17.019108 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf" podUID="ef2f968c-433d-49f8-8fc4-7c8551d647fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 01 20:51:17.152789 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:17.045091 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" podUID="fe8307db-264d-4b8a-b097-f180bc255a9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:17.152789 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:17.071208 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" Nov 01 20:51:17.269650 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:17.198631 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" podUID="e175b298-2c23-45d4-8f9c-0c9b9f6aab92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:17.269650 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:17.248324 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:51:17.480064 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:17.285646 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" podUID="311f9699-6585-4a2c-8c20-e6b696b2f857" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 01 20:51:17.480064 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:17.298177 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" Nov 01 20:51:17.480064 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:17.298923 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" podUID="17a9ac24-19cb-48c4-99b5-d74ce8aac0a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:17.480064 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:17.299394 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" Nov 01 20:51:17.480064 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:17.412266 4484 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 01 20:51:17.480064 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:17.412324 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz" podUID="e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 01 20:51:17.480064 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:17.412394 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" podUID="a65d9fd8-6193-4d03-9a3c-123f48b5dcdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Nov 01 20:51:17.480064 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:17.412756 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" Nov 01 20:51:17.480064 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:17.413600 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" podUID="9f19be29-2c84-4ae4-9674-acca4905f4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 01 20:51:17.480064 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:17.414060 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" Nov 01 20:51:17.511710 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:17.506558 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" podUID="70b4dacf-b3ac-46ec-99e8-a44f05587e7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 01 20:51:17.511710 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:17.507069 4484 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" podUID="311f9699-6585-4a2c-8c20-e6b696b2f857" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": context deadline exceeded" Nov 01 20:51:17.511710 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:17.508062 4484 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Nov 01 20:51:17.511710 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:17.508098 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv" podUID="b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Nov 01 20:51:17.763019 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:17.760375 4484 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 01 20:51:17.763019 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:17.760433 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw" podUID="f4b1073f-60a3-444a-8746-a3bbba04ee86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 01 20:51:18.019082 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:17.994273 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c325b215-sf5cd" Nov 01 20:51:18.069022 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:18.045962 4484 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" podUID="e175b298-2c23-45d4-8f9c-0c9b9f6aab92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:18.225375 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:18.225315 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" podUID="e175b298-2c23-45d4-8f9c-0c9b9f6aab92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:18.363815 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:18.363765 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" podUID="17a9ac24-19cb-48c4-99b5-d74ce8aac0a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:18.395048 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:18.393622 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4507914c-adb9-40c8-8b1e-7494f6cfde91-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4507914c-adb9-40c8-8b1e-7494f6cfde91-maas-api-openshift-ca\") pod \"4507914c-adb9-40c8-8b1e-7494f6cfde91\" (UID: \"4507914c-adb9-40c8-8b1e-7494f6cfde91\") " Nov 01 20:51:18.395048 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:18.393732 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4507914c-adb9-40c8-8b1e-7494f6cfde91-kube-api-access-pp6zc\" (UniqueName: \"kubernetes.io/projected/4507914c-adb9-40c8-8b1e-7494f6cfde91-kube-api-access-pp6zc\") pod \"4507914c-adb9-40c8-8b1e-7494f6cfde91\" (UID: \"4507914c-adb9-40c8-8b1e-7494f6cfde91\") " Nov 01 20:51:18.395048 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:18.394244 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4507914c-adb9-40c8-8b1e-7494f6cfde91-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4507914c-adb9-40c8-8b1e-7494f6cfde91-maas-api-xks-ca\") pod \"4507914c-adb9-40c8-8b1e-7494f6cfde91\" (UID: \"4507914c-adb9-40c8-8b1e-7494f6cfde91\") " Nov 01 20:51:18.511334 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:18.511286 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" podUID="311f9699-6585-4a2c-8c20-e6b696b2f857" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 01 20:51:18.596052 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:18.586750 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4507914c-adb9-40c8-8b1e-7494f6cfde91-maas-api-openshift-ca" pod "4507914c-adb9-40c8-8b1e-7494f6cfde91" (UID: "4507914c-adb9-40c8-8b1e-7494f6cfde91"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:51:18.630791 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:18.630689 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" podUID="9f19be29-2c84-4ae4-9674-acca4905f4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:18.632924 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:18.632889 4484 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" podUID="17a9ac24-19cb-48c4-99b5-d74ce8aac0a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": context deadline exceeded" Nov 01 20:51:18.660804 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:18.660763 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" podUID="a65d9fd8-6193-4d03-9a3c-123f48b5dcdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Nov 01 20:51:18.689059 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:18.662939 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" podUID="fe8307db-264d-4b8a-b097-f180bc255a9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 01 20:51:18.695425 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:18.691717 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4507914c-adb9-40c8-8b1e-7494f6cfde91-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:18.791055 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:18.787769 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4507914c-adb9-40c8-8b1e-7494f6cfde91-kube-api-access-pp6zc" pod "4507914c-adb9-40c8-8b1e-7494f6cfde91" (UID: "4507914c-adb9-40c8-8b1e-7494f6cfde91"). InnerVolumeSpecName "kube-api-access-pp6zc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:51:18.791055 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:18.788024 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" podUID="70b4dacf-b3ac-46ec-99e8-a44f05587e7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:18.835280 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:18.799483 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4507914c-adb9-40c8-8b1e-7494f6cfde91-maas-api-xks-ca" pod "4507914c-adb9-40c8-8b1e-7494f6cfde91" (UID: "4507914c-adb9-40c8-8b1e-7494f6cfde91"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:51:18.865419 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:18.865360 4484 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 01 20:51:18.865745 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:18.865698 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf" podUID="ef2f968c-433d-49f8-8fc4-7c8551d647fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 01 20:51:18.867557 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:18.867531 4484 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Nov 01 20:51:18.867767 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:18.867742 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv" podUID="b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Nov 01 20:51:18.868595 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:18.868573 4484 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 01 20:51:18.868835 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:18.868811 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw" podUID="f4b1073f-60a3-444a-8746-a3bbba04ee86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 01 20:51:19.013298 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:18.982489 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4507914c-adb9-40c8-8b1e-7494f6cfde91-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:19.013298 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:18.982810 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp6zc\" (UniqueName: \"kubernetes.io/projected/4507914c-adb9-40c8-8b1e-7494f6cfde91-kube-api-access-pp6zc\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:19.077022 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:19.068296 4484 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 01 20:51:19.077022 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:19.069071 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz" podUID="e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 01 20:51:19.077022 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:19.070395 4484 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 01 20:51:19.077022 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:19.070439 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf" podUID="ef2f968c-433d-49f8-8fc4-7c8551d647fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 01 20:51:19.269092 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:19.249015 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb324776-9df4-47c0-911c-473b3e74f259" path="/var/lib/kubelet/pods/bb324776-9df4-47c0-911c-473b3e74f259/volumes" Nov 01 20:51:19.350121 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:51:19.350075 4484 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.934s" Nov 01 20:51:19.350828 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:19.350798 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" Nov 01 20:51:19.376046 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:19.376015 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" Nov 01 20:51:19.403024 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:19.377898 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" podUID="17a9ac24-19cb-48c4-99b5-d74ce8aac0a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:19.529082 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:19.520646 4484 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Nov 01 20:51:19.541136 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:19.541081 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv" podUID="b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Nov 01 20:51:19.677745 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:19.677699 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" podUID="fe8307db-264d-4b8a-b097-f180bc255a9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:19.697697 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:19.697644 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" podUID="9f19be29-2c84-4ae4-9674-acca4905f4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 01 20:51:19.786117 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:19.785987 4484 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 01 20:51:19.786355 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:19.786327 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw" podUID="f4b1073f-60a3-444a-8746-a3bbba04ee86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 01 20:51:19.873008 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:19.846719 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" podUID="70b4dacf-b3ac-46ec-99e8-a44f05587e7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 01 20:51:19.947363 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:19.947323 4484 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 01 20:51:19.973078 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:19.947625 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz" podUID="e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 01 20:51:19.981677 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:19.981641 4484 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 01 20:51:19.982205 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:19.982165 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf" podUID="ef2f968c-433d-49f8-8fc4-7c8551d647fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 01 20:51:20.149216 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:20.120888 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" podUID="311f9699-6585-4a2c-8c20-e6b696b2f857" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 01 20:51:20.201446 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:20.201404 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" podUID="a65d9fd8-6193-4d03-9a3c-123f48b5dcdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:20.250020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:20.245695 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c325b215-sf5cd" Nov 01 20:51:20.405179 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:20.405074 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" podUID="17a9ac24-19cb-48c4-99b5-d74ce8aac0a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:20.530658 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:20.512392 4484 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 01 20:51:20.530658 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:20.512454 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz" podUID="e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 01 20:51:20.530658 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:20.516796 4484 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Nov 01 20:51:20.530658 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:20.516846 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv" podUID="b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Nov 01 20:51:20.530658 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:51:20.520123 4484 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.144s" Nov 01 20:51:20.530658 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:20.520156 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49fe0f19-rpvxc" event={"ID":"0c3df29e-158c-4dc1-aef8-849271bdf172","Type":"ContainerStarted","Data":"33dc01fbe844a40d600a22822197b84332afe133620b564b4e4d69782c12529a"} Nov 01 20:51:20.530658 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:20.520246 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" Nov 01 20:51:20.530658 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:20.520309 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c325b215-sf5cd" event={"ID":"4507914c-adb9-40c8-8b1e-7494f6cfde91","Type":"ContainerDied","Data":"f37dc3fda5b2c0e1d3847c98b12234eced0da183b3f53b1297ec753aff93c592"} Nov 01 20:51:20.530658 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:20.520334 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f37dc3fda5b2c0e1d3847c98b12234eced0da183b3f53b1297ec753aff93c592" Nov 01 20:51:20.530658 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:20.520353 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49fe0f19-rpvxc" event={"ID":"0c3df29e-158c-4dc1-aef8-849271bdf172","Type":"ContainerStarted","Data":"51a4e7edbce4d63e0c709cbd8e483cca4707110046613ebab1cac1bb5fb931c9"} Nov 01 20:51:20.530658 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:20.520389 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" Nov 01 20:51:20.530658 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:20.520419 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" Nov 01 20:51:20.530658 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:20.520444 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" Nov 01 20:51:20.530658 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:20.520458 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" Nov 01 20:51:20.774185 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:20.682593 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" podUID="fe8307db-264d-4b8a-b097-f180bc255a9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:20.774185 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:20.723321 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" podUID="9f19be29-2c84-4ae4-9674-acca4905f4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:20.774185 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:20.723741 4484 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" podUID="9f19be29-2c84-4ae4-9674-acca4905f4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:20.796067 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:20.793532 4484 patch_prober.go:29] interesting pod/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 01 20:51:20.796067 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:20.793595 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw" podUID="f4b1073f-60a3-444a-8746-a3bbba04ee86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 01 20:51:20.887021 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:20.878137 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" podUID="70b4dacf-b3ac-46ec-99e8-a44f05587e7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:20.975020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:20.974238 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49fe0f19-rpvxc" podStartSLOduration=6.97421865 podStartE2EDuration="6.97421865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:51:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:51:20.898407751 +0000 UTC m=+1231.676047905" watchObservedRunningTime="2026-11-01 20:51:20.97421865 +0000 UTC m=+1231.751858786" Nov 01 20:51:20.996595 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:20.996249 4484 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 01 20:51:20.996595 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:20.996307 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf" podUID="ef2f968c-433d-49f8-8fc4-7c8551d647fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 01 20:51:21.002739 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:21.002472 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" Nov 01 20:51:21.422806 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:21.422771 4484 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 01 20:51:21.423083 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:21.423050 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz" podUID="e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 01 20:51:21.435753 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:21.435708 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" podUID="17a9ac24-19cb-48c4-99b5-d74ce8aac0a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:21.558044 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:21.557985 4484 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Nov 01 20:51:21.558823 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:21.558792 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv" podUID="b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Nov 01 20:51:21.654967 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:21.654928 4484 generic.go:356] "Generic (PLEG): container finished" podID="0c3df29e-158c-4dc1-aef8-849271bdf172" containerID="51a4e7edbce4d63e0c709cbd8e483cca4707110046613ebab1cac1bb5fb931c9" exitCode=0 Nov 01 20:51:21.978587 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:21.978547 4484 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 01 20:51:21.978880 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:21.978848 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf" podUID="ef2f968c-433d-49f8-8fc4-7c8551d647fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 01 20:51:22.065767 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:22.065721 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:51:22.418023 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:22.416460 4484 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 01 20:51:22.418023 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:22.416520 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz" podUID="e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 01 20:51:22.513233 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:22.508829 4484 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Nov 01 20:51:22.513233 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:22.508886 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv" podUID="b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Nov 01 20:51:22.538324 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:51:22.538293 4484 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.169s" Nov 01 20:51:22.538557 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:22.538540 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54db2327-5gml4"] Nov 01 20:51:22.538705 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:22.538683 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49fe0f19-rpvxc" event={"ID":"0c3df29e-158c-4dc1-aef8-849271bdf172","Type":"ContainerDied","Data":"51a4e7edbce4d63e0c709cbd8e483cca4707110046613ebab1cac1bb5fb931c9"} Nov 01 20:51:22.538928 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:22.538905 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54db2327-5gml4"] Nov 01 20:51:22.539167 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:22.539155 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" Nov 01 20:51:22.539268 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:22.539260 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" Nov 01 20:51:22.539638 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:22.539616 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" Nov 01 20:51:22.539705 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:22.539668 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9279a206-p7mq2"] Nov 01 20:51:22.539705 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:22.539696 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9279a206-p7mq2"] Nov 01 20:51:22.759309 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:22.757662 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw" Nov 01 20:51:22.759309 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:22.759215 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw" Nov 01 20:51:22.768532 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:22.768455 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw" Nov 01 20:51:22.981073 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:22.977842 4484 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 01 20:51:22.981073 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:22.977907 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf" podUID="ef2f968c-433d-49f8-8fc4-7c8551d647fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 01 20:51:23.068366 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:23.068338 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49fe0f19-rpvxc" Nov 01 20:51:23.159018 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:23.154644 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c3df29e-158c-4dc1-aef8-849271bdf172-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c3df29e-158c-4dc1-aef8-849271bdf172-maas-api-xks-ca\") pod \"0c3df29e-158c-4dc1-aef8-849271bdf172\" (UID: \"0c3df29e-158c-4dc1-aef8-849271bdf172\") " Nov 01 20:51:23.159018 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:23.154887 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c3df29e-158c-4dc1-aef8-849271bdf172-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c3df29e-158c-4dc1-aef8-849271bdf172-maas-api-openshift-ca\") pod \"0c3df29e-158c-4dc1-aef8-849271bdf172\" (UID: \"0c3df29e-158c-4dc1-aef8-849271bdf172\") " Nov 01 20:51:23.159018 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:23.154953 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c3df29e-158c-4dc1-aef8-849271bdf172-kube-api-access-hkwxx\" (UniqueName: \"kubernetes.io/projected/0c3df29e-158c-4dc1-aef8-849271bdf172-kube-api-access-hkwxx\") pod \"0c3df29e-158c-4dc1-aef8-849271bdf172\" (UID: \"0c3df29e-158c-4dc1-aef8-849271bdf172\") " Nov 01 20:51:23.159018 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:23.156655 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c3df29e-158c-4dc1-aef8-849271bdf172-maas-api-openshift-ca" pod "0c3df29e-158c-4dc1-aef8-849271bdf172" (UID: "0c3df29e-158c-4dc1-aef8-849271bdf172"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:51:23.178857 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:23.159357 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c3df29e-158c-4dc1-aef8-849271bdf172-kube-api-access-hkwxx" pod "0c3df29e-158c-4dc1-aef8-849271bdf172" (UID: "0c3df29e-158c-4dc1-aef8-849271bdf172"). InnerVolumeSpecName "kube-api-access-hkwxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:51:23.241342 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:23.241278 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c3df29e-158c-4dc1-aef8-849271bdf172-maas-api-xks-ca" pod "0c3df29e-158c-4dc1-aef8-849271bdf172" (UID: "0c3df29e-158c-4dc1-aef8-849271bdf172"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:51:23.259754 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:23.259711 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c3df29e-158c-4dc1-aef8-849271bdf172-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:23.259754 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:23.259756 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c3df29e-158c-4dc1-aef8-849271bdf172-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:23.260072 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:23.259774 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkwxx\" (UniqueName: \"kubernetes.io/projected/0c3df29e-158c-4dc1-aef8-849271bdf172-kube-api-access-hkwxx\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:23.376336 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:23.376292 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bc83511-e182-4a03-97d1-376fed073d0e" path="/var/lib/kubelet/pods/8bc83511-e182-4a03-97d1-376fed073d0e/volumes" Nov 01 20:51:23.376843 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:23.376824 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be70021e-7901-455b-985a-78fb5f4b66e7" path="/var/lib/kubelet/pods/be70021e-7901-455b-985a-78fb5f4b66e7/volumes" Nov 01 20:51:23.410088 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:23.410059 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz" Nov 01 20:51:23.410603 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:23.410584 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz" Nov 01 20:51:23.411549 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:23.411518 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz" Nov 01 20:51:23.670904 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:23.665273 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv" Nov 01 20:51:23.670904 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:23.668973 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv" Nov 01 20:51:23.681118 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:23.671366 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv" Nov 01 20:51:23.722494 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:23.722463 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49fe0f19-rpvxc" Nov 01 20:51:23.722702 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:23.722526 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49fe0f19-rpvxc" event={"ID":"0c3df29e-158c-4dc1-aef8-849271bdf172","Type":"ContainerDied","Data":"33dc01fbe844a40d600a22822197b84332afe133620b564b4e4d69782c12529a"} Nov 01 20:51:23.722702 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:23.722566 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33dc01fbe844a40d600a22822197b84332afe133620b564b4e4d69782c12529a" Nov 01 20:51:24.195920 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:24.195886 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf" Nov 01 20:51:24.196944 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:24.196626 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf" Nov 01 20:51:24.198305 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:24.198272 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf" Nov 01 20:51:25.074425 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:25.074388 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-422d19ef-wqltd"] Nov 01 20:51:25.075119 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:25.075098 4484 cpu_manager.go:414] "containerMap: removing container" podUID="4507914c-adb9-40c8-8b1e-7494f6cfde91" containerName="revoke-keys" Nov 01 20:51:25.075119 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:25.075114 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4507914c-adb9-40c8-8b1e-7494f6cfde91" containerName="revoke-keys" Nov 01 20:51:25.075290 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:25.075146 4484 cpu_manager.go:414] "containerMap: removing container" podUID="0c3df29e-158c-4dc1-aef8-849271bdf172" containerName="revoke-keys" Nov 01 20:51:25.075290 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:25.075152 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c3df29e-158c-4dc1-aef8-849271bdf172" containerName="revoke-keys" Nov 01 20:51:25.075290 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:25.075259 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="4507914c-adb9-40c8-8b1e-7494f6cfde91" containerName="revoke-keys" Nov 01 20:51:25.075290 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:25.075274 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c3df29e-158c-4dc1-aef8-849271bdf172" containerName="revoke-keys" Nov 01 20:51:25.080105 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:25.080081 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-422d19ef-wqltd" Nov 01 20:51:25.103777 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:25.103739 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5b0b65e-dce0-4cf8-a6d1-10d3836e6b83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-422d19ef-wqltd\" (UID: \"d5b0b65e-dce0-4cf8-a6d1-10d3836e6b83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-422d19ef-wqltd" Nov 01 20:51:25.103958 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:25.103937 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5b0b65e-dce0-4cf8-a6d1-10d3836e6b83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-422d19ef-wqltd\" (UID: \"d5b0b65e-dce0-4cf8-a6d1-10d3836e6b83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-422d19ef-wqltd" Nov 01 20:51:25.104110 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:25.104091 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grphz\" (UniqueName: \"kubernetes.io/projected/d5b0b65e-dce0-4cf8-a6d1-10d3836e6b83-kube-api-access-grphz\") pod \"maas-api-revoke-sub-422d19ef-wqltd\" (UID: \"d5b0b65e-dce0-4cf8-a6d1-10d3836e6b83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-422d19ef-wqltd" Nov 01 20:51:25.136908 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:25.136368 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:51:25.136908 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:25.136399 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:51:25.136908 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:25.136444 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 20:51:25.137204 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:25.137135 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:51:25.144356 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:25.144325 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-422d19ef-wqltd"] Nov 01 20:51:25.205724 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:25.205691 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5b0b65e-dce0-4cf8-a6d1-10d3836e6b83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-422d19ef-wqltd\" (UID: \"d5b0b65e-dce0-4cf8-a6d1-10d3836e6b83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-422d19ef-wqltd" Nov 01 20:51:25.206216 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:25.205775 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5b0b65e-dce0-4cf8-a6d1-10d3836e6b83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-422d19ef-wqltd\" (UID: \"d5b0b65e-dce0-4cf8-a6d1-10d3836e6b83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-422d19ef-wqltd" Nov 01 20:51:25.206216 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:25.205842 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grphz\" (UniqueName: \"kubernetes.io/projected/d5b0b65e-dce0-4cf8-a6d1-10d3836e6b83-kube-api-access-grphz\") pod \"maas-api-revoke-sub-422d19ef-wqltd\" (UID: \"d5b0b65e-dce0-4cf8-a6d1-10d3836e6b83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-422d19ef-wqltd" Nov 01 20:51:25.206471 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:25.206450 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5b0b65e-dce0-4cf8-a6d1-10d3836e6b83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-422d19ef-wqltd\" (UID: \"d5b0b65e-dce0-4cf8-a6d1-10d3836e6b83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-422d19ef-wqltd" Nov 01 20:51:25.208473 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:25.208452 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5b0b65e-dce0-4cf8-a6d1-10d3836e6b83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-422d19ef-wqltd\" (UID: \"d5b0b65e-dce0-4cf8-a6d1-10d3836e6b83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-422d19ef-wqltd" Nov 01 20:51:25.263336 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:25.263299 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grphz\" (UniqueName: \"kubernetes.io/projected/d5b0b65e-dce0-4cf8-a6d1-10d3836e6b83-kube-api-access-grphz\") pod \"maas-api-revoke-sub-422d19ef-wqltd\" (UID: \"d5b0b65e-dce0-4cf8-a6d1-10d3836e6b83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-422d19ef-wqltd" Nov 01 20:51:25.398551 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:25.398457 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-422d19ef-wqltd" Nov 01 20:51:25.781923 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:25.781899 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-422d19ef-wqltd"] Nov 01 20:51:26.049867 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.049781 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9eae9a85-c6hsl"] Nov 01 20:51:26.054191 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.054162 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9eae9a85-c6hsl"] Nov 01 20:51:26.143180 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.143150 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn"] Nov 01 20:51:26.147890 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.147870 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn" Nov 01 20:51:26.169392 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.169360 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw"] Nov 01 20:51:26.174055 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.174026 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw" Nov 01 20:51:26.208196 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.208167 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn"] Nov 01 20:51:26.237818 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.237788 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-istio-envoy\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn\" (UID: \"cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn" Nov 01 20:51:26.237985 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.237836 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-credential-socket\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn\" (UID: \"cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn" Nov 01 20:51:26.237985 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.237883 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad03483e-106e-4de1-b896-d71bff38f81d-workload-certs\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw\" (UID: \"ad03483e-106e-4de1-b896-d71bff38f81d\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw" Nov 01 20:51:26.238169 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.238005 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad03483e-106e-4de1-b896-d71bff38f81d-istio-token\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw\" (UID: \"ad03483e-106e-4de1-b896-d71bff38f81d\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw" Nov 01 20:51:26.238169 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.238067 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fgvd\" (UniqueName: \"kubernetes.io/projected/ad03483e-106e-4de1-b896-d71bff38f81d-kube-api-access-8fgvd\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw\" (UID: \"ad03483e-106e-4de1-b896-d71bff38f81d\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw" Nov 01 20:51:26.238276 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.238192 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad03483e-106e-4de1-b896-d71bff38f81d-istiod-ca-cert\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw\" (UID: \"ad03483e-106e-4de1-b896-d71bff38f81d\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw" Nov 01 20:51:26.238276 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.238245 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad03483e-106e-4de1-b896-d71bff38f81d-workload-socket\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw\" (UID: \"ad03483e-106e-4de1-b896-d71bff38f81d\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw" Nov 01 20:51:26.238377 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.238318 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad03483e-106e-4de1-b896-d71bff38f81d-istio-envoy\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw\" (UID: \"ad03483e-106e-4de1-b896-d71bff38f81d\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw" Nov 01 20:51:26.238377 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.238338 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-istiod-ca-cert\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn\" (UID: \"cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn" Nov 01 20:51:26.238561 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.238381 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-workload-certs\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn\" (UID: \"cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn" Nov 01 20:51:26.238561 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.238427 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgstf\" (UniqueName: \"kubernetes.io/projected/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-kube-api-access-bgstf\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn\" (UID: \"cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn" Nov 01 20:51:26.238561 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.238500 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad03483e-106e-4de1-b896-d71bff38f81d-credential-socket\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw\" (UID: \"ad03483e-106e-4de1-b896-d71bff38f81d\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw" Nov 01 20:51:26.238561 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.238527 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-workload-socket\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn\" (UID: \"cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn" Nov 01 20:51:26.238561 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.238543 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-istio-token\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn\" (UID: \"cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn" Nov 01 20:51:26.238826 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.238578 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-istio-data\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn\" (UID: \"cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn" Nov 01 20:51:26.238826 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.238634 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad03483e-106e-4de1-b896-d71bff38f81d-istio-data\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw\" (UID: \"ad03483e-106e-4de1-b896-d71bff38f81d\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw" Nov 01 20:51:26.238826 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.238650 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-istio-podinfo\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn\" (UID: \"cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn" Nov 01 20:51:26.238826 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.238675 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad03483e-106e-4de1-b896-d71bff38f81d-istio-podinfo\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw\" (UID: \"ad03483e-106e-4de1-b896-d71bff38f81d\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw" Nov 01 20:51:26.261622 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.261583 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw"] Nov 01 20:51:26.340232 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.340148 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-credential-socket\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn\" (UID: \"cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn" Nov 01 20:51:26.340232 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.340197 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad03483e-106e-4de1-b896-d71bff38f81d-workload-certs\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw\" (UID: \"ad03483e-106e-4de1-b896-d71bff38f81d\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw" Nov 01 20:51:26.340453 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.340307 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad03483e-106e-4de1-b896-d71bff38f81d-istio-token\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw\" (UID: \"ad03483e-106e-4de1-b896-d71bff38f81d\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw" Nov 01 20:51:26.340453 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.340366 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fgvd\" (UniqueName: \"kubernetes.io/projected/ad03483e-106e-4de1-b896-d71bff38f81d-kube-api-access-8fgvd\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw\" (UID: \"ad03483e-106e-4de1-b896-d71bff38f81d\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw" Nov 01 20:51:26.340453 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.340397 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad03483e-106e-4de1-b896-d71bff38f81d-istiod-ca-cert\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw\" (UID: \"ad03483e-106e-4de1-b896-d71bff38f81d\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw" Nov 01 20:51:26.340453 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.340419 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad03483e-106e-4de1-b896-d71bff38f81d-workload-socket\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw\" (UID: \"ad03483e-106e-4de1-b896-d71bff38f81d\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw" Nov 01 20:51:26.340637 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.340461 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad03483e-106e-4de1-b896-d71bff38f81d-istio-envoy\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw\" (UID: \"ad03483e-106e-4de1-b896-d71bff38f81d\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw" Nov 01 20:51:26.340637 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.340485 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-istiod-ca-cert\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn\" (UID: \"cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn" Nov 01 20:51:26.340637 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.340538 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-workload-certs\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn\" (UID: \"cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn" Nov 01 20:51:26.340637 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.340583 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgstf\" (UniqueName: \"kubernetes.io/projected/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-kube-api-access-bgstf\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn\" (UID: \"cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn" Nov 01 20:51:26.340637 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.340608 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-credential-socket\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn\" (UID: \"cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn" Nov 01 20:51:26.340886 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.340645 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad03483e-106e-4de1-b896-d71bff38f81d-credential-socket\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw\" (UID: \"ad03483e-106e-4de1-b896-d71bff38f81d\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw" Nov 01 20:51:26.340886 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.340661 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad03483e-106e-4de1-b896-d71bff38f81d-workload-certs\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw\" (UID: \"ad03483e-106e-4de1-b896-d71bff38f81d\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw" Nov 01 20:51:26.340886 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.340689 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-workload-socket\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn\" (UID: \"cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn" Nov 01 20:51:26.340886 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.340716 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-istio-token\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn\" (UID: \"cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn" Nov 01 20:51:26.340886 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.340777 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-istio-data\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn\" (UID: \"cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn" Nov 01 20:51:26.340886 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.340842 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad03483e-106e-4de1-b896-d71bff38f81d-workload-socket\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw\" (UID: \"ad03483e-106e-4de1-b896-d71bff38f81d\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw" Nov 01 20:51:26.340886 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.340863 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad03483e-106e-4de1-b896-d71bff38f81d-istio-data\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw\" (UID: \"ad03483e-106e-4de1-b896-d71bff38f81d\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw" Nov 01 20:51:26.340886 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.340867 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-workload-certs\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn\" (UID: \"cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn" Nov 01 20:51:26.340886 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.340888 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-istio-podinfo\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn\" (UID: \"cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn" Nov 01 20:51:26.341498 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.340928 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad03483e-106e-4de1-b896-d71bff38f81d-istio-podinfo\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw\" (UID: \"ad03483e-106e-4de1-b896-d71bff38f81d\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw" Nov 01 20:51:26.341498 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.341071 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-istio-envoy\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn\" (UID: \"cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn" Nov 01 20:51:26.341498 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.341141 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad03483e-106e-4de1-b896-d71bff38f81d-credential-socket\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw\" (UID: \"ad03483e-106e-4de1-b896-d71bff38f81d\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw" Nov 01 20:51:26.341498 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.341236 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-workload-socket\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn\" (UID: \"cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn" Nov 01 20:51:26.341498 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.341270 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad03483e-106e-4de1-b896-d71bff38f81d-istiod-ca-cert\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw\" (UID: \"ad03483e-106e-4de1-b896-d71bff38f81d\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw" Nov 01 20:51:26.341498 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.341279 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-istio-data\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn\" (UID: \"cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn" Nov 01 20:51:26.341498 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.341294 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad03483e-106e-4de1-b896-d71bff38f81d-istio-data\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw\" (UID: \"ad03483e-106e-4de1-b896-d71bff38f81d\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw" Nov 01 20:51:26.341498 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.341395 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-istiod-ca-cert\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn\" (UID: \"cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn" Nov 01 20:51:26.343404 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.343382 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad03483e-106e-4de1-b896-d71bff38f81d-istio-envoy\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw\" (UID: \"ad03483e-106e-4de1-b896-d71bff38f81d\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw" Nov 01 20:51:26.343615 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.343599 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad03483e-106e-4de1-b896-d71bff38f81d-istio-podinfo\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw\" (UID: \"ad03483e-106e-4de1-b896-d71bff38f81d\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw" Nov 01 20:51:26.343820 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.343803 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-istio-envoy\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn\" (UID: \"cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn" Nov 01 20:51:26.344175 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.344154 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-istio-podinfo\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn\" (UID: \"cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn" Nov 01 20:51:26.394219 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.394184 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-feab1a53-8qzv9"] Nov 01 20:51:26.398061 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.398036 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad03483e-106e-4de1-b896-d71bff38f81d-istio-token\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw\" (UID: \"ad03483e-106e-4de1-b896-d71bff38f81d\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw" Nov 01 20:51:26.398231 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.398150 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgstf\" (UniqueName: \"kubernetes.io/projected/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-kube-api-access-bgstf\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn\" (UID: \"cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn" Nov 01 20:51:26.398231 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.398169 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fgvd\" (UniqueName: \"kubernetes.io/projected/ad03483e-106e-4de1-b896-d71bff38f81d-kube-api-access-8fgvd\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw\" (UID: \"ad03483e-106e-4de1-b896-d71bff38f81d\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw" Nov 01 20:51:26.398390 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.398371 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-istio-token\") pod \"e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn\" (UID: \"cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40\") " pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn" Nov 01 20:51:26.400257 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.400233 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-feab1a53-8qzv9" Nov 01 20:51:26.443714 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.443677 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s75tt\" (UniqueName: \"kubernetes.io/projected/18d805d2-dbd0-4576-92ec-204a0c44c078-kube-api-access-s75tt\") pod \"maas-api-revoke-sub-feab1a53-8qzv9\" (UID: \"18d805d2-dbd0-4576-92ec-204a0c44c078\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feab1a53-8qzv9" Nov 01 20:51:26.444475 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.443986 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18d805d2-dbd0-4576-92ec-204a0c44c078-maas-api-xks-ca\") pod \"maas-api-revoke-sub-feab1a53-8qzv9\" (UID: \"18d805d2-dbd0-4576-92ec-204a0c44c078\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feab1a53-8qzv9" Nov 01 20:51:26.444475 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.444109 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18d805d2-dbd0-4576-92ec-204a0c44c078-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-feab1a53-8qzv9\" (UID: \"18d805d2-dbd0-4576-92ec-204a0c44c078\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feab1a53-8qzv9" Nov 01 20:51:26.459875 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.458285 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-feab1a53-8qzv9"] Nov 01 20:51:26.462441 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.462387 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn" Nov 01 20:51:26.488613 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.488565 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw" Nov 01 20:51:26.546382 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.546343 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18d805d2-dbd0-4576-92ec-204a0c44c078-maas-api-xks-ca\") pod \"maas-api-revoke-sub-feab1a53-8qzv9\" (UID: \"18d805d2-dbd0-4576-92ec-204a0c44c078\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feab1a53-8qzv9" Nov 01 20:51:26.546522 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.546452 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18d805d2-dbd0-4576-92ec-204a0c44c078-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-feab1a53-8qzv9\" (UID: \"18d805d2-dbd0-4576-92ec-204a0c44c078\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feab1a53-8qzv9" Nov 01 20:51:26.546672 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.546653 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s75tt\" (UniqueName: \"kubernetes.io/projected/18d805d2-dbd0-4576-92ec-204a0c44c078-kube-api-access-s75tt\") pod \"maas-api-revoke-sub-feab1a53-8qzv9\" (UID: \"18d805d2-dbd0-4576-92ec-204a0c44c078\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feab1a53-8qzv9" Nov 01 20:51:26.547694 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.547670 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18d805d2-dbd0-4576-92ec-204a0c44c078-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-feab1a53-8qzv9\" (UID: \"18d805d2-dbd0-4576-92ec-204a0c44c078\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feab1a53-8qzv9" Nov 01 20:51:26.550219 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.549686 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18d805d2-dbd0-4576-92ec-204a0c44c078-maas-api-xks-ca\") pod \"maas-api-revoke-sub-feab1a53-8qzv9\" (UID: \"18d805d2-dbd0-4576-92ec-204a0c44c078\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feab1a53-8qzv9" Nov 01 20:51:26.605490 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.605457 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s75tt\" (UniqueName: \"kubernetes.io/projected/18d805d2-dbd0-4576-92ec-204a0c44c078-kube-api-access-s75tt\") pod \"maas-api-revoke-sub-feab1a53-8qzv9\" (UID: \"18d805d2-dbd0-4576-92ec-204a0c44c078\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feab1a53-8qzv9" Nov 01 20:51:26.739575 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.739545 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-feab1a53-8qzv9" Nov 01 20:51:26.754198 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.754164 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-422d19ef-wqltd" event={"ID":"d5b0b65e-dce0-4cf8-a6d1-10d3836e6b83","Type":"ContainerStarted","Data":"98eb9f67995f9d65e35ae193d13444057820e3d6c51777451704895e968334bb"} Nov 01 20:51:26.754343 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.754225 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-422d19ef-wqltd" event={"ID":"d5b0b65e-dce0-4cf8-a6d1-10d3836e6b83","Type":"ContainerStarted","Data":"0af150bd542c388ffc0dc80ebd42fbaa60941a13d24cbfbf01cdb3e25a926f18"} Nov 01 20:51:26.854175 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.854083 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:51:26.935937 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.907921 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-422d19ef-wqltd" podStartSLOduration=1.9079013599999999 podStartE2EDuration="1.90790136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:51:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:51:26.889536324 +0000 UTC m=+1237.667176460" watchObservedRunningTime="2026-11-01 20:51:26.90790136 +0000 UTC m=+1237.685541476" Nov 01 20:51:26.977347 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.976421 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:51:26.977347 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.976518 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:51:26.977347 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:26.976565 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:51:27.053944 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:27.053902 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn"] Nov 01 20:51:27.054413 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:27.054359 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:51:27.054525 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:27.054503 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:51:27.054665 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:27.054596 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:51:27.064138 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:27.063932 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw"] Nov 01 20:51:27.132798 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:27.132772 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-feab1a53-8qzv9"] Nov 01 20:51:27.134050 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:51:27.134025 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18d805d2_dbd0_4576_92ec_204a0c44c078.slice/crio-dc25deb15c5d3500e61cc9d89e3864071950ac3b62f8dc9078fea3009a3d05a4 WatchSource:0}: Error finding container dc25deb15c5d3500e61cc9d89e3864071950ac3b62f8dc9078fea3009a3d05a4: Status 404 returned error can't find the container with id dc25deb15c5d3500e61cc9d89e3864071950ac3b62f8dc9078fea3009a3d05a4 Nov 01 20:51:27.395968 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:27.395890 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cac12c9e-f52b-4786-ab06-ca7daea3acb9" path="/var/lib/kubelet/pods/cac12c9e-f52b-4786-ab06-ca7daea3acb9/volumes" Nov 01 20:51:27.500967 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:27.500933 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d2f4c30-gskrl"] Nov 01 20:51:27.506959 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:27.506933 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d2f4c30-gskrl" Nov 01 20:51:27.565617 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:27.564391 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d2f4c30-gskrl"] Nov 01 20:51:27.593184 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:27.593142 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2kgj\" (UniqueName: \"kubernetes.io/projected/a086ec6d-bd79-41b5-b29d-64994f49a0a7-kube-api-access-t2kgj\") pod \"maas-api-revoke-sub-0d2f4c30-gskrl\" (UID: \"a086ec6d-bd79-41b5-b29d-64994f49a0a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d2f4c30-gskrl" Nov 01 20:51:27.593353 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:27.593336 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a086ec6d-bd79-41b5-b29d-64994f49a0a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d2f4c30-gskrl\" (UID: \"a086ec6d-bd79-41b5-b29d-64994f49a0a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d2f4c30-gskrl" Nov 01 20:51:27.593412 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:27.593389 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a086ec6d-bd79-41b5-b29d-64994f49a0a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d2f4c30-gskrl\" (UID: \"a086ec6d-bd79-41b5-b29d-64994f49a0a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d2f4c30-gskrl" Nov 01 20:51:27.700069 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:27.695812 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a086ec6d-bd79-41b5-b29d-64994f49a0a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d2f4c30-gskrl\" (UID: \"a086ec6d-bd79-41b5-b29d-64994f49a0a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d2f4c30-gskrl" Nov 01 20:51:27.700069 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:27.696526 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a086ec6d-bd79-41b5-b29d-64994f49a0a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d2f4c30-gskrl\" (UID: \"a086ec6d-bd79-41b5-b29d-64994f49a0a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d2f4c30-gskrl" Nov 01 20:51:27.700069 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:27.697375 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a086ec6d-bd79-41b5-b29d-64994f49a0a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d2f4c30-gskrl\" (UID: \"a086ec6d-bd79-41b5-b29d-64994f49a0a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d2f4c30-gskrl" Nov 01 20:51:27.704559 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:27.700633 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2kgj\" (UniqueName: \"kubernetes.io/projected/a086ec6d-bd79-41b5-b29d-64994f49a0a7-kube-api-access-t2kgj\") pod \"maas-api-revoke-sub-0d2f4c30-gskrl\" (UID: \"a086ec6d-bd79-41b5-b29d-64994f49a0a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d2f4c30-gskrl" Nov 01 20:51:27.732849 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:27.729310 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a086ec6d-bd79-41b5-b29d-64994f49a0a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d2f4c30-gskrl\" (UID: \"a086ec6d-bd79-41b5-b29d-64994f49a0a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d2f4c30-gskrl" Nov 01 20:51:27.770059 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:27.762545 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2kgj\" (UniqueName: \"kubernetes.io/projected/a086ec6d-bd79-41b5-b29d-64994f49a0a7-kube-api-access-t2kgj\") pod \"maas-api-revoke-sub-0d2f4c30-gskrl\" (UID: \"a086ec6d-bd79-41b5-b29d-64994f49a0a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d2f4c30-gskrl" Nov 01 20:51:27.782950 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:27.782921 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw" event={"ID":"ad03483e-106e-4de1-b896-d71bff38f81d","Type":"ContainerStarted","Data":"a6b8ef447c1a49b73cfe47caef054e5e93533fb23b177823a88f4f99cd89e50f"} Nov 01 20:51:27.783453 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:27.783434 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw" event={"ID":"ad03483e-106e-4de1-b896-d71bff38f81d","Type":"ContainerStarted","Data":"c7c067e5a73d17a5d51be8430355c02f9c2d546d577d745074a9e97a836492c9"} Nov 01 20:51:27.791911 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:27.791880 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn" event={"ID":"cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40","Type":"ContainerStarted","Data":"b7ae38d1714f3cf72f0431d85bfe89e750a9f756180a4e9965625a1df1452ac4"} Nov 01 20:51:27.792065 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:27.791928 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn" event={"ID":"cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40","Type":"ContainerStarted","Data":"b2c3b2a511288a25fd80a93342e8f8cbb60486b1f38c925231b38b1c3fd2a246"} Nov 01 20:51:27.794476 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:27.794449 4484 generic.go:356] "Generic (PLEG): container finished" podID="d5b0b65e-dce0-4cf8-a6d1-10d3836e6b83" containerID="98eb9f67995f9d65e35ae193d13444057820e3d6c51777451704895e968334bb" exitCode=0 Nov 01 20:51:27.794609 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:27.794595 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-422d19ef-wqltd" event={"ID":"d5b0b65e-dce0-4cf8-a6d1-10d3836e6b83","Type":"ContainerDied","Data":"98eb9f67995f9d65e35ae193d13444057820e3d6c51777451704895e968334bb"} Nov 01 20:51:27.796488 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:27.796463 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-feab1a53-8qzv9" event={"ID":"18d805d2-dbd0-4576-92ec-204a0c44c078","Type":"ContainerStarted","Data":"dc25deb15c5d3500e61cc9d89e3864071950ac3b62f8dc9078fea3009a3d05a4"} Nov 01 20:51:27.848024 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:27.832973 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d2f4c30-gskrl" Nov 01 20:51:27.966623 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:27.966519 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw" podStartSLOduration=1.966505511 podStartE2EDuration="1.966505511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:51:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:51:27.961269125 +0000 UTC m=+1238.738909260" watchObservedRunningTime="2026-11-01 20:51:27.966505511 +0000 UTC m=+1238.744145692" Nov 01 20:51:28.081559 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:28.080189 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn" podStartSLOduration=2.080171631 podStartE2EDuration="2.080171631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:51:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:51:28.079557039 +0000 UTC m=+1238.857197173" watchObservedRunningTime="2026-11-01 20:51:28.080171631 +0000 UTC m=+1238.857811765" Nov 01 20:51:28.267024 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:28.265609 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-feab1a53-8qzv9" podStartSLOduration=2.265586162 podStartE2EDuration="2.265586162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:51:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:51:28.263327448 +0000 UTC m=+1239.040967587" watchObservedRunningTime="2026-11-01 20:51:28.265586162 +0000 UTC m=+1239.043226299" Nov 01 20:51:28.351671 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:28.351629 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d2f4c30-gskrl"] Nov 01 20:51:28.466248 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:28.466206 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn" Nov 01 20:51:28.467835 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:28.467799 4484 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 01 20:51:28.467957 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:28.467854 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn" podUID="cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 01 20:51:28.488974 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:28.488949 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw" Nov 01 20:51:28.490073 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:28.490049 4484 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 01 20:51:28.490174 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:28.490095 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw" podUID="ad03483e-106e-4de1-b896-d71bff38f81d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 01 20:51:28.837022 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:28.836687 4484 generic.go:356] "Generic (PLEG): container finished" podID="18d805d2-dbd0-4576-92ec-204a0c44c078" containerID="3cf46e97af4bfca2ed387dce8a73001b403e6a9a36219440497dd1d920d68748" exitCode=0 Nov 01 20:51:28.837022 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:28.836889 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-feab1a53-8qzv9" event={"ID":"18d805d2-dbd0-4576-92ec-204a0c44c078","Type":"ContainerDied","Data":"3cf46e97af4bfca2ed387dce8a73001b403e6a9a36219440497dd1d920d68748"} Nov 01 20:51:28.847018 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:28.846640 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d2f4c30-gskrl" event={"ID":"a086ec6d-bd79-41b5-b29d-64994f49a0a7","Type":"ContainerStarted","Data":"66432158e71b2403641919ce6f1b48a77c9bce791c9fab37ae31a61b39f355be"} Nov 01 20:51:29.361920 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:29.361888 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-422d19ef-wqltd" Nov 01 20:51:29.461712 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:29.461676 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5b0b65e-dce0-4cf8-a6d1-10d3836e6b83-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5b0b65e-dce0-4cf8-a6d1-10d3836e6b83-maas-api-openshift-ca\") pod \"d5b0b65e-dce0-4cf8-a6d1-10d3836e6b83\" (UID: \"d5b0b65e-dce0-4cf8-a6d1-10d3836e6b83\") " Nov 01 20:51:29.462036 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:29.461977 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5b0b65e-dce0-4cf8-a6d1-10d3836e6b83-maas-api-openshift-ca" pod "d5b0b65e-dce0-4cf8-a6d1-10d3836e6b83" (UID: "d5b0b65e-dce0-4cf8-a6d1-10d3836e6b83"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:51:29.462161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:29.462045 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5b0b65e-dce0-4cf8-a6d1-10d3836e6b83-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5b0b65e-dce0-4cf8-a6d1-10d3836e6b83-maas-api-xks-ca\") pod \"d5b0b65e-dce0-4cf8-a6d1-10d3836e6b83\" (UID: \"d5b0b65e-dce0-4cf8-a6d1-10d3836e6b83\") " Nov 01 20:51:29.462161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:29.462154 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5b0b65e-dce0-4cf8-a6d1-10d3836e6b83-kube-api-access-grphz\" (UniqueName: \"kubernetes.io/projected/d5b0b65e-dce0-4cf8-a6d1-10d3836e6b83-kube-api-access-grphz\") pod \"d5b0b65e-dce0-4cf8-a6d1-10d3836e6b83\" (UID: \"d5b0b65e-dce0-4cf8-a6d1-10d3836e6b83\") " Nov 01 20:51:29.462894 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:29.462870 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5b0b65e-dce0-4cf8-a6d1-10d3836e6b83-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:29.465867 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:29.465840 4484 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 01 20:51:29.465970 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:29.465896 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn" podUID="cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 01 20:51:29.466482 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:29.466455 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5b0b65e-dce0-4cf8-a6d1-10d3836e6b83-maas-api-xks-ca" pod "d5b0b65e-dce0-4cf8-a6d1-10d3836e6b83" (UID: "d5b0b65e-dce0-4cf8-a6d1-10d3836e6b83"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:51:29.466873 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:29.466775 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5b0b65e-dce0-4cf8-a6d1-10d3836e6b83-kube-api-access-grphz" pod "d5b0b65e-dce0-4cf8-a6d1-10d3836e6b83" (UID: "d5b0b65e-dce0-4cf8-a6d1-10d3836e6b83"). InnerVolumeSpecName "kube-api-access-grphz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:51:29.489154 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:29.489123 4484 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 01 20:51:29.489310 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:29.489182 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw" podUID="ad03483e-106e-4de1-b896-d71bff38f81d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 01 20:51:29.570515 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:29.569644 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5b0b65e-dce0-4cf8-a6d1-10d3836e6b83-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:29.570515 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:29.569684 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grphz\" (UniqueName: \"kubernetes.io/projected/d5b0b65e-dce0-4cf8-a6d1-10d3836e6b83-kube-api-access-grphz\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:29.864770 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:29.864685 4484 generic.go:356] "Generic (PLEG): container finished" podID="a086ec6d-bd79-41b5-b29d-64994f49a0a7" containerID="bb3743c52e4388a3d036efae5c92ffd5c1316d3f7c3cf23926178116d3ba5708" exitCode=0 Nov 01 20:51:29.864931 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:29.864828 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d2f4c30-gskrl" event={"ID":"a086ec6d-bd79-41b5-b29d-64994f49a0a7","Type":"ContainerDied","Data":"bb3743c52e4388a3d036efae5c92ffd5c1316d3f7c3cf23926178116d3ba5708"} Nov 01 20:51:29.866663 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:29.866643 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-422d19ef-wqltd" Nov 01 20:51:29.867604 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:29.867567 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-422d19ef-wqltd" event={"ID":"d5b0b65e-dce0-4cf8-a6d1-10d3836e6b83","Type":"ContainerDied","Data":"0af150bd542c388ffc0dc80ebd42fbaa60941a13d24cbfbf01cdb3e25a926f18"} Nov 01 20:51:29.867719 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:29.867611 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0af150bd542c388ffc0dc80ebd42fbaa60941a13d24cbfbf01cdb3e25a926f18" Nov 01 20:51:30.188477 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:30.188445 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-feab1a53-8qzv9" Nov 01 20:51:30.281577 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:30.281542 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18d805d2-dbd0-4576-92ec-204a0c44c078-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18d805d2-dbd0-4576-92ec-204a0c44c078-maas-api-openshift-ca\") pod \"18d805d2-dbd0-4576-92ec-204a0c44c078\" (UID: \"18d805d2-dbd0-4576-92ec-204a0c44c078\") " Nov 01 20:51:30.281727 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:30.281716 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18d805d2-dbd0-4576-92ec-204a0c44c078-kube-api-access-s75tt\" (UniqueName: \"kubernetes.io/projected/18d805d2-dbd0-4576-92ec-204a0c44c078-kube-api-access-s75tt\") pod \"18d805d2-dbd0-4576-92ec-204a0c44c078\" (UID: \"18d805d2-dbd0-4576-92ec-204a0c44c078\") " Nov 01 20:51:30.281792 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:30.281775 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18d805d2-dbd0-4576-92ec-204a0c44c078-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18d805d2-dbd0-4576-92ec-204a0c44c078-maas-api-xks-ca\") pod \"18d805d2-dbd0-4576-92ec-204a0c44c078\" (UID: \"18d805d2-dbd0-4576-92ec-204a0c44c078\") " Nov 01 20:51:30.281950 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:30.281923 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18d805d2-dbd0-4576-92ec-204a0c44c078-maas-api-openshift-ca" pod "18d805d2-dbd0-4576-92ec-204a0c44c078" (UID: "18d805d2-dbd0-4576-92ec-204a0c44c078"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:51:30.282299 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:30.282278 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18d805d2-dbd0-4576-92ec-204a0c44c078-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:30.283874 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:30.283847 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18d805d2-dbd0-4576-92ec-204a0c44c078-maas-api-xks-ca" pod "18d805d2-dbd0-4576-92ec-204a0c44c078" (UID: "18d805d2-dbd0-4576-92ec-204a0c44c078"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:51:30.283964 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:30.283898 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18d805d2-dbd0-4576-92ec-204a0c44c078-kube-api-access-s75tt" pod "18d805d2-dbd0-4576-92ec-204a0c44c078" (UID: "18d805d2-dbd0-4576-92ec-204a0c44c078"). InnerVolumeSpecName "kube-api-access-s75tt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:51:30.383555 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:30.383516 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s75tt\" (UniqueName: \"kubernetes.io/projected/18d805d2-dbd0-4576-92ec-204a0c44c078-kube-api-access-s75tt\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:30.383555 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:30.383544 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18d805d2-dbd0-4576-92ec-204a0c44c078-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:30.467474 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:30.467396 4484 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 01 20:51:30.467474 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:30.467451 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn" podUID="cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 01 20:51:30.489843 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:30.489815 4484 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 01 20:51:30.489989 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:30.489863 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw" podUID="ad03483e-106e-4de1-b896-d71bff38f81d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 01 20:51:30.876818 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:30.876781 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-feab1a53-8qzv9" Nov 01 20:51:30.876818 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:30.876806 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-feab1a53-8qzv9" event={"ID":"18d805d2-dbd0-4576-92ec-204a0c44c078","Type":"ContainerDied","Data":"dc25deb15c5d3500e61cc9d89e3864071950ac3b62f8dc9078fea3009a3d05a4"} Nov 01 20:51:30.877085 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:30.876847 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc25deb15c5d3500e61cc9d89e3864071950ac3b62f8dc9078fea3009a3d05a4" Nov 01 20:51:31.190121 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:31.190096 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d2f4c30-gskrl" Nov 01 20:51:31.298440 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:31.297582 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a086ec6d-bd79-41b5-b29d-64994f49a0a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a086ec6d-bd79-41b5-b29d-64994f49a0a7-maas-api-openshift-ca\") pod \"a086ec6d-bd79-41b5-b29d-64994f49a0a7\" (UID: \"a086ec6d-bd79-41b5-b29d-64994f49a0a7\") " Nov 01 20:51:31.298440 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:31.297742 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a086ec6d-bd79-41b5-b29d-64994f49a0a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a086ec6d-bd79-41b5-b29d-64994f49a0a7-maas-api-xks-ca\") pod \"a086ec6d-bd79-41b5-b29d-64994f49a0a7\" (UID: \"a086ec6d-bd79-41b5-b29d-64994f49a0a7\") " Nov 01 20:51:31.298440 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:31.297848 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a086ec6d-bd79-41b5-b29d-64994f49a0a7-kube-api-access-t2kgj\" (UniqueName: \"kubernetes.io/projected/a086ec6d-bd79-41b5-b29d-64994f49a0a7-kube-api-access-t2kgj\") pod \"a086ec6d-bd79-41b5-b29d-64994f49a0a7\" (UID: \"a086ec6d-bd79-41b5-b29d-64994f49a0a7\") " Nov 01 20:51:31.298922 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:31.298892 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a086ec6d-bd79-41b5-b29d-64994f49a0a7-maas-api-openshift-ca" pod "a086ec6d-bd79-41b5-b29d-64994f49a0a7" (UID: "a086ec6d-bd79-41b5-b29d-64994f49a0a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:51:31.300414 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:31.300386 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a086ec6d-bd79-41b5-b29d-64994f49a0a7-kube-api-access-t2kgj" pod "a086ec6d-bd79-41b5-b29d-64994f49a0a7" (UID: "a086ec6d-bd79-41b5-b29d-64994f49a0a7"). InnerVolumeSpecName "kube-api-access-t2kgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:51:31.300924 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:31.300906 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a086ec6d-bd79-41b5-b29d-64994f49a0a7-maas-api-xks-ca" pod "a086ec6d-bd79-41b5-b29d-64994f49a0a7" (UID: "a086ec6d-bd79-41b5-b29d-64994f49a0a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:51:31.399682 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:31.399657 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2kgj\" (UniqueName: \"kubernetes.io/projected/a086ec6d-bd79-41b5-b29d-64994f49a0a7-kube-api-access-t2kgj\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:31.399843 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:31.399831 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a086ec6d-bd79-41b5-b29d-64994f49a0a7-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:31.399905 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:31.399898 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a086ec6d-bd79-41b5-b29d-64994f49a0a7-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:31.465838 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:31.465757 4484 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 01 20:51:31.465838 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:31.465806 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn" podUID="cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 01 20:51:31.489658 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:31.489619 4484 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 01 20:51:31.490055 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:31.489680 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw" podUID="ad03483e-106e-4de1-b896-d71bff38f81d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 01 20:51:31.796272 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:31.790101 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:51:31.947020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:31.946794 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d2f4c30-gskrl" event={"ID":"a086ec6d-bd79-41b5-b29d-64994f49a0a7","Type":"ContainerDied","Data":"66432158e71b2403641919ce6f1b48a77c9bce791c9fab37ae31a61b39f355be"} Nov 01 20:51:31.947020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:31.946864 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66432158e71b2403641919ce6f1b48a77c9bce791c9fab37ae31a61b39f355be" Nov 01 20:51:31.947020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:31.946952 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d2f4c30-gskrl" Nov 01 20:51:32.465473 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:32.465440 4484 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 01 20:51:32.465653 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:32.465489 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn" podUID="cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 01 20:51:32.489761 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:32.489721 4484 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 01 20:51:32.490169 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:32.489784 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw" podUID="ad03483e-106e-4de1-b896-d71bff38f81d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 01 20:51:33.059814 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:33.059765 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10fcaf59-jcdrh"] Nov 01 20:51:33.064783 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:33.064738 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10fcaf59-jcdrh"] Nov 01 20:51:33.375656 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:33.375562 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebd12d3e-e201-4170-b5f0-18ed8d300682" path="/var/lib/kubelet/pods/ebd12d3e-e201-4170-b5f0-18ed8d300682/volumes" Nov 01 20:51:33.466488 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:33.466452 4484 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 01 20:51:33.466633 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:33.466511 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn" podUID="cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 01 20:51:33.490835 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:33.490796 4484 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 01 20:51:33.491432 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:33.491401 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw" podUID="ad03483e-106e-4de1-b896-d71bff38f81d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 01 20:51:34.788109 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:34.788078 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn" Nov 01 20:51:34.790061 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:34.789693 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn" Nov 01 20:51:34.798315 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:34.798290 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn" Nov 01 20:51:34.829463 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:34.829435 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw" Nov 01 20:51:34.829757 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:34.829733 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw" Nov 01 20:51:34.830587 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:34.830567 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw" Nov 01 20:51:34.898359 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:34.898326 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fcc929e-fgr7c"] Nov 01 20:51:34.899042 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:34.899028 4484 cpu_manager.go:414] "containerMap: removing container" podUID="d5b0b65e-dce0-4cf8-a6d1-10d3836e6b83" containerName="revoke-keys" Nov 01 20:51:34.899106 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:34.899043 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5b0b65e-dce0-4cf8-a6d1-10d3836e6b83" containerName="revoke-keys" Nov 01 20:51:34.899106 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:34.899070 4484 cpu_manager.go:414] "containerMap: removing container" podUID="a086ec6d-bd79-41b5-b29d-64994f49a0a7" containerName="revoke-keys" Nov 01 20:51:34.899106 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:34.899075 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a086ec6d-bd79-41b5-b29d-64994f49a0a7" containerName="revoke-keys" Nov 01 20:51:34.899106 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:34.899096 4484 cpu_manager.go:414] "containerMap: removing container" podUID="18d805d2-dbd0-4576-92ec-204a0c44c078" containerName="revoke-keys" Nov 01 20:51:34.899106 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:34.899100 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18d805d2-dbd0-4576-92ec-204a0c44c078" containerName="revoke-keys" Nov 01 20:51:34.899260 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:34.899211 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="a086ec6d-bd79-41b5-b29d-64994f49a0a7" containerName="revoke-keys" Nov 01 20:51:34.899260 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:34.899223 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="18d805d2-dbd0-4576-92ec-204a0c44c078" containerName="revoke-keys" Nov 01 20:51:34.899260 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:34.899234 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5b0b65e-dce0-4cf8-a6d1-10d3836e6b83" containerName="revoke-keys" Nov 01 20:51:34.902847 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:34.902829 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fcc929e-fgr7c" Nov 01 20:51:34.914149 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:34.914120 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k6ft\" (UniqueName: \"kubernetes.io/projected/146ef2d2-52b4-43ea-8d0b-c9eac81f4ff5-kube-api-access-7k6ft\") pod \"maas-api-revoke-sub-1fcc929e-fgr7c\" (UID: \"146ef2d2-52b4-43ea-8d0b-c9eac81f4ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fcc929e-fgr7c" Nov 01 20:51:34.914346 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:34.914325 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/146ef2d2-52b4-43ea-8d0b-c9eac81f4ff5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fcc929e-fgr7c\" (UID: \"146ef2d2-52b4-43ea-8d0b-c9eac81f4ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fcc929e-fgr7c" Nov 01 20:51:34.914447 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:34.914434 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/146ef2d2-52b4-43ea-8d0b-c9eac81f4ff5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fcc929e-fgr7c\" (UID: \"146ef2d2-52b4-43ea-8d0b-c9eac81f4ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fcc929e-fgr7c" Nov 01 20:51:34.957884 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:34.957848 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:51:34.958070 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:34.957967 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:51:34.958159 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:34.958143 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:51:34.960242 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:34.960218 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fcc929e-fgr7c"] Nov 01 20:51:35.008387 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:35.008358 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 20:51:35.016368 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:35.016340 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k6ft\" (UniqueName: \"kubernetes.io/projected/146ef2d2-52b4-43ea-8d0b-c9eac81f4ff5-kube-api-access-7k6ft\") pod \"maas-api-revoke-sub-1fcc929e-fgr7c\" (UID: \"146ef2d2-52b4-43ea-8d0b-c9eac81f4ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fcc929e-fgr7c" Nov 01 20:51:35.016539 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:35.016522 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/146ef2d2-52b4-43ea-8d0b-c9eac81f4ff5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fcc929e-fgr7c\" (UID: \"146ef2d2-52b4-43ea-8d0b-c9eac81f4ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fcc929e-fgr7c" Nov 01 20:51:35.016730 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:35.016706 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/146ef2d2-52b4-43ea-8d0b-c9eac81f4ff5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fcc929e-fgr7c\" (UID: \"146ef2d2-52b4-43ea-8d0b-c9eac81f4ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fcc929e-fgr7c" Nov 01 20:51:35.017550 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:35.017524 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/146ef2d2-52b4-43ea-8d0b-c9eac81f4ff5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fcc929e-fgr7c\" (UID: \"146ef2d2-52b4-43ea-8d0b-c9eac81f4ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fcc929e-fgr7c" Nov 01 20:51:35.019135 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:35.019110 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/146ef2d2-52b4-43ea-8d0b-c9eac81f4ff5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fcc929e-fgr7c\" (UID: \"146ef2d2-52b4-43ea-8d0b-c9eac81f4ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fcc929e-fgr7c" Nov 01 20:51:35.074055 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:35.073973 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k6ft\" (UniqueName: \"kubernetes.io/projected/146ef2d2-52b4-43ea-8d0b-c9eac81f4ff5-kube-api-access-7k6ft\") pod \"maas-api-revoke-sub-1fcc929e-fgr7c\" (UID: \"146ef2d2-52b4-43ea-8d0b-c9eac81f4ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fcc929e-fgr7c" Nov 01 20:51:35.219231 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:35.219187 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fcc929e-fgr7c" Nov 01 20:51:35.622678 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:51:35.622646 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod146ef2d2_52b4_43ea_8d0b_c9eac81f4ff5.slice/crio-908ef5abb954e78402fbf4163eab960721e051a98d2884240ba2a1a1935865e2 WatchSource:0}: Error finding container 908ef5abb954e78402fbf4163eab960721e051a98d2884240ba2a1a1935865e2: Status 404 returned error can't find the container with id 908ef5abb954e78402fbf4163eab960721e051a98d2884240ba2a1a1935865e2 Nov 01 20:51:35.624226 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:35.624199 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fcc929e-fgr7c"] Nov 01 20:51:35.820408 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:35.820376 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e407a6a4-55qlv"] Nov 01 20:51:35.829538 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:35.829515 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e407a6a4-55qlv" Nov 01 20:51:35.881664 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:35.881585 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e407a6a4-55qlv"] Nov 01 20:51:35.929841 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:35.929812 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95b4299f-f0db-4901-b902-9acc154edc3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e407a6a4-55qlv\" (UID: \"95b4299f-f0db-4901-b902-9acc154edc3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e407a6a4-55qlv" Nov 01 20:51:35.930011 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:35.929889 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgcpt\" (UniqueName: \"kubernetes.io/projected/95b4299f-f0db-4901-b902-9acc154edc3f-kube-api-access-rgcpt\") pod \"maas-api-revoke-sub-e407a6a4-55qlv\" (UID: \"95b4299f-f0db-4901-b902-9acc154edc3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e407a6a4-55qlv" Nov 01 20:51:35.930116 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:35.930083 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95b4299f-f0db-4901-b902-9acc154edc3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e407a6a4-55qlv\" (UID: \"95b4299f-f0db-4901-b902-9acc154edc3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e407a6a4-55qlv" Nov 01 20:51:35.991615 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:35.991580 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fcc929e-fgr7c" event={"ID":"146ef2d2-52b4-43ea-8d0b-c9eac81f4ff5","Type":"ContainerStarted","Data":"11755af73def77184bacdf0b2e1be9d68fe4b5db0906f6aefb7486f58dd525ff"} Nov 01 20:51:35.991799 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:35.991634 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fcc929e-fgr7c" event={"ID":"146ef2d2-52b4-43ea-8d0b-c9eac81f4ff5","Type":"ContainerStarted","Data":"908ef5abb954e78402fbf4163eab960721e051a98d2884240ba2a1a1935865e2"} Nov 01 20:51:36.031439 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:36.031389 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95b4299f-f0db-4901-b902-9acc154edc3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e407a6a4-55qlv\" (UID: \"95b4299f-f0db-4901-b902-9acc154edc3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e407a6a4-55qlv" Nov 01 20:51:36.031634 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:36.031459 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgcpt\" (UniqueName: \"kubernetes.io/projected/95b4299f-f0db-4901-b902-9acc154edc3f-kube-api-access-rgcpt\") pod \"maas-api-revoke-sub-e407a6a4-55qlv\" (UID: \"95b4299f-f0db-4901-b902-9acc154edc3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e407a6a4-55qlv" Nov 01 20:51:36.031634 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:36.031577 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95b4299f-f0db-4901-b902-9acc154edc3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e407a6a4-55qlv\" (UID: \"95b4299f-f0db-4901-b902-9acc154edc3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e407a6a4-55qlv" Nov 01 20:51:36.032385 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:36.032355 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95b4299f-f0db-4901-b902-9acc154edc3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e407a6a4-55qlv\" (UID: \"95b4299f-f0db-4901-b902-9acc154edc3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e407a6a4-55qlv" Nov 01 20:51:36.033972 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:36.033950 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95b4299f-f0db-4901-b902-9acc154edc3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e407a6a4-55qlv\" (UID: \"95b4299f-f0db-4901-b902-9acc154edc3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e407a6a4-55qlv" Nov 01 20:51:36.089386 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:36.089354 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgcpt\" (UniqueName: \"kubernetes.io/projected/95b4299f-f0db-4901-b902-9acc154edc3f-kube-api-access-rgcpt\") pod \"maas-api-revoke-sub-e407a6a4-55qlv\" (UID: \"95b4299f-f0db-4901-b902-9acc154edc3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e407a6a4-55qlv" Nov 01 20:51:36.107440 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:36.107388 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fcc929e-fgr7c" podStartSLOduration=2.107373526 podStartE2EDuration="2.107373526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:51:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:51:36.106290012 +0000 UTC m=+1246.883930146" watchObservedRunningTime="2026-11-01 20:51:36.107373526 +0000 UTC m=+1246.885013662" Nov 01 20:51:36.143373 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:36.143291 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e407a6a4-55qlv" Nov 01 20:51:36.523839 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:36.523812 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e407a6a4-55qlv"] Nov 01 20:51:36.542027 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:51:36.541978 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95b4299f_f0db_4901_b902_9acc154edc3f.slice/crio-14f5d9f04ef80897e4f8e773aae2c267e3748eaccbc15074d3985c0990152289 WatchSource:0}: Error finding container 14f5d9f04ef80897e4f8e773aae2c267e3748eaccbc15074d3985c0990152289: Status 404 returned error can't find the container with id 14f5d9f04ef80897e4f8e773aae2c267e3748eaccbc15074d3985c0990152289 Nov 01 20:51:36.649243 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:36.649209 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:51:37.002934 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:37.002848 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e407a6a4-55qlv" event={"ID":"95b4299f-f0db-4901-b902-9acc154edc3f","Type":"ContainerStarted","Data":"6d64ae249961d215d70fc1faf813ab0e7424a23286ea1d20e1f1fd2ca42da3d6"} Nov 01 20:51:37.002934 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:37.002906 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e407a6a4-55qlv" event={"ID":"95b4299f-f0db-4901-b902-9acc154edc3f","Type":"ContainerStarted","Data":"14f5d9f04ef80897e4f8e773aae2c267e3748eaccbc15074d3985c0990152289"} Nov 01 20:51:37.004853 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:37.004827 4484 generic.go:356] "Generic (PLEG): container finished" podID="146ef2d2-52b4-43ea-8d0b-c9eac81f4ff5" containerID="11755af73def77184bacdf0b2e1be9d68fe4b5db0906f6aefb7486f58dd525ff" exitCode=0 Nov 01 20:51:37.004971 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:37.004872 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fcc929e-fgr7c" event={"ID":"146ef2d2-52b4-43ea-8d0b-c9eac81f4ff5","Type":"ContainerDied","Data":"11755af73def77184bacdf0b2e1be9d68fe4b5db0906f6aefb7486f58dd525ff"} Nov 01 20:51:37.122265 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:37.122215 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e407a6a4-55qlv" podStartSLOduration=2.122199837 podStartE2EDuration="2.122199837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:51:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:51:37.118330585 +0000 UTC m=+1247.895970720" watchObservedRunningTime="2026-11-01 20:51:37.122199837 +0000 UTC m=+1247.899839971" Nov 01 20:51:37.171890 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:37.171854 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70bcb6f1-fbph2"] Nov 01 20:51:37.176514 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:37.176490 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70bcb6f1-fbph2" Nov 01 20:51:37.232901 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:37.232871 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70bcb6f1-fbph2"] Nov 01 20:51:37.250769 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:37.250734 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2e9e863-ca7f-4f6f-9943-557293100f33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70bcb6f1-fbph2\" (UID: \"c2e9e863-ca7f-4f6f-9943-557293100f33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70bcb6f1-fbph2" Nov 01 20:51:37.250963 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:37.250887 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fr76\" (UniqueName: \"kubernetes.io/projected/c2e9e863-ca7f-4f6f-9943-557293100f33-kube-api-access-4fr76\") pod \"maas-api-revoke-sub-70bcb6f1-fbph2\" (UID: \"c2e9e863-ca7f-4f6f-9943-557293100f33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70bcb6f1-fbph2" Nov 01 20:51:37.251083 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:37.251052 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2e9e863-ca7f-4f6f-9943-557293100f33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70bcb6f1-fbph2\" (UID: \"c2e9e863-ca7f-4f6f-9943-557293100f33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70bcb6f1-fbph2" Nov 01 20:51:37.354375 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:37.354276 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2e9e863-ca7f-4f6f-9943-557293100f33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70bcb6f1-fbph2\" (UID: \"c2e9e863-ca7f-4f6f-9943-557293100f33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70bcb6f1-fbph2" Nov 01 20:51:37.354539 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:37.354412 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fr76\" (UniqueName: \"kubernetes.io/projected/c2e9e863-ca7f-4f6f-9943-557293100f33-kube-api-access-4fr76\") pod \"maas-api-revoke-sub-70bcb6f1-fbph2\" (UID: \"c2e9e863-ca7f-4f6f-9943-557293100f33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70bcb6f1-fbph2" Nov 01 20:51:37.354539 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:37.354516 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2e9e863-ca7f-4f6f-9943-557293100f33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70bcb6f1-fbph2\" (UID: \"c2e9e863-ca7f-4f6f-9943-557293100f33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70bcb6f1-fbph2" Nov 01 20:51:37.357984 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:37.357776 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2e9e863-ca7f-4f6f-9943-557293100f33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70bcb6f1-fbph2\" (UID: \"c2e9e863-ca7f-4f6f-9943-557293100f33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70bcb6f1-fbph2" Nov 01 20:51:37.359154 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:37.358704 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2e9e863-ca7f-4f6f-9943-557293100f33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70bcb6f1-fbph2\" (UID: \"c2e9e863-ca7f-4f6f-9943-557293100f33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70bcb6f1-fbph2" Nov 01 20:51:37.417051 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:37.417020 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fr76\" (UniqueName: \"kubernetes.io/projected/c2e9e863-ca7f-4f6f-9943-557293100f33-kube-api-access-4fr76\") pod \"maas-api-revoke-sub-70bcb6f1-fbph2\" (UID: \"c2e9e863-ca7f-4f6f-9943-557293100f33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70bcb6f1-fbph2" Nov 01 20:51:37.490733 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:37.490706 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70bcb6f1-fbph2" Nov 01 20:51:37.867636 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:51:37.867601 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2e9e863_ca7f_4f6f_9943_557293100f33.slice/crio-a1ca45586ce4beaab9bf9444845da663093e3a4f6151622c29dd7366bd8f5fa4 WatchSource:0}: Error finding container a1ca45586ce4beaab9bf9444845da663093e3a4f6151622c29dd7366bd8f5fa4: Status 404 returned error can't find the container with id a1ca45586ce4beaab9bf9444845da663093e3a4f6151622c29dd7366bd8f5fa4 Nov 01 20:51:37.869134 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:37.869112 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70bcb6f1-fbph2"] Nov 01 20:51:38.016089 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:38.016056 4484 generic.go:356] "Generic (PLEG): container finished" podID="95b4299f-f0db-4901-b902-9acc154edc3f" containerID="6d64ae249961d215d70fc1faf813ab0e7424a23286ea1d20e1f1fd2ca42da3d6" exitCode=0 Nov 01 20:51:38.016517 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:38.016142 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e407a6a4-55qlv" event={"ID":"95b4299f-f0db-4901-b902-9acc154edc3f","Type":"ContainerDied","Data":"6d64ae249961d215d70fc1faf813ab0e7424a23286ea1d20e1f1fd2ca42da3d6"} Nov 01 20:51:38.017804 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:38.017781 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70bcb6f1-fbph2" event={"ID":"c2e9e863-ca7f-4f6f-9943-557293100f33","Type":"ContainerStarted","Data":"a1ca45586ce4beaab9bf9444845da663093e3a4f6151622c29dd7366bd8f5fa4"} Nov 01 20:51:38.190585 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:38.190559 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fcc929e-fgr7c" Nov 01 20:51:38.269471 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:38.269442 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/146ef2d2-52b4-43ea-8d0b-c9eac81f4ff5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/146ef2d2-52b4-43ea-8d0b-c9eac81f4ff5-maas-api-openshift-ca\") pod \"146ef2d2-52b4-43ea-8d0b-c9eac81f4ff5\" (UID: \"146ef2d2-52b4-43ea-8d0b-c9eac81f4ff5\") " Nov 01 20:51:38.269620 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:38.269580 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/146ef2d2-52b4-43ea-8d0b-c9eac81f4ff5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/146ef2d2-52b4-43ea-8d0b-c9eac81f4ff5-maas-api-xks-ca\") pod \"146ef2d2-52b4-43ea-8d0b-c9eac81f4ff5\" (UID: \"146ef2d2-52b4-43ea-8d0b-c9eac81f4ff5\") " Nov 01 20:51:38.269704 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:38.269688 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/146ef2d2-52b4-43ea-8d0b-c9eac81f4ff5-kube-api-access-7k6ft\" (UniqueName: \"kubernetes.io/projected/146ef2d2-52b4-43ea-8d0b-c9eac81f4ff5-kube-api-access-7k6ft\") pod \"146ef2d2-52b4-43ea-8d0b-c9eac81f4ff5\" (UID: \"146ef2d2-52b4-43ea-8d0b-c9eac81f4ff5\") " Nov 01 20:51:38.269879 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:38.269859 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/146ef2d2-52b4-43ea-8d0b-c9eac81f4ff5-maas-api-openshift-ca" pod "146ef2d2-52b4-43ea-8d0b-c9eac81f4ff5" (UID: "146ef2d2-52b4-43ea-8d0b-c9eac81f4ff5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:51:38.270329 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:38.270307 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/146ef2d2-52b4-43ea-8d0b-c9eac81f4ff5-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:38.271688 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:38.271663 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/146ef2d2-52b4-43ea-8d0b-c9eac81f4ff5-maas-api-xks-ca" pod "146ef2d2-52b4-43ea-8d0b-c9eac81f4ff5" (UID: "146ef2d2-52b4-43ea-8d0b-c9eac81f4ff5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:51:38.271817 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:38.271799 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/146ef2d2-52b4-43ea-8d0b-c9eac81f4ff5-kube-api-access-7k6ft" pod "146ef2d2-52b4-43ea-8d0b-c9eac81f4ff5" (UID: "146ef2d2-52b4-43ea-8d0b-c9eac81f4ff5"). InnerVolumeSpecName "kube-api-access-7k6ft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:51:38.372177 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:38.372145 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/146ef2d2-52b4-43ea-8d0b-c9eac81f4ff5-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:38.372177 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:38.372173 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k6ft\" (UniqueName: \"kubernetes.io/projected/146ef2d2-52b4-43ea-8d0b-c9eac81f4ff5-kube-api-access-7k6ft\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:39.028492 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:39.028460 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fcc929e-fgr7c" Nov 01 20:51:39.028880 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:39.028494 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fcc929e-fgr7c" event={"ID":"146ef2d2-52b4-43ea-8d0b-c9eac81f4ff5","Type":"ContainerDied","Data":"908ef5abb954e78402fbf4163eab960721e051a98d2884240ba2a1a1935865e2"} Nov 01 20:51:39.028880 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:39.028548 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="908ef5abb954e78402fbf4163eab960721e051a98d2884240ba2a1a1935865e2" Nov 01 20:51:39.031179 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:39.031155 4484 generic.go:356] "Generic (PLEG): container finished" podID="c2e9e863-ca7f-4f6f-9943-557293100f33" containerID="f68543a4be9c89f1cf16d5dd15cea72562f324c3063b356926c91efbdfb5ffcd" exitCode=0 Nov 01 20:51:39.031301 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:39.031238 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70bcb6f1-fbph2" event={"ID":"c2e9e863-ca7f-4f6f-9943-557293100f33","Type":"ContainerDied","Data":"f68543a4be9c89f1cf16d5dd15cea72562f324c3063b356926c91efbdfb5ffcd"} Nov 01 20:51:39.200623 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:39.200594 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e407a6a4-55qlv" Nov 01 20:51:39.285207 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:39.285172 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95b4299f-f0db-4901-b902-9acc154edc3f-kube-api-access-rgcpt\" (UniqueName: \"kubernetes.io/projected/95b4299f-f0db-4901-b902-9acc154edc3f-kube-api-access-rgcpt\") pod \"95b4299f-f0db-4901-b902-9acc154edc3f\" (UID: \"95b4299f-f0db-4901-b902-9acc154edc3f\") " Nov 01 20:51:39.285396 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:39.285240 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95b4299f-f0db-4901-b902-9acc154edc3f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95b4299f-f0db-4901-b902-9acc154edc3f-maas-api-openshift-ca\") pod \"95b4299f-f0db-4901-b902-9acc154edc3f\" (UID: \"95b4299f-f0db-4901-b902-9acc154edc3f\") " Nov 01 20:51:39.285396 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:39.285358 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95b4299f-f0db-4901-b902-9acc154edc3f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95b4299f-f0db-4901-b902-9acc154edc3f-maas-api-xks-ca\") pod \"95b4299f-f0db-4901-b902-9acc154edc3f\" (UID: \"95b4299f-f0db-4901-b902-9acc154edc3f\") " Nov 01 20:51:39.285639 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:39.285616 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95b4299f-f0db-4901-b902-9acc154edc3f-maas-api-openshift-ca" pod "95b4299f-f0db-4901-b902-9acc154edc3f" (UID: "95b4299f-f0db-4901-b902-9acc154edc3f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:51:39.286047 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:39.286027 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95b4299f-f0db-4901-b902-9acc154edc3f-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:39.287475 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:39.287443 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95b4299f-f0db-4901-b902-9acc154edc3f-maas-api-xks-ca" pod "95b4299f-f0db-4901-b902-9acc154edc3f" (UID: "95b4299f-f0db-4901-b902-9acc154edc3f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:51:39.287475 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:39.287456 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95b4299f-f0db-4901-b902-9acc154edc3f-kube-api-access-rgcpt" pod "95b4299f-f0db-4901-b902-9acc154edc3f" (UID: "95b4299f-f0db-4901-b902-9acc154edc3f"). InnerVolumeSpecName "kube-api-access-rgcpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:51:39.387936 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:39.387906 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgcpt\" (UniqueName: \"kubernetes.io/projected/95b4299f-f0db-4901-b902-9acc154edc3f-kube-api-access-rgcpt\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:39.387936 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:39.387935 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95b4299f-f0db-4901-b902-9acc154edc3f-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:39.887502 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:39.887471 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:51:40.041510 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:40.041470 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e407a6a4-55qlv" event={"ID":"95b4299f-f0db-4901-b902-9acc154edc3f","Type":"ContainerDied","Data":"14f5d9f04ef80897e4f8e773aae2c267e3748eaccbc15074d3985c0990152289"} Nov 01 20:51:40.041967 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:40.041532 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14f5d9f04ef80897e4f8e773aae2c267e3748eaccbc15074d3985c0990152289" Nov 01 20:51:40.041967 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:40.041485 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e407a6a4-55qlv" Nov 01 20:51:40.199322 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:40.199286 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53208b70-czwcx"] Nov 01 20:51:40.200440 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:40.200415 4484 cpu_manager.go:414] "containerMap: removing container" podUID="146ef2d2-52b4-43ea-8d0b-c9eac81f4ff5" containerName="revoke-keys" Nov 01 20:51:40.200440 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:40.200440 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="146ef2d2-52b4-43ea-8d0b-c9eac81f4ff5" containerName="revoke-keys" Nov 01 20:51:40.200573 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:40.200490 4484 cpu_manager.go:414] "containerMap: removing container" podUID="95b4299f-f0db-4901-b902-9acc154edc3f" containerName="revoke-keys" Nov 01 20:51:40.200573 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:40.200498 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95b4299f-f0db-4901-b902-9acc154edc3f" containerName="revoke-keys" Nov 01 20:51:40.200688 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:40.200676 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="146ef2d2-52b4-43ea-8d0b-c9eac81f4ff5" containerName="revoke-keys" Nov 01 20:51:40.200757 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:40.200696 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="95b4299f-f0db-4901-b902-9acc154edc3f" containerName="revoke-keys" Nov 01 20:51:40.204618 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:40.204599 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53208b70-czwcx" Nov 01 20:51:40.220152 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:40.220126 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70bcb6f1-fbph2" Nov 01 20:51:40.286076 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:40.286034 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53208b70-czwcx"] Nov 01 20:51:40.301460 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:40.301433 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2e9e863-ca7f-4f6f-9943-557293100f33-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2e9e863-ca7f-4f6f-9943-557293100f33-maas-api-xks-ca\") pod \"c2e9e863-ca7f-4f6f-9943-557293100f33\" (UID: \"c2e9e863-ca7f-4f6f-9943-557293100f33\") " Nov 01 20:51:40.301599 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:40.301486 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2e9e863-ca7f-4f6f-9943-557293100f33-kube-api-access-4fr76\" (UniqueName: \"kubernetes.io/projected/c2e9e863-ca7f-4f6f-9943-557293100f33-kube-api-access-4fr76\") pod \"c2e9e863-ca7f-4f6f-9943-557293100f33\" (UID: \"c2e9e863-ca7f-4f6f-9943-557293100f33\") " Nov 01 20:51:40.301599 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:40.301540 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2e9e863-ca7f-4f6f-9943-557293100f33-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2e9e863-ca7f-4f6f-9943-557293100f33-maas-api-openshift-ca\") pod \"c2e9e863-ca7f-4f6f-9943-557293100f33\" (UID: \"c2e9e863-ca7f-4f6f-9943-557293100f33\") " Nov 01 20:51:40.301963 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:40.301936 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ca4d667-38b7-4e87-9f63-cf66f78ca2bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53208b70-czwcx\" (UID: \"2ca4d667-38b7-4e87-9f63-cf66f78ca2bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53208b70-czwcx" Nov 01 20:51:40.302086 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:40.302036 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2e9e863-ca7f-4f6f-9943-557293100f33-maas-api-openshift-ca" pod "c2e9e863-ca7f-4f6f-9943-557293100f33" (UID: "c2e9e863-ca7f-4f6f-9943-557293100f33"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:51:40.302166 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:40.302111 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5tjh\" (UniqueName: \"kubernetes.io/projected/2ca4d667-38b7-4e87-9f63-cf66f78ca2bd-kube-api-access-z5tjh\") pod \"maas-api-revoke-sub-53208b70-czwcx\" (UID: \"2ca4d667-38b7-4e87-9f63-cf66f78ca2bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53208b70-czwcx" Nov 01 20:51:40.302375 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:40.302352 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ca4d667-38b7-4e87-9f63-cf66f78ca2bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53208b70-czwcx\" (UID: \"2ca4d667-38b7-4e87-9f63-cf66f78ca2bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53208b70-czwcx" Nov 01 20:51:40.302491 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:40.302472 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2e9e863-ca7f-4f6f-9943-557293100f33-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:40.305138 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:40.305113 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2e9e863-ca7f-4f6f-9943-557293100f33-maas-api-xks-ca" pod "c2e9e863-ca7f-4f6f-9943-557293100f33" (UID: "c2e9e863-ca7f-4f6f-9943-557293100f33"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:51:40.305208 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:40.305160 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2e9e863-ca7f-4f6f-9943-557293100f33-kube-api-access-4fr76" pod "c2e9e863-ca7f-4f6f-9943-557293100f33" (UID: "c2e9e863-ca7f-4f6f-9943-557293100f33"). InnerVolumeSpecName "kube-api-access-4fr76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:51:40.403412 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:40.403378 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5tjh\" (UniqueName: \"kubernetes.io/projected/2ca4d667-38b7-4e87-9f63-cf66f78ca2bd-kube-api-access-z5tjh\") pod \"maas-api-revoke-sub-53208b70-czwcx\" (UID: \"2ca4d667-38b7-4e87-9f63-cf66f78ca2bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53208b70-czwcx" Nov 01 20:51:40.403567 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:40.403556 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ca4d667-38b7-4e87-9f63-cf66f78ca2bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53208b70-czwcx\" (UID: \"2ca4d667-38b7-4e87-9f63-cf66f78ca2bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53208b70-czwcx" Nov 01 20:51:40.403606 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:40.403599 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ca4d667-38b7-4e87-9f63-cf66f78ca2bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53208b70-czwcx\" (UID: \"2ca4d667-38b7-4e87-9f63-cf66f78ca2bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53208b70-czwcx" Nov 01 20:51:40.403696 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:40.403686 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2e9e863-ca7f-4f6f-9943-557293100f33-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:40.403760 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:40.403698 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fr76\" (UniqueName: \"kubernetes.io/projected/c2e9e863-ca7f-4f6f-9943-557293100f33-kube-api-access-4fr76\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:40.404220 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:40.404196 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ca4d667-38b7-4e87-9f63-cf66f78ca2bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53208b70-czwcx\" (UID: \"2ca4d667-38b7-4e87-9f63-cf66f78ca2bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53208b70-czwcx" Nov 01 20:51:40.406127 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:40.406101 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ca4d667-38b7-4e87-9f63-cf66f78ca2bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53208b70-czwcx\" (UID: \"2ca4d667-38b7-4e87-9f63-cf66f78ca2bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53208b70-czwcx" Nov 01 20:51:40.464916 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:40.464844 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5tjh\" (UniqueName: \"kubernetes.io/projected/2ca4d667-38b7-4e87-9f63-cf66f78ca2bd-kube-api-access-z5tjh\") pod \"maas-api-revoke-sub-53208b70-czwcx\" (UID: \"2ca4d667-38b7-4e87-9f63-cf66f78ca2bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53208b70-czwcx" Nov 01 20:51:40.529794 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:40.529760 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53208b70-czwcx" Nov 01 20:51:40.907645 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:51:40.907608 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ca4d667_38b7_4e87_9f63_cf66f78ca2bd.slice/crio-f4af10f9e02fb5a6f7ed097dd8cb5b64a0ad13615731b11c2a10cd5a8fa34e63 WatchSource:0}: Error finding container f4af10f9e02fb5a6f7ed097dd8cb5b64a0ad13615731b11c2a10cd5a8fa34e63: Status 404 returned error can't find the container with id f4af10f9e02fb5a6f7ed097dd8cb5b64a0ad13615731b11c2a10cd5a8fa34e63 Nov 01 20:51:40.908825 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:40.908804 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53208b70-czwcx"] Nov 01 20:51:41.052823 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:41.052790 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53208b70-czwcx" event={"ID":"2ca4d667-38b7-4e87-9f63-cf66f78ca2bd","Type":"ContainerStarted","Data":"f4af10f9e02fb5a6f7ed097dd8cb5b64a0ad13615731b11c2a10cd5a8fa34e63"} Nov 01 20:51:41.054334 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:41.054306 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70bcb6f1-fbph2" event={"ID":"c2e9e863-ca7f-4f6f-9943-557293100f33","Type":"ContainerDied","Data":"a1ca45586ce4beaab9bf9444845da663093e3a4f6151622c29dd7366bd8f5fa4"} Nov 01 20:51:41.054461 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:41.054336 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70bcb6f1-fbph2" Nov 01 20:51:41.054461 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:41.054354 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1ca45586ce4beaab9bf9444845da663093e3a4f6151622c29dd7366bd8f5fa4" Nov 01 20:51:41.533974 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:41.533938 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:51:42.064728 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:42.064694 4484 generic.go:356] "Generic (PLEG): container finished" podID="2ca4d667-38b7-4e87-9f63-cf66f78ca2bd" containerID="7e1fe1e09d9023483aa7983f2815e0bf7d1c03612ec228ea1ce60b2dd73e877c" exitCode=0 Nov 01 20:51:42.065227 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:42.064777 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53208b70-czwcx" event={"ID":"2ca4d667-38b7-4e87-9f63-cf66f78ca2bd","Type":"ContainerDied","Data":"7e1fe1e09d9023483aa7983f2815e0bf7d1c03612ec228ea1ce60b2dd73e877c"} Nov 01 20:51:42.979733 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:42.979698 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b58f354-bczft"] Nov 01 20:51:42.980440 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:42.980425 4484 cpu_manager.go:414] "containerMap: removing container" podUID="c2e9e863-ca7f-4f6f-9943-557293100f33" containerName="revoke-keys" Nov 01 20:51:42.980508 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:42.980442 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2e9e863-ca7f-4f6f-9943-557293100f33" containerName="revoke-keys" Nov 01 20:51:42.980582 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:42.980573 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2e9e863-ca7f-4f6f-9943-557293100f33" containerName="revoke-keys" Nov 01 20:51:42.984327 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:42.984311 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b58f354-bczft" Nov 01 20:51:43.042710 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:43.042675 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b58f354-bczft"] Nov 01 20:51:43.048190 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:43.048159 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79f92\" (UniqueName: \"kubernetes.io/projected/76bbc184-657e-4bf3-a072-92d47164ce86-kube-api-access-79f92\") pod \"maas-api-revoke-sub-6b58f354-bczft\" (UID: \"76bbc184-657e-4bf3-a072-92d47164ce86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b58f354-bczft" Nov 01 20:51:43.048417 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:43.048399 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76bbc184-657e-4bf3-a072-92d47164ce86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b58f354-bczft\" (UID: \"76bbc184-657e-4bf3-a072-92d47164ce86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b58f354-bczft" Nov 01 20:51:43.048485 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:43.048472 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76bbc184-657e-4bf3-a072-92d47164ce86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b58f354-bczft\" (UID: \"76bbc184-657e-4bf3-a072-92d47164ce86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b58f354-bczft" Nov 01 20:51:43.103912 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:43.103842 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ce9c445-gwxch"] Nov 01 20:51:43.114507 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:43.114470 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ce9c445-gwxch"] Nov 01 20:51:43.150436 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:43.150407 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76bbc184-657e-4bf3-a072-92d47164ce86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b58f354-bczft\" (UID: \"76bbc184-657e-4bf3-a072-92d47164ce86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b58f354-bczft" Nov 01 20:51:43.150591 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:43.150509 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79f92\" (UniqueName: \"kubernetes.io/projected/76bbc184-657e-4bf3-a072-92d47164ce86-kube-api-access-79f92\") pod \"maas-api-revoke-sub-6b58f354-bczft\" (UID: \"76bbc184-657e-4bf3-a072-92d47164ce86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b58f354-bczft" Nov 01 20:51:43.150774 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:43.150752 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76bbc184-657e-4bf3-a072-92d47164ce86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b58f354-bczft\" (UID: \"76bbc184-657e-4bf3-a072-92d47164ce86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b58f354-bczft" Nov 01 20:51:43.151639 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:43.151607 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76bbc184-657e-4bf3-a072-92d47164ce86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b58f354-bczft\" (UID: \"76bbc184-657e-4bf3-a072-92d47164ce86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b58f354-bczft" Nov 01 20:51:43.154213 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:43.154188 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76bbc184-657e-4bf3-a072-92d47164ce86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b58f354-bczft\" (UID: \"76bbc184-657e-4bf3-a072-92d47164ce86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b58f354-bczft" Nov 01 20:51:43.210852 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:43.210817 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79f92\" (UniqueName: \"kubernetes.io/projected/76bbc184-657e-4bf3-a072-92d47164ce86-kube-api-access-79f92\") pod \"maas-api-revoke-sub-6b58f354-bczft\" (UID: \"76bbc184-657e-4bf3-a072-92d47164ce86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b58f354-bczft" Nov 01 20:51:43.251897 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:43.251869 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53208b70-czwcx" Nov 01 20:51:43.298759 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:43.298725 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b58f354-bczft" Nov 01 20:51:43.353624 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:43.353522 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ca4d667-38b7-4e87-9f63-cf66f78ca2bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ca4d667-38b7-4e87-9f63-cf66f78ca2bd-maas-api-xks-ca\") pod \"2ca4d667-38b7-4e87-9f63-cf66f78ca2bd\" (UID: \"2ca4d667-38b7-4e87-9f63-cf66f78ca2bd\") " Nov 01 20:51:43.353624 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:43.353635 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ca4d667-38b7-4e87-9f63-cf66f78ca2bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ca4d667-38b7-4e87-9f63-cf66f78ca2bd-maas-api-openshift-ca\") pod \"2ca4d667-38b7-4e87-9f63-cf66f78ca2bd\" (UID: \"2ca4d667-38b7-4e87-9f63-cf66f78ca2bd\") " Nov 01 20:51:43.354277 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:43.354045 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ca4d667-38b7-4e87-9f63-cf66f78ca2bd-maas-api-openshift-ca" pod "2ca4d667-38b7-4e87-9f63-cf66f78ca2bd" (UID: "2ca4d667-38b7-4e87-9f63-cf66f78ca2bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:51:43.354277 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:43.354212 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ca4d667-38b7-4e87-9f63-cf66f78ca2bd-kube-api-access-z5tjh\" (UniqueName: \"kubernetes.io/projected/2ca4d667-38b7-4e87-9f63-cf66f78ca2bd-kube-api-access-z5tjh\") pod \"2ca4d667-38b7-4e87-9f63-cf66f78ca2bd\" (UID: \"2ca4d667-38b7-4e87-9f63-cf66f78ca2bd\") " Nov 01 20:51:43.355062 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:43.355025 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ca4d667-38b7-4e87-9f63-cf66f78ca2bd-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:43.359597 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:43.359562 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ca4d667-38b7-4e87-9f63-cf66f78ca2bd-kube-api-access-z5tjh" pod "2ca4d667-38b7-4e87-9f63-cf66f78ca2bd" (UID: "2ca4d667-38b7-4e87-9f63-cf66f78ca2bd"). InnerVolumeSpecName "kube-api-access-z5tjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:51:43.359919 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:43.359897 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ca4d667-38b7-4e87-9f63-cf66f78ca2bd-maas-api-xks-ca" pod "2ca4d667-38b7-4e87-9f63-cf66f78ca2bd" (UID: "2ca4d667-38b7-4e87-9f63-cf66f78ca2bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:51:43.387689 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:43.387654 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ed3c297-00e9-48ed-992e-84df9d758a06" path="/var/lib/kubelet/pods/3ed3c297-00e9-48ed-992e-84df9d758a06/volumes" Nov 01 20:51:43.457622 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:43.457591 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5tjh\" (UniqueName: \"kubernetes.io/projected/2ca4d667-38b7-4e87-9f63-cf66f78ca2bd-kube-api-access-z5tjh\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:43.457622 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:43.457616 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ca4d667-38b7-4e87-9f63-cf66f78ca2bd-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:43.677057 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:51:43.677027 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76bbc184_657e_4bf3_a072_92d47164ce86.slice/crio-84d536c9a4e8ad34cb03c651a39ecbaf5570cc8e053509a4f9764c6417fed124 WatchSource:0}: Error finding container 84d536c9a4e8ad34cb03c651a39ecbaf5570cc8e053509a4f9764c6417fed124: Status 404 returned error can't find the container with id 84d536c9a4e8ad34cb03c651a39ecbaf5570cc8e053509a4f9764c6417fed124 Nov 01 20:51:43.678675 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:43.678591 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b58f354-bczft"] Nov 01 20:51:44.084942 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:44.084903 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b58f354-bczft" event={"ID":"76bbc184-657e-4bf3-a072-92d47164ce86","Type":"ContainerStarted","Data":"3aaf04809a886b4f96cd8303331e933e722909e72caab7599e702f8f696e7198"} Nov 01 20:51:44.085172 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:44.084970 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b58f354-bczft" event={"ID":"76bbc184-657e-4bf3-a072-92d47164ce86","Type":"ContainerStarted","Data":"84d536c9a4e8ad34cb03c651a39ecbaf5570cc8e053509a4f9764c6417fed124"} Nov 01 20:51:44.086543 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:44.086526 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53208b70-czwcx" Nov 01 20:51:44.086650 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:44.086568 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53208b70-czwcx" event={"ID":"2ca4d667-38b7-4e87-9f63-cf66f78ca2bd","Type":"ContainerDied","Data":"f4af10f9e02fb5a6f7ed097dd8cb5b64a0ad13615731b11c2a10cd5a8fa34e63"} Nov 01 20:51:44.086650 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:44.086610 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4af10f9e02fb5a6f7ed097dd8cb5b64a0ad13615731b11c2a10cd5a8fa34e63" Nov 01 20:51:44.200075 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:44.199988 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b58f354-bczft" podStartSLOduration=2.199973751 podStartE2EDuration="2.199973751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:51:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:51:44.199248231 +0000 UTC m=+1254.976888389" watchObservedRunningTime="2026-11-01 20:51:44.199973751 +0000 UTC m=+1254.977613941" Nov 01 20:51:45.100073 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:45.100034 4484 generic.go:356] "Generic (PLEG): container finished" podID="76bbc184-657e-4bf3-a072-92d47164ce86" containerID="3aaf04809a886b4f96cd8303331e933e722909e72caab7599e702f8f696e7198" exitCode=0 Nov 01 20:51:45.100280 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:45.100119 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b58f354-bczft" event={"ID":"76bbc184-657e-4bf3-a072-92d47164ce86","Type":"ContainerDied","Data":"3aaf04809a886b4f96cd8303331e933e722909e72caab7599e702f8f696e7198"} Nov 01 20:51:45.638367 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:45.638336 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2607e260-dgmfz"] Nov 01 20:51:45.639199 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:45.639179 4484 cpu_manager.go:414] "containerMap: removing container" podUID="2ca4d667-38b7-4e87-9f63-cf66f78ca2bd" containerName="revoke-keys" Nov 01 20:51:45.639199 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:45.639199 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ca4d667-38b7-4e87-9f63-cf66f78ca2bd" containerName="revoke-keys" Nov 01 20:51:45.639317 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:45.639312 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ca4d667-38b7-4e87-9f63-cf66f78ca2bd" containerName="revoke-keys" Nov 01 20:51:45.643324 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:45.643304 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2607e260-dgmfz" Nov 01 20:51:45.699527 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:45.699496 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2607e260-dgmfz"] Nov 01 20:51:45.776420 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:51:45.776381 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 20:51:45.794395 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:45.794356 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q65wr\" (UniqueName: \"kubernetes.io/projected/004f0b18-2fdc-443f-9b43-91b998037058-kube-api-access-q65wr\") pod \"maas-api-revoke-sub-2607e260-dgmfz\" (UID: \"004f0b18-2fdc-443f-9b43-91b998037058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2607e260-dgmfz" Nov 01 20:51:45.794558 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:45.794495 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/004f0b18-2fdc-443f-9b43-91b998037058-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2607e260-dgmfz\" (UID: \"004f0b18-2fdc-443f-9b43-91b998037058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2607e260-dgmfz" Nov 01 20:51:45.794793 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:45.794770 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/004f0b18-2fdc-443f-9b43-91b998037058-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2607e260-dgmfz\" (UID: \"004f0b18-2fdc-443f-9b43-91b998037058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2607e260-dgmfz" Nov 01 20:51:45.896273 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:45.896191 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q65wr\" (UniqueName: \"kubernetes.io/projected/004f0b18-2fdc-443f-9b43-91b998037058-kube-api-access-q65wr\") pod \"maas-api-revoke-sub-2607e260-dgmfz\" (UID: \"004f0b18-2fdc-443f-9b43-91b998037058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2607e260-dgmfz" Nov 01 20:51:45.896452 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:45.896274 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/004f0b18-2fdc-443f-9b43-91b998037058-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2607e260-dgmfz\" (UID: \"004f0b18-2fdc-443f-9b43-91b998037058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2607e260-dgmfz" Nov 01 20:51:45.896518 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:45.896459 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/004f0b18-2fdc-443f-9b43-91b998037058-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2607e260-dgmfz\" (UID: \"004f0b18-2fdc-443f-9b43-91b998037058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2607e260-dgmfz" Nov 01 20:51:45.897119 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:45.897095 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/004f0b18-2fdc-443f-9b43-91b998037058-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2607e260-dgmfz\" (UID: \"004f0b18-2fdc-443f-9b43-91b998037058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2607e260-dgmfz" Nov 01 20:51:45.898702 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:45.898681 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/004f0b18-2fdc-443f-9b43-91b998037058-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2607e260-dgmfz\" (UID: \"004f0b18-2fdc-443f-9b43-91b998037058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2607e260-dgmfz" Nov 01 20:51:45.953332 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:45.953309 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q65wr\" (UniqueName: \"kubernetes.io/projected/004f0b18-2fdc-443f-9b43-91b998037058-kube-api-access-q65wr\") pod \"maas-api-revoke-sub-2607e260-dgmfz\" (UID: \"004f0b18-2fdc-443f-9b43-91b998037058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2607e260-dgmfz" Nov 01 20:51:45.958160 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:45.958138 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2607e260-dgmfz" Nov 01 20:51:46.290033 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:46.276153 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b58f354-bczft" Nov 01 20:51:46.385451 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:46.381752 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:51:46.413140 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:46.411296 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76bbc184-657e-4bf3-a072-92d47164ce86-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76bbc184-657e-4bf3-a072-92d47164ce86-maas-api-xks-ca\") pod \"76bbc184-657e-4bf3-a072-92d47164ce86\" (UID: \"76bbc184-657e-4bf3-a072-92d47164ce86\") " Nov 01 20:51:46.413140 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:46.411380 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76bbc184-657e-4bf3-a072-92d47164ce86-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76bbc184-657e-4bf3-a072-92d47164ce86-maas-api-openshift-ca\") pod \"76bbc184-657e-4bf3-a072-92d47164ce86\" (UID: \"76bbc184-657e-4bf3-a072-92d47164ce86\") " Nov 01 20:51:46.413140 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:46.411738 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76bbc184-657e-4bf3-a072-92d47164ce86-kube-api-access-79f92\" (UniqueName: \"kubernetes.io/projected/76bbc184-657e-4bf3-a072-92d47164ce86-kube-api-access-79f92\") pod \"76bbc184-657e-4bf3-a072-92d47164ce86\" (UID: \"76bbc184-657e-4bf3-a072-92d47164ce86\") " Nov 01 20:51:46.417689 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:46.416868 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76bbc184-657e-4bf3-a072-92d47164ce86-maas-api-openshift-ca" pod "76bbc184-657e-4bf3-a072-92d47164ce86" (UID: "76bbc184-657e-4bf3-a072-92d47164ce86"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:51:46.418234 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:46.418199 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76bbc184-657e-4bf3-a072-92d47164ce86-maas-api-xks-ca" pod "76bbc184-657e-4bf3-a072-92d47164ce86" (UID: "76bbc184-657e-4bf3-a072-92d47164ce86"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:51:46.420666 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:46.420541 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76bbc184-657e-4bf3-a072-92d47164ce86-kube-api-access-79f92" pod "76bbc184-657e-4bf3-a072-92d47164ce86" (UID: "76bbc184-657e-4bf3-a072-92d47164ce86"). InnerVolumeSpecName "kube-api-access-79f92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:51:46.424832 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:46.424769 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2607e260-dgmfz"] Nov 01 20:51:46.514625 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:46.514584 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79f92\" (UniqueName: \"kubernetes.io/projected/76bbc184-657e-4bf3-a072-92d47164ce86-kube-api-access-79f92\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:46.514625 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:46.514625 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76bbc184-657e-4bf3-a072-92d47164ce86-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:46.514625 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:46.514635 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76bbc184-657e-4bf3-a072-92d47164ce86-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:47.123465 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:47.123369 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2607e260-dgmfz" event={"ID":"004f0b18-2fdc-443f-9b43-91b998037058","Type":"ContainerStarted","Data":"723cd48916e6b719f4fc6ebf8c2a8b85dd05268a945a82ee0e827353d89364a3"} Nov 01 20:51:47.123465 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:47.123430 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2607e260-dgmfz" event={"ID":"004f0b18-2fdc-443f-9b43-91b998037058","Type":"ContainerStarted","Data":"8bd8f13594fd517705e5bbf51f7fc69ada02323ac16363e6923dc7b8798084c5"} Nov 01 20:51:47.124746 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:47.124725 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b58f354-bczft" Nov 01 20:51:47.124875 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:47.124763 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b58f354-bczft" event={"ID":"76bbc184-657e-4bf3-a072-92d47164ce86","Type":"ContainerDied","Data":"84d536c9a4e8ad34cb03c651a39ecbaf5570cc8e053509a4f9764c6417fed124"} Nov 01 20:51:47.124875 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:47.124803 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84d536c9a4e8ad34cb03c651a39ecbaf5570cc8e053509a4f9764c6417fed124" Nov 01 20:51:47.248309 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:47.248258 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2607e260-dgmfz" podStartSLOduration=2.248243681 podStartE2EDuration="2.248243681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:51:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:51:47.246522737 +0000 UTC m=+1258.024162873" watchObservedRunningTime="2026-11-01 20:51:47.248243681 +0000 UTC m=+1258.025883816" Nov 01 20:51:48.222738 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:48.221125 4484 generic.go:356] "Generic (PLEG): container finished" podID="004f0b18-2fdc-443f-9b43-91b998037058" containerID="723cd48916e6b719f4fc6ebf8c2a8b85dd05268a945a82ee0e827353d89364a3" exitCode=0 Nov 01 20:51:48.222738 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:48.221207 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2607e260-dgmfz" event={"ID":"004f0b18-2fdc-443f-9b43-91b998037058","Type":"ContainerDied","Data":"723cd48916e6b719f4fc6ebf8c2a8b85dd05268a945a82ee0e827353d89364a3"} Nov 01 20:51:49.722072 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:49.695280 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" podUID="fe8307db-264d-4b8a-b097-f180bc255a9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:49.796071 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:49.774838 4484 scope.go:122] "RemoveContainer" containerID="06ffa97ef20777ec7985cf9af9c8d2186c891dc6bf6e55ef12120f009d8b4f27" Nov 01 20:51:49.917301 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:49.917264 4484 scope.go:122] "RemoveContainer" containerID="355fc072c5aa5cc2c2e94123d319d22f10b0cf82d1c8dfe7023320c1c6d973d9" Nov 01 20:51:50.117023 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:50.037574 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" podUID="a65d9fd8-6193-4d03-9a3c-123f48b5dcdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:50.117570 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:50.117527 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" podUID="311f9699-6585-4a2c-8c20-e6b696b2f857" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:50.159627 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:50.159580 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" podUID="e175b298-2c23-45d4-8f9c-0c9b9f6aab92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:50.267950 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:50.267915 4484 scope.go:122] "RemoveContainer" containerID="ab015295258d73e4d54c396fe05bf46f76537e0b764aef0d8ea3d3b556ce1df1" Nov 01 20:51:50.763136 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:50.763091 4484 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" podUID="9f19be29-2c84-4ae4-9674-acca4905f4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/health\": context deadline exceeded" Nov 01 20:51:50.931086 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:50.813358 4484 scope.go:122] "RemoveContainer" containerID="312bf148a6b8e8c89912c215d67b686484ce8f8b16a69e7a8a96fd5430685806" Nov 01 20:51:50.931086 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:50.813717 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" podUID="9f19be29-2c84-4ae4-9674-acca4905f4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 01 20:51:50.931086 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:50.816604 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" podUID="fe8307db-264d-4b8a-b097-f180bc255a9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 01 20:51:50.931086 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:50.856911 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" Nov 01 20:51:50.931086 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:50.863621 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" podUID="17a9ac24-19cb-48c4-99b5-d74ce8aac0a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 01 20:51:51.083513 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:51.083470 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" podUID="70b4dacf-b3ac-46ec-99e8-a44f05587e7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 01 20:51:51.119591 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:51.119538 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" podUID="a65d9fd8-6193-4d03-9a3c-123f48b5dcdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Nov 01 20:51:51.151265 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:51.126642 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" Nov 01 20:51:51.582020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:51.130077 4484 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:51:51.627314 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:51.627271 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq" podUID="50906dd8-8d76-4fcb-8415-f8f32cd47ffc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:51.627619 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:51.169581 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" podUID="e175b298-2c23-45d4-8f9c-0c9b9f6aab92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:51.635064 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:51.147738 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" podUID="311f9699-6585-4a2c-8c20-e6b696b2f857" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:51.685022 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:51.673141 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" Nov 01 20:51:51.710024 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:51.685351 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" Nov 01 20:51:51.847528 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:51.845140 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" podUID="9f19be29-2c84-4ae4-9674-acca4905f4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 01 20:51:51.847528 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:51.845627 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" Nov 01 20:51:51.867152 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:51.867105 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" podUID="fe8307db-264d-4b8a-b097-f180bc255a9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:51.997072 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:51.869048 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" podUID="17a9ac24-19cb-48c4-99b5-d74ce8aac0a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:52.112204 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:52.112096 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" podUID="70b4dacf-b3ac-46ec-99e8-a44f05587e7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:52.122332 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:52.121682 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" podUID="a65d9fd8-6193-4d03-9a3c-123f48b5dcdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Nov 01 20:51:52.335070 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:52.315322 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2607e260-dgmfz" Nov 01 20:51:52.335070 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:52.327559 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/004f0b18-2fdc-443f-9b43-91b998037058-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/004f0b18-2fdc-443f-9b43-91b998037058-maas-api-openshift-ca\") pod \"004f0b18-2fdc-443f-9b43-91b998037058\" (UID: \"004f0b18-2fdc-443f-9b43-91b998037058\") " Nov 01 20:51:52.352258 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:52.335692 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/004f0b18-2fdc-443f-9b43-91b998037058-kube-api-access-q65wr\" (UniqueName: \"kubernetes.io/projected/004f0b18-2fdc-443f-9b43-91b998037058-kube-api-access-q65wr\") pod \"004f0b18-2fdc-443f-9b43-91b998037058\" (UID: \"004f0b18-2fdc-443f-9b43-91b998037058\") " Nov 01 20:51:52.363933 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:52.362602 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/004f0b18-2fdc-443f-9b43-91b998037058-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/004f0b18-2fdc-443f-9b43-91b998037058-maas-api-xks-ca\") pod \"004f0b18-2fdc-443f-9b43-91b998037058\" (UID: \"004f0b18-2fdc-443f-9b43-91b998037058\") " Nov 01 20:51:52.463643 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:52.463604 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/004f0b18-2fdc-443f-9b43-91b998037058-maas-api-openshift-ca" pod "004f0b18-2fdc-443f-9b43-91b998037058" (UID: "004f0b18-2fdc-443f-9b43-91b998037058"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:51:52.596045 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:52.569439 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/004f0b18-2fdc-443f-9b43-91b998037058-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:52.596045 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:52.569581 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/004f0b18-2fdc-443f-9b43-91b998037058-maas-api-xks-ca" pod "004f0b18-2fdc-443f-9b43-91b998037058" (UID: "004f0b18-2fdc-443f-9b43-91b998037058"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:51:52.596045 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:52.581812 4484 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" podUID="a65d9fd8-6193-4d03-9a3c-123f48b5dcdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/health\": context deadline exceeded" Nov 01 20:51:52.764943 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:52.760972 4484 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" podUID="70b4dacf-b3ac-46ec-99e8-a44f05587e7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:52.825545 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:52.822779 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" podUID="e175b298-2c23-45d4-8f9c-0c9b9f6aab92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": context deadline exceeded" Nov 01 20:51:52.825545 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:52.823493 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" podUID="311f9699-6585-4a2c-8c20-e6b696b2f857" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 01 20:51:52.839015 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:52.838830 4484 patch_prober.go:29] interesting pod/metrics-server-84f999b6b9-g8p4q container/metrics-server namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.134.0.17:10250/livez\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:51:52.839015 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:52.838918 4484 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/metrics-server-84f999b6b9-g8p4q" podUID="8995aa4c-ac0d-4f43-aa03-bebced1e08e0" containerName="metrics-server" probeResult="failure" output="Get \"https://10.134.0.17:10250/livez\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:52.855916 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:52.854200 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" podUID="9f19be29-2c84-4ae4-9674-acca4905f4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:52.860156 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:52.857712 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/004f0b18-2fdc-443f-9b43-91b998037058-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:52.860640 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:52.860526 4484 scope.go:122] "RemoveContainer" containerID="5889a5b844671e651c59fe07269fba75e3198fe99b1a33fa8e629cb0f39c190a" Nov 01 20:51:52.885381 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:52.868291 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" podUID="fe8307db-264d-4b8a-b097-f180bc255a9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 01 20:51:52.915580 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:52.915527 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" podUID="17a9ac24-19cb-48c4-99b5-d74ce8aac0a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:52.980282 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:52.980240 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/004f0b18-2fdc-443f-9b43-91b998037058-kube-api-access-q65wr" pod "004f0b18-2fdc-443f-9b43-91b998037058" (UID: "004f0b18-2fdc-443f-9b43-91b998037058"). InnerVolumeSpecName "kube-api-access-q65wr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:51:53.104103 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:51:53.059265 4484 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.191s" Nov 01 20:51:53.104103 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:53.059467 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" Nov 01 20:51:53.104103 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:53.059505 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" Nov 01 20:51:53.141153 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:53.141114 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" podUID="70b4dacf-b3ac-46ec-99e8-a44f05587e7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:53.186035 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:53.181479 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" podUID="a65d9fd8-6193-4d03-9a3c-123f48b5dcdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:53.481165 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:53.465139 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:51:53.660836 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:53.615972 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q65wr\" (UniqueName: \"kubernetes.io/projected/004f0b18-2fdc-443f-9b43-91b998037058-kube-api-access-q65wr\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:53.704226 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:53.689435 4484 scope.go:122] "RemoveContainer" containerID="38364e9a9caa053c7725f945528c63de0a4954b68063236647bb05c742a1ffbe" Nov 01 20:51:53.889095 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:53.867369 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" podUID="e175b298-2c23-45d4-8f9c-0c9b9f6aab92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": context deadline exceeded" Nov 01 20:51:53.889095 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:53.868360 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" podUID="311f9699-6585-4a2c-8c20-e6b696b2f857" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 01 20:51:53.923096 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:53.921907 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" podUID="fe8307db-264d-4b8a-b097-f180bc255a9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:54.118899 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:54.087870 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" podUID="17a9ac24-19cb-48c4-99b5-d74ce8aac0a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 01 20:51:54.118899 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:54.118792 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" podUID="9f19be29-2c84-4ae4-9674-acca4905f4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:54.252936 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:54.245105 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" podUID="70b4dacf-b3ac-46ec-99e8-a44f05587e7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:54.342791 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:54.315446 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" podUID="a65d9fd8-6193-4d03-9a3c-123f48b5dcdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Nov 01 20:51:54.644761 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:54.644654 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2607e260-dgmfz" event={"ID":"004f0b18-2fdc-443f-9b43-91b998037058","Type":"ContainerDied","Data":"8bd8f13594fd517705e5bbf51f7fc69ada02323ac16363e6923dc7b8798084c5"} Nov 01 20:51:54.645030 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:54.644986 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bd8f13594fd517705e5bbf51f7fc69ada02323ac16363e6923dc7b8798084c5" Nov 01 20:51:54.645256 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:54.645242 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2607e260-dgmfz" Nov 01 20:51:54.678098 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:54.665836 4484 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" podUID="fe8307db-264d-4b8a-b097-f180bc255a9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:54.898132 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:54.892134 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" podUID="311f9699-6585-4a2c-8c20-e6b696b2f857" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 01 20:51:54.898132 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:54.893805 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" podUID="e175b298-2c23-45d4-8f9c-0c9b9f6aab92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": context deadline exceeded" Nov 01 20:51:54.898132 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:54.895103 4484 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:51:54.898132 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:54.895141 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv" podUID="b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:54.898132 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:54.895204 4484 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:51:54.898132 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:54.895224 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz" podUID="e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:54.953689 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:54.935916 4484 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-dt84g" podUID="3060c0d1-44f0-48ec-843f-d6e8c0833982" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.22:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:54.953689 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:54.942858 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" podUID="fe8307db-264d-4b8a-b097-f180bc255a9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 01 20:51:55.125834 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:55.125802 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" Nov 01 20:51:55.167015 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:55.127516 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" podUID="9f19be29-2c84-4ae4-9674-acca4905f4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:55.169131 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:55.139149 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" podUID="17a9ac24-19cb-48c4-99b5-d74ce8aac0a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 01 20:51:55.245973 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:55.245928 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" podUID="70b4dacf-b3ac-46ec-99e8-a44f05587e7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:55.315869 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:55.315298 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" Nov 01 20:51:55.401020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:55.393681 4484 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:51:55.641022 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:55.615114 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:51:55.777022 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:55.748131 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf" podUID="ef2f968c-433d-49f8-8fc4-7c8551d647fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:55.945510 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:55.945402 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" podUID="fe8307db-264d-4b8a-b097-f180bc255a9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 01 20:51:55.982243 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:55.982211 4484 scope.go:122] "RemoveContainer" containerID="a346337f718291adb82a33b9ef360d3eaa2314580ce99f180567b3ab72f6cd8c" Nov 01 20:51:56.377231 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:56.355181 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" podUID="a65d9fd8-6193-4d03-9a3c-123f48b5dcdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Nov 01 20:51:56.586029 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:56.585560 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" podUID="e175b298-2c23-45d4-8f9c-0c9b9f6aab92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:56.714051 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:56.713521 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" podUID="17a9ac24-19cb-48c4-99b5-d74ce8aac0a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:51:56.720282 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:51:56.720246 4484 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.067s" Nov 01 20:51:56.750042 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:56.744947 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" Nov 01 20:51:56.750042 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:56.745151 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" Nov 01 20:51:56.750042 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:56.745217 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" Nov 01 20:51:56.784071 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:56.781792 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" Nov 01 20:51:56.871374 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:56.871311 4484 scope.go:122] "RemoveContainer" containerID="33606dc0bbd0a7596e351dce4be71457e29f1648345463715ec13bdde553bdab" Nov 01 20:51:56.945911 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:56.945864 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" Nov 01 20:51:56.985305 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:56.984776 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e42be329-mvc5r"] Nov 01 20:51:56.988091 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:56.985845 4484 cpu_manager.go:414] "containerMap: removing container" podUID="004f0b18-2fdc-443f-9b43-91b998037058" containerName="revoke-keys" Nov 01 20:51:56.988091 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:56.985867 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="004f0b18-2fdc-443f-9b43-91b998037058" containerName="revoke-keys" Nov 01 20:51:56.988091 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:56.985914 4484 cpu_manager.go:414] "containerMap: removing container" podUID="76bbc184-657e-4bf3-a072-92d47164ce86" containerName="revoke-keys" Nov 01 20:51:56.988091 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:56.985921 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76bbc184-657e-4bf3-a072-92d47164ce86" containerName="revoke-keys" Nov 01 20:51:56.988091 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:56.986105 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="76bbc184-657e-4bf3-a072-92d47164ce86" containerName="revoke-keys" Nov 01 20:51:56.988091 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:56.986126 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="004f0b18-2fdc-443f-9b43-91b998037058" containerName="revoke-keys" Nov 01 20:51:57.004727 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:57.004691 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:51:57.020737 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:57.020707 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e42be329-mvc5r" Nov 01 20:51:57.030504 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:57.030480 4484 scope.go:122] "RemoveContainer" containerID="0876d9565c2f13a9c2e32b006db9feed08d6be664a86909fef10f3dc7b07ad86" Nov 01 20:51:57.070968 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:57.070809 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e42be329-mvc5r"] Nov 01 20:51:57.119727 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:57.119687 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 20:51:57.119727 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:57.119712 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:51:57.119727 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:57.119697 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:51:57.120013 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:57.119861 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:51:57.128776 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:57.128748 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkr8t\" (UniqueName: \"kubernetes.io/projected/ae5dd0f4-7d3f-47c7-9510-c12d528db8bc-kube-api-access-vkr8t\") pod \"maas-api-revoke-sub-e42be329-mvc5r\" (UID: \"ae5dd0f4-7d3f-47c7-9510-c12d528db8bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e42be329-mvc5r" Nov 01 20:51:57.128893 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:57.128812 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae5dd0f4-7d3f-47c7-9510-c12d528db8bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e42be329-mvc5r\" (UID: \"ae5dd0f4-7d3f-47c7-9510-c12d528db8bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e42be329-mvc5r" Nov 01 20:51:57.128958 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:57.128941 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae5dd0f4-7d3f-47c7-9510-c12d528db8bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e42be329-mvc5r\" (UID: \"ae5dd0f4-7d3f-47c7-9510-c12d528db8bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e42be329-mvc5r" Nov 01 20:51:57.230983 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:57.230905 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkr8t\" (UniqueName: \"kubernetes.io/projected/ae5dd0f4-7d3f-47c7-9510-c12d528db8bc-kube-api-access-vkr8t\") pod \"maas-api-revoke-sub-e42be329-mvc5r\" (UID: \"ae5dd0f4-7d3f-47c7-9510-c12d528db8bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e42be329-mvc5r" Nov 01 20:51:57.231152 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:57.230979 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae5dd0f4-7d3f-47c7-9510-c12d528db8bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e42be329-mvc5r\" (UID: \"ae5dd0f4-7d3f-47c7-9510-c12d528db8bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e42be329-mvc5r" Nov 01 20:51:57.231152 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:57.231048 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae5dd0f4-7d3f-47c7-9510-c12d528db8bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e42be329-mvc5r\" (UID: \"ae5dd0f4-7d3f-47c7-9510-c12d528db8bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e42be329-mvc5r" Nov 01 20:51:57.231725 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:57.231706 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae5dd0f4-7d3f-47c7-9510-c12d528db8bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e42be329-mvc5r\" (UID: \"ae5dd0f4-7d3f-47c7-9510-c12d528db8bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e42be329-mvc5r" Nov 01 20:51:57.233385 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:57.233364 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae5dd0f4-7d3f-47c7-9510-c12d528db8bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e42be329-mvc5r\" (UID: \"ae5dd0f4-7d3f-47c7-9510-c12d528db8bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e42be329-mvc5r" Nov 01 20:51:57.288598 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:57.288566 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkr8t\" (UniqueName: \"kubernetes.io/projected/ae5dd0f4-7d3f-47c7-9510-c12d528db8bc-kube-api-access-vkr8t\") pod \"maas-api-revoke-sub-e42be329-mvc5r\" (UID: \"ae5dd0f4-7d3f-47c7-9510-c12d528db8bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e42be329-mvc5r" Nov 01 20:51:57.397312 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:57.397277 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e42be329-mvc5r" Nov 01 20:51:57.789774 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:51:57.789740 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae5dd0f4_7d3f_47c7_9510_c12d528db8bc.slice/crio-e9ced1abc632f875b12033074c80d374bc7829361b3876b52c189f27ff4ee89f WatchSource:0}: Error finding container e9ced1abc632f875b12033074c80d374bc7829361b3876b52c189f27ff4ee89f: Status 404 returned error can't find the container with id e9ced1abc632f875b12033074c80d374bc7829361b3876b52c189f27ff4ee89f Nov 01 20:51:57.791121 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:57.791095 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e42be329-mvc5r"] Nov 01 20:51:57.960424 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:57.960391 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e42be329-mvc5r" event={"ID":"ae5dd0f4-7d3f-47c7-9510-c12d528db8bc","Type":"ContainerStarted","Data":"e9ced1abc632f875b12033074c80d374bc7829361b3876b52c189f27ff4ee89f"} Nov 01 20:51:57.983182 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:57.983149 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:51:58.337781 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:58.337727 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:51:58.513005 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:58.512958 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv"] Nov 01 20:51:58.517725 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:58.517701 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv" Nov 01 20:51:58.577551 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:58.577517 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-f955d283-openshift-default-dockercfg-kbnzk\"" Nov 01 20:51:58.580645 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:58.580614 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5v2b\" (UniqueName: \"kubernetes.io/projected/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-kube-api-access-s5v2b\") pod \"e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv\" (UID: \"dff8b3d7-cade-4cba-9e07-3e912f0cabc8\") " pod="openshift-ingress/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv" Nov 01 20:51:58.580793 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:58.580679 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-workload-certs\") pod \"e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv\" (UID: \"dff8b3d7-cade-4cba-9e07-3e912f0cabc8\") " pod="openshift-ingress/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv" Nov 01 20:51:58.580793 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:58.580779 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-istio-token\") pod \"e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv\" (UID: \"dff8b3d7-cade-4cba-9e07-3e912f0cabc8\") " pod="openshift-ingress/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv" Nov 01 20:51:58.580976 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:58.580954 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-istio-data\") pod \"e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv\" (UID: \"dff8b3d7-cade-4cba-9e07-3e912f0cabc8\") " pod="openshift-ingress/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv" Nov 01 20:51:58.581112 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:58.581092 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-istiod-ca-cert\") pod \"e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv\" (UID: \"dff8b3d7-cade-4cba-9e07-3e912f0cabc8\") " pod="openshift-ingress/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv" Nov 01 20:51:58.581171 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:58.581152 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-istio-envoy\") pod \"e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv\" (UID: \"dff8b3d7-cade-4cba-9e07-3e912f0cabc8\") " pod="openshift-ingress/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv" Nov 01 20:51:58.581273 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:58.581253 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-workload-socket\") pod \"e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv\" (UID: \"dff8b3d7-cade-4cba-9e07-3e912f0cabc8\") " pod="openshift-ingress/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv" Nov 01 20:51:58.581336 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:58.581322 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-credential-socket\") pod \"e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv\" (UID: \"dff8b3d7-cade-4cba-9e07-3e912f0cabc8\") " pod="openshift-ingress/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv" Nov 01 20:51:58.581488 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:58.581451 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-istio-podinfo\") pod \"e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv\" (UID: \"dff8b3d7-cade-4cba-9e07-3e912f0cabc8\") " pod="openshift-ingress/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv" Nov 01 20:51:58.622951 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:58.622915 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv"] Nov 01 20:51:58.642253 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:58.640798 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:51:58.683893 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:58.683848 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-workload-socket\") pod \"e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv\" (UID: \"dff8b3d7-cade-4cba-9e07-3e912f0cabc8\") " pod="openshift-ingress/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv" Nov 01 20:51:58.684140 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:58.683924 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-credential-socket\") pod \"e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv\" (UID: \"dff8b3d7-cade-4cba-9e07-3e912f0cabc8\") " pod="openshift-ingress/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv" Nov 01 20:51:58.684140 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:58.684025 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-istio-podinfo\") pod \"e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv\" (UID: \"dff8b3d7-cade-4cba-9e07-3e912f0cabc8\") " pod="openshift-ingress/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv" Nov 01 20:51:58.684140 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:58.684097 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5v2b\" (UniqueName: \"kubernetes.io/projected/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-kube-api-access-s5v2b\") pod \"e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv\" (UID: \"dff8b3d7-cade-4cba-9e07-3e912f0cabc8\") " pod="openshift-ingress/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv" Nov 01 20:51:58.684369 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:58.684142 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-workload-certs\") pod \"e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv\" (UID: \"dff8b3d7-cade-4cba-9e07-3e912f0cabc8\") " pod="openshift-ingress/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv" Nov 01 20:51:58.684369 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:58.684166 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-istio-token\") pod \"e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv\" (UID: \"dff8b3d7-cade-4cba-9e07-3e912f0cabc8\") " pod="openshift-ingress/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv" Nov 01 20:51:58.684369 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:58.684297 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-istio-data\") pod \"e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv\" (UID: \"dff8b3d7-cade-4cba-9e07-3e912f0cabc8\") " pod="openshift-ingress/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv" Nov 01 20:51:58.685229 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:58.685196 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-istiod-ca-cert\") pod \"e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv\" (UID: \"dff8b3d7-cade-4cba-9e07-3e912f0cabc8\") " pod="openshift-ingress/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv" Nov 01 20:51:58.693031 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:58.691521 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-workload-socket\") pod \"e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv\" (UID: \"dff8b3d7-cade-4cba-9e07-3e912f0cabc8\") " pod="openshift-ingress/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv" Nov 01 20:51:58.693031 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:58.691797 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-credential-socket\") pod \"e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv\" (UID: \"dff8b3d7-cade-4cba-9e07-3e912f0cabc8\") " pod="openshift-ingress/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv" Nov 01 20:51:58.695100 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:58.693492 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-istio-data\") pod \"e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv\" (UID: \"dff8b3d7-cade-4cba-9e07-3e912f0cabc8\") " pod="openshift-ingress/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv" Nov 01 20:51:58.695100 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:58.694246 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-istiod-ca-cert\") pod \"e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv\" (UID: \"dff8b3d7-cade-4cba-9e07-3e912f0cabc8\") " pod="openshift-ingress/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv" Nov 01 20:51:58.695100 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:58.694317 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-workload-certs\") pod \"e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv\" (UID: \"dff8b3d7-cade-4cba-9e07-3e912f0cabc8\") " pod="openshift-ingress/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv" Nov 01 20:51:58.706030 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:58.697634 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-istio-envoy\") pod \"e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv\" (UID: \"dff8b3d7-cade-4cba-9e07-3e912f0cabc8\") " pod="openshift-ingress/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv" Nov 01 20:51:58.706030 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:58.700304 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-istio-podinfo\") pod \"e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv\" (UID: \"dff8b3d7-cade-4cba-9e07-3e912f0cabc8\") " pod="openshift-ingress/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv" Nov 01 20:51:58.710071 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:58.710036 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-istio-envoy\") pod \"e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv\" (UID: \"dff8b3d7-cade-4cba-9e07-3e912f0cabc8\") " pod="openshift-ingress/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv" Nov 01 20:51:58.807961 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:58.807881 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-istio-token\") pod \"e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv\" (UID: \"dff8b3d7-cade-4cba-9e07-3e912f0cabc8\") " pod="openshift-ingress/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv" Nov 01 20:51:58.849018 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:58.839442 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5v2b\" (UniqueName: \"kubernetes.io/projected/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-kube-api-access-s5v2b\") pod \"e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv\" (UID: \"dff8b3d7-cade-4cba-9e07-3e912f0cabc8\") " pod="openshift-ingress/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv" Nov 01 20:51:58.865448 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:58.865392 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:51:58.980492 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:58.980452 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e42be329-mvc5r" event={"ID":"ae5dd0f4-7d3f-47c7-9510-c12d528db8bc","Type":"ContainerStarted","Data":"5f732772f0519f0c71593dc3d9fdf6db9f20929f0dea6ce334406305ef8c4d6d"} Nov 01 20:51:59.087984 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:59.087841 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:51:59.128866 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:59.128799 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e42be329-mvc5r" podStartSLOduration=6.128776887 podStartE2EDuration="6.128776887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:51:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:51:59.106332989 +0000 UTC m=+1269.883973125" watchObservedRunningTime="2026-11-01 20:51:59.128776887 +0000 UTC m=+1269.906417022" Nov 01 20:51:59.168048 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:59.168014 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv" Nov 01 20:51:59.264357 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:59.264309 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:51:59.686952 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:59.686914 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv"] Nov 01 20:51:59.789477 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:59.788784 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:51:59.789477 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:59.788877 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:51:59.789477 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:59.788920 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:51:59.909916 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:59.909888 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:51:59.992892 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:59.992853 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv" event={"ID":"dff8b3d7-cade-4cba-9e07-3e912f0cabc8","Type":"ContainerStarted","Data":"2a690d6c47f89babe853ff1cbb0cba0cae8655a7b71cf5f7e282e35e76f4e95d"} Nov 01 20:51:59.993377 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:59.992917 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv" event={"ID":"dff8b3d7-cade-4cba-9e07-3e912f0cabc8","Type":"ContainerStarted","Data":"5233362a6e2f749f2302627d73be858707c9334139689fb2dc75ce99d072c8fe"} Nov 01 20:51:59.995050 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:59.995021 4484 generic.go:356] "Generic (PLEG): container finished" podID="ae5dd0f4-7d3f-47c7-9510-c12d528db8bc" containerID="5f732772f0519f0c71593dc3d9fdf6db9f20929f0dea6ce334406305ef8c4d6d" exitCode=0 Nov 01 20:51:59.995198 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:51:59.995152 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e42be329-mvc5r" event={"ID":"ae5dd0f4-7d3f-47c7-9510-c12d528db8bc","Type":"ContainerDied","Data":"5f732772f0519f0c71593dc3d9fdf6db9f20929f0dea6ce334406305ef8c4d6d"} Nov 01 20:52:00.066464 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:00.066425 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bafb9fe-8vwzd"] Nov 01 20:52:00.105401 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:00.105362 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bafb9fe-8vwzd"] Nov 01 20:52:00.130855 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:00.130820 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:00.133010 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:00.132925 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv" podStartSLOduration=2.132911421 podStartE2EDuration="2.132911421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:51:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:52:00.128570095 +0000 UTC m=+1270.906210232" watchObservedRunningTime="2026-11-01 20:52:00.132911421 +0000 UTC m=+1270.910551555" Nov 01 20:52:00.168382 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:00.168345 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv" Nov 01 20:52:00.169655 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:00.169625 4484 patch_prober.go:29] interesting pod/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Nov 01 20:52:00.169780 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:00.169684 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv" podUID="dff8b3d7-cade-4cba-9e07-3e912f0cabc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Nov 01 20:52:00.301225 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:00.301186 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:00.498923 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:00.498313 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:01.082082 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:01.082030 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:01.254520 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:01.253904 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv" Nov 01 20:52:01.289203 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:01.289110 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:01.535256 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:01.535175 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8257bace-423a-4c35-bea7-0d046eef7144" path="/var/lib/kubelet/pods/8257bace-423a-4c35-bea7-0d046eef7144/volumes" Nov 01 20:52:01.603827 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:01.603801 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e42be329-mvc5r" Nov 01 20:52:01.727875 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:01.727837 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae5dd0f4-7d3f-47c7-9510-c12d528db8bc-kube-api-access-vkr8t\" (UniqueName: \"kubernetes.io/projected/ae5dd0f4-7d3f-47c7-9510-c12d528db8bc-kube-api-access-vkr8t\") pod \"ae5dd0f4-7d3f-47c7-9510-c12d528db8bc\" (UID: \"ae5dd0f4-7d3f-47c7-9510-c12d528db8bc\") " Nov 01 20:52:01.728083 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:01.727964 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae5dd0f4-7d3f-47c7-9510-c12d528db8bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae5dd0f4-7d3f-47c7-9510-c12d528db8bc-maas-api-xks-ca\") pod \"ae5dd0f4-7d3f-47c7-9510-c12d528db8bc\" (UID: \"ae5dd0f4-7d3f-47c7-9510-c12d528db8bc\") " Nov 01 20:52:01.728183 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:01.728166 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae5dd0f4-7d3f-47c7-9510-c12d528db8bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae5dd0f4-7d3f-47c7-9510-c12d528db8bc-maas-api-openshift-ca\") pod \"ae5dd0f4-7d3f-47c7-9510-c12d528db8bc\" (UID: \"ae5dd0f4-7d3f-47c7-9510-c12d528db8bc\") " Nov 01 20:52:01.728591 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:01.728556 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae5dd0f4-7d3f-47c7-9510-c12d528db8bc-maas-api-openshift-ca" pod "ae5dd0f4-7d3f-47c7-9510-c12d528db8bc" (UID: "ae5dd0f4-7d3f-47c7-9510-c12d528db8bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:52:01.729138 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:01.729108 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae5dd0f4-7d3f-47c7-9510-c12d528db8bc-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:52:01.730281 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:01.730252 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae5dd0f4-7d3f-47c7-9510-c12d528db8bc-maas-api-xks-ca" pod "ae5dd0f4-7d3f-47c7-9510-c12d528db8bc" (UID: "ae5dd0f4-7d3f-47c7-9510-c12d528db8bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:52:01.730409 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:01.730390 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae5dd0f4-7d3f-47c7-9510-c12d528db8bc-kube-api-access-vkr8t" pod "ae5dd0f4-7d3f-47c7-9510-c12d528db8bc" (UID: "ae5dd0f4-7d3f-47c7-9510-c12d528db8bc"). InnerVolumeSpecName "kube-api-access-vkr8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:52:01.830795 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:01.830704 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkr8t\" (UniqueName: \"kubernetes.io/projected/ae5dd0f4-7d3f-47c7-9510-c12d528db8bc-kube-api-access-vkr8t\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:52:01.830795 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:01.830740 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae5dd0f4-7d3f-47c7-9510-c12d528db8bc-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:52:02.037565 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:02.037534 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e42be329-mvc5r" Nov 01 20:52:02.043216 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:02.043170 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e42be329-mvc5r" event={"ID":"ae5dd0f4-7d3f-47c7-9510-c12d528db8bc","Type":"ContainerDied","Data":"e9ced1abc632f875b12033074c80d374bc7829361b3876b52c189f27ff4ee89f"} Nov 01 20:52:02.043456 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:02.043428 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9ced1abc632f875b12033074c80d374bc7829361b3876b52c189f27ff4ee89f" Nov 01 20:52:02.043609 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:02.043466 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv" Nov 01 20:52:02.049344 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:02.049316 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv" Nov 01 20:52:02.315198 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:02.315162 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:02.500360 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:02.500326 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:02.665185 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:02.665090 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:03.326565 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:03.326502 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:03.551237 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:03.551199 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:03.742120 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:03.742068 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:04.544105 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:04.544070 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d6e880b-vjrfs"] Nov 01 20:52:04.544792 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:04.544778 4484 cpu_manager.go:414] "containerMap: removing container" podUID="ae5dd0f4-7d3f-47c7-9510-c12d528db8bc" containerName="revoke-keys" Nov 01 20:52:04.544836 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:04.544793 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae5dd0f4-7d3f-47c7-9510-c12d528db8bc" containerName="revoke-keys" Nov 01 20:52:04.544941 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:04.544931 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae5dd0f4-7d3f-47c7-9510-c12d528db8bc" containerName="revoke-keys" Nov 01 20:52:04.548714 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:04.548697 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d6e880b-vjrfs" Nov 01 20:52:04.605112 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:04.605071 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d6e880b-vjrfs"] Nov 01 20:52:04.627049 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:04.626980 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de777fdf-8903-433f-b82e-46ed1771bc1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d6e880b-vjrfs\" (UID: \"de777fdf-8903-433f-b82e-46ed1771bc1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d6e880b-vjrfs" Nov 01 20:52:04.627251 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:04.627109 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k48zd\" (UniqueName: \"kubernetes.io/projected/de777fdf-8903-433f-b82e-46ed1771bc1a-kube-api-access-k48zd\") pod \"maas-api-revoke-sub-0d6e880b-vjrfs\" (UID: \"de777fdf-8903-433f-b82e-46ed1771bc1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d6e880b-vjrfs" Nov 01 20:52:04.627320 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:04.627262 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de777fdf-8903-433f-b82e-46ed1771bc1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d6e880b-vjrfs\" (UID: \"de777fdf-8903-433f-b82e-46ed1771bc1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d6e880b-vjrfs" Nov 01 20:52:04.650321 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:04.650278 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:52:04.650576 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:04.650296 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:52:04.650576 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:04.650297 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:52:04.650744 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:04.650345 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 20:52:04.729162 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:04.729122 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de777fdf-8903-433f-b82e-46ed1771bc1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d6e880b-vjrfs\" (UID: \"de777fdf-8903-433f-b82e-46ed1771bc1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d6e880b-vjrfs" Nov 01 20:52:04.729352 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:04.729193 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k48zd\" (UniqueName: \"kubernetes.io/projected/de777fdf-8903-433f-b82e-46ed1771bc1a-kube-api-access-k48zd\") pod \"maas-api-revoke-sub-0d6e880b-vjrfs\" (UID: \"de777fdf-8903-433f-b82e-46ed1771bc1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d6e880b-vjrfs" Nov 01 20:52:04.729421 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:04.729386 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de777fdf-8903-433f-b82e-46ed1771bc1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d6e880b-vjrfs\" (UID: \"de777fdf-8903-433f-b82e-46ed1771bc1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d6e880b-vjrfs" Nov 01 20:52:04.729946 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:04.729920 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de777fdf-8903-433f-b82e-46ed1771bc1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d6e880b-vjrfs\" (UID: \"de777fdf-8903-433f-b82e-46ed1771bc1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d6e880b-vjrfs" Nov 01 20:52:04.731892 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:04.731868 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de777fdf-8903-433f-b82e-46ed1771bc1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d6e880b-vjrfs\" (UID: \"de777fdf-8903-433f-b82e-46ed1771bc1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d6e880b-vjrfs" Nov 01 20:52:04.788088 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:04.788055 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k48zd\" (UniqueName: \"kubernetes.io/projected/de777fdf-8903-433f-b82e-46ed1771bc1a-kube-api-access-k48zd\") pod \"maas-api-revoke-sub-0d6e880b-vjrfs\" (UID: \"de777fdf-8903-433f-b82e-46ed1771bc1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d6e880b-vjrfs" Nov 01 20:52:04.813635 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:04.813534 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:04.862685 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:04.862654 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d6e880b-vjrfs" Nov 01 20:52:05.014220 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:05.014183 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:05.203030 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:05.202902 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:05.252828 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:52:05.252799 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde777fdf_8903_433f_b82e_46ed1771bc1a.slice/crio-e3c6f84116d7de1c8f8134ef5b6b849d342c3bb1568d5753ce319a51200498d6 WatchSource:0}: Error finding container e3c6f84116d7de1c8f8134ef5b6b849d342c3bb1568d5753ce319a51200498d6: Status 404 returned error can't find the container with id e3c6f84116d7de1c8f8134ef5b6b849d342c3bb1568d5753ce319a51200498d6 Nov 01 20:52:05.253750 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:05.253729 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d6e880b-vjrfs"] Nov 01 20:52:05.371090 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:05.371050 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:05.650548 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:05.650517 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:05.837337 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:05.837119 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:06.269064 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:06.252367 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f955d283-6798d4f4dc-b22zs"] Nov 01 20:52:06.269064 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:06.258451 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f955d283-6798d4f4dc-b22zs" Nov 01 20:52:06.269064 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:06.258806 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d6e880b-vjrfs" event={"ID":"de777fdf-8903-433f-b82e-46ed1771bc1a","Type":"ContainerStarted","Data":"8fe384a79f7056515d0875e05970efc6cbd12689ab7976abdb12ea42a8b62561"} Nov 01 20:52:06.269064 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:06.258862 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d6e880b-vjrfs" event={"ID":"de777fdf-8903-433f-b82e-46ed1771bc1a","Type":"ContainerStarted","Data":"e3c6f84116d7de1c8f8134ef5b6b849d342c3bb1568d5753ce319a51200498d6"} Nov 01 20:52:06.269064 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:06.263847 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f955d283-6798d4f4dc-b22zs"] Nov 01 20:52:06.565229 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:06.565187 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f955d283-ddc87bc75-fdqxb"] Nov 01 20:52:06.640455 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:06.640423 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f955d283-ddc87bc75-fdqxb" Nov 01 20:52:06.662708 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:06.660441 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f955d283-dockercfg-trjft\"" Nov 01 20:52:06.662708 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:06.662299 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f955d283\"" Nov 01 20:52:06.680341 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:06.680294 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f955d283-ddc87bc75-fdqxb"] Nov 01 20:52:06.702182 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:06.702151 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:06.769862 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:06.769825 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpfsj\" (UniqueName: \"kubernetes.io/projected/d597fefb-c575-4838-87ef-bced5e8f82f5-kube-api-access-bpfsj\") pod \"payload-pre-processing-e2e-ait-f955d283-6798d4f4dc-b22zs\" (UID: \"d597fefb-c575-4838-87ef-bced5e8f82f5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f955d283-6798d4f4dc-b22zs" Nov 01 20:52:06.770065 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:06.770041 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d597fefb-c575-4838-87ef-bced5e8f82f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f955d283-6798d4f4dc-b22zs\" (UID: \"d597fefb-c575-4838-87ef-bced5e8f82f5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f955d283-6798d4f4dc-b22zs" Nov 01 20:52:06.839386 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:06.839281 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d6e880b-vjrfs" podStartSLOduration=2.839265389 podStartE2EDuration="2.839265389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:52:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:52:06.838255424 +0000 UTC m=+1277.615895569" watchObservedRunningTime="2026-11-01 20:52:06.839265389 +0000 UTC m=+1277.616905554" Nov 01 20:52:06.865595 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:06.865563 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:06.871501 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:06.871469 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkssl\" (UniqueName: \"kubernetes.io/projected/b15235c4-9944-48a8-b4b2-5cd176532f0c-kube-api-access-pkssl\") pod \"payload-processing-e2e-ait-f955d283-ddc87bc75-fdqxb\" (UID: \"b15235c4-9944-48a8-b4b2-5cd176532f0c\") " pod="openshift-ingress/payload-processing-e2e-ait-f955d283-ddc87bc75-fdqxb" Nov 01 20:52:06.871697 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:06.871513 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d597fefb-c575-4838-87ef-bced5e8f82f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f955d283-6798d4f4dc-b22zs\" (UID: \"d597fefb-c575-4838-87ef-bced5e8f82f5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f955d283-6798d4f4dc-b22zs" Nov 01 20:52:06.871697 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:06.871680 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b15235c4-9944-48a8-b4b2-5cd176532f0c-plugins-config-volume\") pod \"payload-processing-e2e-ait-f955d283-ddc87bc75-fdqxb\" (UID: \"b15235c4-9944-48a8-b4b2-5cd176532f0c\") " pod="openshift-ingress/payload-processing-e2e-ait-f955d283-ddc87bc75-fdqxb" Nov 01 20:52:06.871869 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:06.871841 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpfsj\" (UniqueName: \"kubernetes.io/projected/d597fefb-c575-4838-87ef-bced5e8f82f5-kube-api-access-bpfsj\") pod \"payload-pre-processing-e2e-ait-f955d283-6798d4f4dc-b22zs\" (UID: \"d597fefb-c575-4838-87ef-bced5e8f82f5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f955d283-6798d4f4dc-b22zs" Nov 01 20:52:06.872347 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:06.872324 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d597fefb-c575-4838-87ef-bced5e8f82f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f955d283-6798d4f4dc-b22zs\" (UID: \"d597fefb-c575-4838-87ef-bced5e8f82f5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f955d283-6798d4f4dc-b22zs" Nov 01 20:52:06.930757 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:06.930723 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpfsj\" (UniqueName: \"kubernetes.io/projected/d597fefb-c575-4838-87ef-bced5e8f82f5-kube-api-access-bpfsj\") pod \"payload-pre-processing-e2e-ait-f955d283-6798d4f4dc-b22zs\" (UID: \"d597fefb-c575-4838-87ef-bced5e8f82f5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f955d283-6798d4f4dc-b22zs" Nov 01 20:52:06.973915 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:06.973875 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkssl\" (UniqueName: \"kubernetes.io/projected/b15235c4-9944-48a8-b4b2-5cd176532f0c-kube-api-access-pkssl\") pod \"payload-processing-e2e-ait-f955d283-ddc87bc75-fdqxb\" (UID: \"b15235c4-9944-48a8-b4b2-5cd176532f0c\") " pod="openshift-ingress/payload-processing-e2e-ait-f955d283-ddc87bc75-fdqxb" Nov 01 20:52:06.974132 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:06.974042 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b15235c4-9944-48a8-b4b2-5cd176532f0c-plugins-config-volume\") pod \"payload-processing-e2e-ait-f955d283-ddc87bc75-fdqxb\" (UID: \"b15235c4-9944-48a8-b4b2-5cd176532f0c\") " pod="openshift-ingress/payload-processing-e2e-ait-f955d283-ddc87bc75-fdqxb" Nov 01 20:52:06.974763 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:06.974737 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b15235c4-9944-48a8-b4b2-5cd176532f0c-plugins-config-volume\") pod \"payload-processing-e2e-ait-f955d283-ddc87bc75-fdqxb\" (UID: \"b15235c4-9944-48a8-b4b2-5cd176532f0c\") " pod="openshift-ingress/payload-processing-e2e-ait-f955d283-ddc87bc75-fdqxb" Nov 01 20:52:06.978527 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:06.978509 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f955d283-6798d4f4dc-b22zs" Nov 01 20:52:07.031588 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:07.031558 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkssl\" (UniqueName: \"kubernetes.io/projected/b15235c4-9944-48a8-b4b2-5cd176532f0c-kube-api-access-pkssl\") pod \"payload-processing-e2e-ait-f955d283-ddc87bc75-fdqxb\" (UID: \"b15235c4-9944-48a8-b4b2-5cd176532f0c\") " pod="openshift-ingress/payload-processing-e2e-ait-f955d283-ddc87bc75-fdqxb" Nov 01 20:52:07.269110 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:07.269069 4484 generic.go:356] "Generic (PLEG): container finished" podID="de777fdf-8903-433f-b82e-46ed1771bc1a" containerID="8fe384a79f7056515d0875e05970efc6cbd12689ab7976abdb12ea42a8b62561" exitCode=0 Nov 01 20:52:07.269364 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:07.269139 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d6e880b-vjrfs" event={"ID":"de777fdf-8903-433f-b82e-46ed1771bc1a","Type":"ContainerDied","Data":"8fe384a79f7056515d0875e05970efc6cbd12689ab7976abdb12ea42a8b62561"} Nov 01 20:52:07.294542 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:07.294507 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f955d283-ddc87bc75-fdqxb" Nov 01 20:52:07.309312 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:07.309286 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:07.379120 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:07.379041 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f955d283-6798d4f4dc-b22zs"] Nov 01 20:52:07.483864 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:07.483837 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:07.711515 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:52:07.711484 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb15235c4_9944_48a8_b4b2_5cd176532f0c.slice/crio-c7f8f1d803a94f640167b694a5a28090f8d9af2ea72b8ccc2a59680ec7969da8 WatchSource:0}: Error finding container c7f8f1d803a94f640167b694a5a28090f8d9af2ea72b8ccc2a59680ec7969da8: Status 404 returned error can't find the container with id c7f8f1d803a94f640167b694a5a28090f8d9af2ea72b8ccc2a59680ec7969da8 Nov 01 20:52:07.713077 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:07.713053 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f955d283-ddc87bc75-fdqxb"] Nov 01 20:52:08.279479 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:08.279444 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f955d283-6798d4f4dc-b22zs" event={"ID":"d597fefb-c575-4838-87ef-bced5e8f82f5","Type":"ContainerStarted","Data":"c0dd97eb6cdfd6230ea1d3113c8987a8da47eb15cbf055b6d6c737fcf1b52343"} Nov 01 20:52:08.279695 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:08.279498 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f955d283-6798d4f4dc-b22zs" event={"ID":"d597fefb-c575-4838-87ef-bced5e8f82f5","Type":"ContainerStarted","Data":"f463ca1fe6255c79d2e426ff0b55c93a51b7f21c919a28259a16db7c95f175a5"} Nov 01 20:52:08.279695 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:08.279547 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f955d283-6798d4f4dc-b22zs" Nov 01 20:52:08.281403 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:08.281374 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f955d283-ddc87bc75-fdqxb" event={"ID":"b15235c4-9944-48a8-b4b2-5cd176532f0c","Type":"ContainerStarted","Data":"643e26895373ab05406e3e925f23690bd7cb814fd3e59d174145400aa1beefe7"} Nov 01 20:52:08.281537 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:08.281423 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f955d283-ddc87bc75-fdqxb" event={"ID":"b15235c4-9944-48a8-b4b2-5cd176532f0c","Type":"ContainerStarted","Data":"c7f8f1d803a94f640167b694a5a28090f8d9af2ea72b8ccc2a59680ec7969da8"} Nov 01 20:52:08.281580 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:08.281563 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-f955d283-ddc87bc75-fdqxb" Nov 01 20:52:08.397760 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:08.397706 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-f955d283-6798d4f4dc-b22zs" podStartSLOduration=3.39769161 podStartE2EDuration="3.39769161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:52:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:52:08.395184089 +0000 UTC m=+1279.172824224" watchObservedRunningTime="2026-11-01 20:52:08.39769161 +0000 UTC m=+1279.175331744" Nov 01 20:52:08.457245 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:08.457220 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d6e880b-vjrfs" Nov 01 20:52:08.497679 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:08.497647 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de777fdf-8903-433f-b82e-46ed1771bc1a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de777fdf-8903-433f-b82e-46ed1771bc1a-maas-api-openshift-ca\") pod \"de777fdf-8903-433f-b82e-46ed1771bc1a\" (UID: \"de777fdf-8903-433f-b82e-46ed1771bc1a\") " Nov 01 20:52:08.497853 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:08.497696 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de777fdf-8903-433f-b82e-46ed1771bc1a-kube-api-access-k48zd\" (UniqueName: \"kubernetes.io/projected/de777fdf-8903-433f-b82e-46ed1771bc1a-kube-api-access-k48zd\") pod \"de777fdf-8903-433f-b82e-46ed1771bc1a\" (UID: \"de777fdf-8903-433f-b82e-46ed1771bc1a\") " Nov 01 20:52:08.497950 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:08.497937 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de777fdf-8903-433f-b82e-46ed1771bc1a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de777fdf-8903-433f-b82e-46ed1771bc1a-maas-api-xks-ca\") pod \"de777fdf-8903-433f-b82e-46ed1771bc1a\" (UID: \"de777fdf-8903-433f-b82e-46ed1771bc1a\") " Nov 01 20:52:08.498108 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:08.498079 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de777fdf-8903-433f-b82e-46ed1771bc1a-maas-api-openshift-ca" pod "de777fdf-8903-433f-b82e-46ed1771bc1a" (UID: "de777fdf-8903-433f-b82e-46ed1771bc1a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:52:08.498484 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:08.498468 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de777fdf-8903-433f-b82e-46ed1771bc1a-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:52:08.499942 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:08.499913 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de777fdf-8903-433f-b82e-46ed1771bc1a-kube-api-access-k48zd" pod "de777fdf-8903-433f-b82e-46ed1771bc1a" (UID: "de777fdf-8903-433f-b82e-46ed1771bc1a"). InnerVolumeSpecName "kube-api-access-k48zd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:52:08.500089 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:08.500007 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de777fdf-8903-433f-b82e-46ed1771bc1a-maas-api-xks-ca" pod "de777fdf-8903-433f-b82e-46ed1771bc1a" (UID: "de777fdf-8903-433f-b82e-46ed1771bc1a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:52:08.564022 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:08.550819 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-f955d283-ddc87bc75-fdqxb" podStartSLOduration=2.550798113 podStartE2EDuration="2.550798113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:52:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:52:08.528411696 +0000 UTC m=+1279.306051832" watchObservedRunningTime="2026-11-01 20:52:08.550798113 +0000 UTC m=+1279.328438249" Nov 01 20:52:08.600158 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:08.600118 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k48zd\" (UniqueName: \"kubernetes.io/projected/de777fdf-8903-433f-b82e-46ed1771bc1a-kube-api-access-k48zd\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:52:08.600158 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:08.600163 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de777fdf-8903-433f-b82e-46ed1771bc1a-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:52:09.140050 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:09.139968 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:09.308960 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:09.308929 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d6e880b-vjrfs" Nov 01 20:52:09.308960 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:09.308944 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d6e880b-vjrfs" event={"ID":"de777fdf-8903-433f-b82e-46ed1771bc1a","Type":"ContainerDied","Data":"e3c6f84116d7de1c8f8134ef5b6b849d342c3bb1568d5753ce319a51200498d6"} Nov 01 20:52:09.309232 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:09.309042 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3c6f84116d7de1c8f8134ef5b6b849d342c3bb1568d5753ce319a51200498d6" Nov 01 20:52:10.625582 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:10.625546 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:10.820031 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:10.819296 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:11.082006 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:11.081275 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-d0d57533-8598cb8b99-n7sv6"] Nov 01 20:52:11.089015 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:11.082769 4484 cpu_manager.go:414] "containerMap: removing container" podUID="de777fdf-8903-433f-b82e-46ed1771bc1a" containerName="revoke-keys" Nov 01 20:52:11.089015 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:11.082800 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de777fdf-8903-433f-b82e-46ed1771bc1a" containerName="revoke-keys" Nov 01 20:52:11.089015 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:11.083041 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="de777fdf-8903-433f-b82e-46ed1771bc1a" containerName="revoke-keys" Nov 01 20:52:11.099065 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:11.092681 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-d0d57533-8598cb8b99-n7sv6" Nov 01 20:52:11.113053 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:11.112822 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-d0d57533-8598cb8b99-n7sv6"] Nov 01 20:52:11.164876 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:11.158129 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42x8w\" (UniqueName: \"kubernetes.io/projected/59b0a447-bbb7-441e-977f-0fa8fe7696d0-kube-api-access-42x8w\") pod \"payload-pre-processing-e2e-derive-d0d57533-8598cb8b99-n7sv6\" (UID: \"59b0a447-bbb7-441e-977f-0fa8fe7696d0\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-d0d57533-8598cb8b99-n7sv6" Nov 01 20:52:11.164876 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:11.158462 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59b0a447-bbb7-441e-977f-0fa8fe7696d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-d0d57533-8598cb8b99-n7sv6\" (UID: \"59b0a447-bbb7-441e-977f-0fa8fe7696d0\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-d0d57533-8598cb8b99-n7sv6" Nov 01 20:52:11.164876 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:11.160902 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-d0d57533-dockercfg-dml7c\"" Nov 01 20:52:11.269017 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:11.260650 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-d0d57533\"" Nov 01 20:52:11.337522 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:11.318647 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59b0a447-bbb7-441e-977f-0fa8fe7696d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-d0d57533-8598cb8b99-n7sv6\" (UID: \"59b0a447-bbb7-441e-977f-0fa8fe7696d0\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-d0d57533-8598cb8b99-n7sv6" Nov 01 20:52:11.337522 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:11.318826 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42x8w\" (UniqueName: \"kubernetes.io/projected/59b0a447-bbb7-441e-977f-0fa8fe7696d0-kube-api-access-42x8w\") pod \"payload-pre-processing-e2e-derive-d0d57533-8598cb8b99-n7sv6\" (UID: \"59b0a447-bbb7-441e-977f-0fa8fe7696d0\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-d0d57533-8598cb8b99-n7sv6" Nov 01 20:52:11.337522 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:11.320235 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59b0a447-bbb7-441e-977f-0fa8fe7696d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-d0d57533-8598cb8b99-n7sv6\" (UID: \"59b0a447-bbb7-441e-977f-0fa8fe7696d0\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-d0d57533-8598cb8b99-n7sv6" Nov 01 20:52:11.579023 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:11.551420 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d0d57533-7b666f8685-dlwfv"] Nov 01 20:52:11.621340 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:11.621199 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-d0d57533-7b666f8685-dlwfv" Nov 01 20:52:11.623137 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:11.623096 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42x8w\" (UniqueName: \"kubernetes.io/projected/59b0a447-bbb7-441e-977f-0fa8fe7696d0-kube-api-access-42x8w\") pod \"payload-pre-processing-e2e-derive-d0d57533-8598cb8b99-n7sv6\" (UID: \"59b0a447-bbb7-441e-977f-0fa8fe7696d0\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-d0d57533-8598cb8b99-n7sv6" Nov 01 20:52:11.639109 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:11.639082 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d0d57533-7b666f8685-dlwfv"] Nov 01 20:52:11.647355 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:11.647327 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5"] Nov 01 20:52:11.651916 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:11.651895 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5" Nov 01 20:52:11.653044 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:11.653021 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5"] Nov 01 20:52:11.717544 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:11.717508 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba0637f3-6fc9-4052-9808-ffc561da21ca-istiod-ca-cert\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5\" (UID: \"ba0637f3-6fc9-4052-9808-ffc561da21ca\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5" Nov 01 20:52:11.717718 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:11.717645 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba0637f3-6fc9-4052-9808-ffc561da21ca-istio-token\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5\" (UID: \"ba0637f3-6fc9-4052-9808-ffc561da21ca\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5" Nov 01 20:52:11.717718 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:11.717681 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba0637f3-6fc9-4052-9808-ffc561da21ca-credential-socket\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5\" (UID: \"ba0637f3-6fc9-4052-9808-ffc561da21ca\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5" Nov 01 20:52:11.717718 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:11.717712 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba0637f3-6fc9-4052-9808-ffc561da21ca-istio-envoy\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5\" (UID: \"ba0637f3-6fc9-4052-9808-ffc561da21ca\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5" Nov 01 20:52:11.717873 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:11.717852 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b95260f8-613a-4ab5-9780-4e8c271a2bbe-plugins-config-volume\") pod \"payload-processing-e2e-derive-d0d57533-7b666f8685-dlwfv\" (UID: \"b95260f8-613a-4ab5-9780-4e8c271a2bbe\") " pod="openshift-ingress/payload-processing-e2e-derive-d0d57533-7b666f8685-dlwfv" Nov 01 20:52:11.717913 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:11.717890 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba0637f3-6fc9-4052-9808-ffc561da21ca-workload-certs\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5\" (UID: \"ba0637f3-6fc9-4052-9808-ffc561da21ca\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5" Nov 01 20:52:11.717947 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:11.717928 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba0637f3-6fc9-4052-9808-ffc561da21ca-istio-podinfo\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5\" (UID: \"ba0637f3-6fc9-4052-9808-ffc561da21ca\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5" Nov 01 20:52:11.718013 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:11.717987 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba0637f3-6fc9-4052-9808-ffc561da21ca-istio-data\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5\" (UID: \"ba0637f3-6fc9-4052-9808-ffc561da21ca\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5" Nov 01 20:52:11.718067 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:11.718056 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba0637f3-6fc9-4052-9808-ffc561da21ca-workload-socket\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5\" (UID: \"ba0637f3-6fc9-4052-9808-ffc561da21ca\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5" Nov 01 20:52:11.718106 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:11.718079 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j5pv\" (UniqueName: \"kubernetes.io/projected/b95260f8-613a-4ab5-9780-4e8c271a2bbe-kube-api-access-9j5pv\") pod \"payload-processing-e2e-derive-d0d57533-7b666f8685-dlwfv\" (UID: \"b95260f8-613a-4ab5-9780-4e8c271a2bbe\") " pod="openshift-ingress/payload-processing-e2e-derive-d0d57533-7b666f8685-dlwfv" Nov 01 20:52:11.718140 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:11.718129 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgdq5\" (UniqueName: \"kubernetes.io/projected/ba0637f3-6fc9-4052-9808-ffc561da21ca-kube-api-access-qgdq5\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5\" (UID: \"ba0637f3-6fc9-4052-9808-ffc561da21ca\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5" Nov 01 20:52:11.748017 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:11.747964 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-d0d57533-8598cb8b99-n7sv6" Nov 01 20:52:11.834189 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:11.827578 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba0637f3-6fc9-4052-9808-ffc561da21ca-istio-token\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5\" (UID: \"ba0637f3-6fc9-4052-9808-ffc561da21ca\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5" Nov 01 20:52:11.834189 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:11.827631 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba0637f3-6fc9-4052-9808-ffc561da21ca-credential-socket\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5\" (UID: \"ba0637f3-6fc9-4052-9808-ffc561da21ca\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5" Nov 01 20:52:11.834189 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:11.827666 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba0637f3-6fc9-4052-9808-ffc561da21ca-istio-envoy\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5\" (UID: \"ba0637f3-6fc9-4052-9808-ffc561da21ca\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5" Nov 01 20:52:11.834189 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:11.827747 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b95260f8-613a-4ab5-9780-4e8c271a2bbe-plugins-config-volume\") pod \"payload-processing-e2e-derive-d0d57533-7b666f8685-dlwfv\" (UID: \"b95260f8-613a-4ab5-9780-4e8c271a2bbe\") " pod="openshift-ingress/payload-processing-e2e-derive-d0d57533-7b666f8685-dlwfv" Nov 01 20:52:11.834189 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:11.827773 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba0637f3-6fc9-4052-9808-ffc561da21ca-workload-certs\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5\" (UID: \"ba0637f3-6fc9-4052-9808-ffc561da21ca\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5" Nov 01 20:52:11.834189 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:11.827820 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba0637f3-6fc9-4052-9808-ffc561da21ca-istio-podinfo\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5\" (UID: \"ba0637f3-6fc9-4052-9808-ffc561da21ca\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5" Nov 01 20:52:11.834189 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:11.827866 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba0637f3-6fc9-4052-9808-ffc561da21ca-istio-data\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5\" (UID: \"ba0637f3-6fc9-4052-9808-ffc561da21ca\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5" Nov 01 20:52:11.834189 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:11.827928 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba0637f3-6fc9-4052-9808-ffc561da21ca-workload-socket\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5\" (UID: \"ba0637f3-6fc9-4052-9808-ffc561da21ca\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5" Nov 01 20:52:11.834189 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:11.827954 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j5pv\" (UniqueName: \"kubernetes.io/projected/b95260f8-613a-4ab5-9780-4e8c271a2bbe-kube-api-access-9j5pv\") pod \"payload-processing-e2e-derive-d0d57533-7b666f8685-dlwfv\" (UID: \"b95260f8-613a-4ab5-9780-4e8c271a2bbe\") " pod="openshift-ingress/payload-processing-e2e-derive-d0d57533-7b666f8685-dlwfv" Nov 01 20:52:11.834189 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:11.828023 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgdq5\" (UniqueName: \"kubernetes.io/projected/ba0637f3-6fc9-4052-9808-ffc561da21ca-kube-api-access-qgdq5\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5\" (UID: \"ba0637f3-6fc9-4052-9808-ffc561da21ca\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5" Nov 01 20:52:11.834189 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:11.828169 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba0637f3-6fc9-4052-9808-ffc561da21ca-istiod-ca-cert\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5\" (UID: \"ba0637f3-6fc9-4052-9808-ffc561da21ca\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5" Nov 01 20:52:11.834189 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:11.828832 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba0637f3-6fc9-4052-9808-ffc561da21ca-workload-socket\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5\" (UID: \"ba0637f3-6fc9-4052-9808-ffc561da21ca\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5" Nov 01 20:52:11.834189 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:11.829518 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba0637f3-6fc9-4052-9808-ffc561da21ca-workload-certs\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5\" (UID: \"ba0637f3-6fc9-4052-9808-ffc561da21ca\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5" Nov 01 20:52:11.834189 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:11.830082 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba0637f3-6fc9-4052-9808-ffc561da21ca-credential-socket\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5\" (UID: \"ba0637f3-6fc9-4052-9808-ffc561da21ca\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5" Nov 01 20:52:11.834189 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:11.831827 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b95260f8-613a-4ab5-9780-4e8c271a2bbe-plugins-config-volume\") pod \"payload-processing-e2e-derive-d0d57533-7b666f8685-dlwfv\" (UID: \"b95260f8-613a-4ab5-9780-4e8c271a2bbe\") " pod="openshift-ingress/payload-processing-e2e-derive-d0d57533-7b666f8685-dlwfv" Nov 01 20:52:11.834189 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:11.832443 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba0637f3-6fc9-4052-9808-ffc561da21ca-istio-data\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5\" (UID: \"ba0637f3-6fc9-4052-9808-ffc561da21ca\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5" Nov 01 20:52:11.834189 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:11.833454 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba0637f3-6fc9-4052-9808-ffc561da21ca-istio-envoy\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5\" (UID: \"ba0637f3-6fc9-4052-9808-ffc561da21ca\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5" Nov 01 20:52:11.834189 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:11.834053 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba0637f3-6fc9-4052-9808-ffc561da21ca-istiod-ca-cert\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5\" (UID: \"ba0637f3-6fc9-4052-9808-ffc561da21ca\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5" Nov 01 20:52:11.845676 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:11.843624 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba0637f3-6fc9-4052-9808-ffc561da21ca-istio-podinfo\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5\" (UID: \"ba0637f3-6fc9-4052-9808-ffc561da21ca\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5" Nov 01 20:52:11.891817 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:11.891740 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba0637f3-6fc9-4052-9808-ffc561da21ca-istio-token\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5\" (UID: \"ba0637f3-6fc9-4052-9808-ffc561da21ca\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5" Nov 01 20:52:11.892309 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:11.892282 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgdq5\" (UniqueName: \"kubernetes.io/projected/ba0637f3-6fc9-4052-9808-ffc561da21ca-kube-api-access-qgdq5\") pod \"e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5\" (UID: \"ba0637f3-6fc9-4052-9808-ffc561da21ca\") " pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5" Nov 01 20:52:11.895035 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:11.894983 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:11.903052 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:11.903026 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j5pv\" (UniqueName: \"kubernetes.io/projected/b95260f8-613a-4ab5-9780-4e8c271a2bbe-kube-api-access-9j5pv\") pod \"payload-processing-e2e-derive-d0d57533-7b666f8685-dlwfv\" (UID: \"b95260f8-613a-4ab5-9780-4e8c271a2bbe\") " pod="openshift-ingress/payload-processing-e2e-derive-d0d57533-7b666f8685-dlwfv" Nov 01 20:52:11.948180 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:11.948139 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-d0d57533-7b666f8685-dlwfv" Nov 01 20:52:11.966088 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:11.966063 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5" Nov 01 20:52:12.135380 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:52:12.135324 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59b0a447_bbb7_441e_977f_0fa8fe7696d0.slice/crio-edde11fcd074e34c19a00259ed0b6568edbdc8a5cd7e7e69391c5b45ce7c864a WatchSource:0}: Error finding container edde11fcd074e34c19a00259ed0b6568edbdc8a5cd7e7e69391c5b45ce7c864a: Status 404 returned error can't find the container with id edde11fcd074e34c19a00259ed0b6568edbdc8a5cd7e7e69391c5b45ce7c864a Nov 01 20:52:12.139980 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:12.139954 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-d0d57533-8598cb8b99-n7sv6"] Nov 01 20:52:12.332101 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:52:12.332072 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb95260f8_613a_4ab5_9780_4e8c271a2bbe.slice/crio-b0a7ef1bfd27e3c69fa8b5705443a03b168466106ef85a0ce09693a232334baf WatchSource:0}: Error finding container b0a7ef1bfd27e3c69fa8b5705443a03b168466106ef85a0ce09693a232334baf: Status 404 returned error can't find the container with id b0a7ef1bfd27e3c69fa8b5705443a03b168466106ef85a0ce09693a232334baf Nov 01 20:52:12.334376 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:12.334355 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d0d57533-7b666f8685-dlwfv"] Nov 01 20:52:12.385470 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:12.385441 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5"] Nov 01 20:52:12.393626 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:52:12.393599 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba0637f3_6fc9_4052_9808_ffc561da21ca.slice/crio-716d2c8de7d4b1849b2373f06181ef426480dd1616012c8e087f7eade84a4abe WatchSource:0}: Error finding container 716d2c8de7d4b1849b2373f06181ef426480dd1616012c8e087f7eade84a4abe: Status 404 returned error can't find the container with id 716d2c8de7d4b1849b2373f06181ef426480dd1616012c8e087f7eade84a4abe Nov 01 20:52:12.396300 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:12.396268 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:52:12.396381 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:12.396336 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:52:12.396381 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:12.396366 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:52:12.635583 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:12.635550 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5" event={"ID":"ba0637f3-6fc9-4052-9808-ffc561da21ca","Type":"ContainerStarted","Data":"364e65f65e27277e3131247864654cd2e076a2ebeb6e6f655f30ba7f54da675c"} Nov 01 20:52:12.635774 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:12.635609 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5" event={"ID":"ba0637f3-6fc9-4052-9808-ffc561da21ca","Type":"ContainerStarted","Data":"716d2c8de7d4b1849b2373f06181ef426480dd1616012c8e087f7eade84a4abe"} Nov 01 20:52:12.639057 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:12.638923 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-d0d57533-7b666f8685-dlwfv" event={"ID":"b95260f8-613a-4ab5-9780-4e8c271a2bbe","Type":"ContainerStarted","Data":"b1c070c5ee5a88062051ba0e1ee3d346eb7d9ab66741e7bd1b202b2a8b48efd2"} Nov 01 20:52:12.639057 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:12.639006 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-d0d57533-7b666f8685-dlwfv" Nov 01 20:52:12.639057 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:12.639030 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-d0d57533-7b666f8685-dlwfv" event={"ID":"b95260f8-613a-4ab5-9780-4e8c271a2bbe","Type":"ContainerStarted","Data":"b0a7ef1bfd27e3c69fa8b5705443a03b168466106ef85a0ce09693a232334baf"} Nov 01 20:52:12.641815 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:12.641790 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-d0d57533-8598cb8b99-n7sv6" event={"ID":"59b0a447-bbb7-441e-977f-0fa8fe7696d0","Type":"ContainerStarted","Data":"9395ed24f40cd1e58d431811f321bca45f3887c0f574a23c12910f044ac36f8e"} Nov 01 20:52:12.641931 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:12.641841 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-d0d57533-8598cb8b99-n7sv6" event={"ID":"59b0a447-bbb7-441e-977f-0fa8fe7696d0","Type":"ContainerStarted","Data":"edde11fcd074e34c19a00259ed0b6568edbdc8a5cd7e7e69391c5b45ce7c864a"} Nov 01 20:52:12.641931 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:12.641871 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-d0d57533-8598cb8b99-n7sv6" Nov 01 20:52:12.714829 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:12.714744 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:12.761897 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:12.761828 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5" podStartSLOduration=1.761807368 podStartE2EDuration="1.761807368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:52:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:52:12.757641358 +0000 UTC m=+1283.535281511" watchObservedRunningTime="2026-11-01 20:52:12.761807368 +0000 UTC m=+1283.539447504" Nov 01 20:52:12.880317 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:12.880261 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-d0d57533-8598cb8b99-n7sv6" podStartSLOduration=2.880245778 podStartE2EDuration="2.880245778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:52:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:52:12.878026935 +0000 UTC m=+1283.655667069" watchObservedRunningTime="2026-11-01 20:52:12.880245778 +0000 UTC m=+1283.657885969" Nov 01 20:52:12.889691 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:12.889658 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:12.966765 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:12.966675 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5" Nov 01 20:52:13.014030 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:13.010807 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-d0d57533-7b666f8685-dlwfv" podStartSLOduration=2.010781946 podStartE2EDuration="2.010781946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:52:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:52:12.998267969 +0000 UTC m=+1283.775908105" watchObservedRunningTime="2026-11-01 20:52:13.010781946 +0000 UTC m=+1283.788422082" Nov 01 20:52:13.231079 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:13.222861 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:13.413033 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:13.409257 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:14.006479 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:14.004570 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a0ef08b4-7bd7876d5f-jcbw7"] Nov 01 20:52:14.012053 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:14.012019 4484 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 01 20:52:14.012356 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:14.012324 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5" podUID="ba0637f3-6fc9-4052-9808-ffc561da21ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 01 20:52:14.022251 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:14.022215 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:14.025514 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:14.025490 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a0ef08b4-7bd7876d5f-jcbw7" Nov 01 20:52:14.255840 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:14.195311 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:14.301170 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:14.297365 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njzjl\" (UniqueName: \"kubernetes.io/projected/e8ce7f55-690d-4fc8-944c-d7a7c82f6126-kube-api-access-njzjl\") pod \"payload-processing-e2e-ait-a0ef08b4-7bd7876d5f-jcbw7\" (UID: \"e8ce7f55-690d-4fc8-944c-d7a7c82f6126\") " pod="openshift-ingress/payload-processing-e2e-ait-a0ef08b4-7bd7876d5f-jcbw7" Nov 01 20:52:14.335022 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:14.309346 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8ce7f55-690d-4fc8-944c-d7a7c82f6126-plugins-config-volume\") pod \"payload-processing-e2e-ait-a0ef08b4-7bd7876d5f-jcbw7\" (UID: \"e8ce7f55-690d-4fc8-944c-d7a7c82f6126\") " pod="openshift-ingress/payload-processing-e2e-ait-a0ef08b4-7bd7876d5f-jcbw7" Nov 01 20:52:14.335751 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:14.333913 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a0ef08b4\"" Nov 01 20:52:14.336135 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:14.333980 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a0ef08b4-dockercfg-fmkd7\"" Nov 01 20:52:14.336325 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:14.336307 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a0ef08b4-7bd7876d5f-jcbw7"] Nov 01 20:52:14.415025 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:14.412230 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:14.439173 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:14.437589 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njzjl\" (UniqueName: \"kubernetes.io/projected/e8ce7f55-690d-4fc8-944c-d7a7c82f6126-kube-api-access-njzjl\") pod \"payload-processing-e2e-ait-a0ef08b4-7bd7876d5f-jcbw7\" (UID: \"e8ce7f55-690d-4fc8-944c-d7a7c82f6126\") " pod="openshift-ingress/payload-processing-e2e-ait-a0ef08b4-7bd7876d5f-jcbw7" Nov 01 20:52:14.439173 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:14.437708 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8ce7f55-690d-4fc8-944c-d7a7c82f6126-plugins-config-volume\") pod \"payload-processing-e2e-ait-a0ef08b4-7bd7876d5f-jcbw7\" (UID: \"e8ce7f55-690d-4fc8-944c-d7a7c82f6126\") " pod="openshift-ingress/payload-processing-e2e-ait-a0ef08b4-7bd7876d5f-jcbw7" Nov 01 20:52:14.448284 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:14.439844 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8ce7f55-690d-4fc8-944c-d7a7c82f6126-plugins-config-volume\") pod \"payload-processing-e2e-ait-a0ef08b4-7bd7876d5f-jcbw7\" (UID: \"e8ce7f55-690d-4fc8-944c-d7a7c82f6126\") " pod="openshift-ingress/payload-processing-e2e-ait-a0ef08b4-7bd7876d5f-jcbw7" Nov 01 20:52:14.531022 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:14.517873 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njzjl\" (UniqueName: \"kubernetes.io/projected/e8ce7f55-690d-4fc8-944c-d7a7c82f6126-kube-api-access-njzjl\") pod \"payload-processing-e2e-ait-a0ef08b4-7bd7876d5f-jcbw7\" (UID: \"e8ce7f55-690d-4fc8-944c-d7a7c82f6126\") " pod="openshift-ingress/payload-processing-e2e-ait-a0ef08b4-7bd7876d5f-jcbw7" Nov 01 20:52:14.690777 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:14.690695 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a0ef08b4-7bd7876d5f-jcbw7" Nov 01 20:52:14.967134 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:14.966975 4484 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 01 20:52:14.967134 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:14.967100 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5" podUID="ba0637f3-6fc9-4052-9808-ffc561da21ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 01 20:52:15.138624 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:15.138586 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a0ef08b4-7bd7876d5f-jcbw7"] Nov 01 20:52:15.373989 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:15.373953 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d6e1806-9fddr"] Nov 01 20:52:15.379623 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:15.379598 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6e1806-9fddr" Nov 01 20:52:15.434331 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:15.434291 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:52:15.434571 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:15.434346 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:52:15.438310 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:15.438282 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d6e1806-9fddr"] Nov 01 20:52:15.471879 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:15.471839 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f896c2a-12e2-44b2-9dbe-92af1c8b324e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d6e1806-9fddr\" (UID: \"2f896c2a-12e2-44b2-9dbe-92af1c8b324e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6e1806-9fddr" Nov 01 20:52:15.472956 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:15.472925 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f896c2a-12e2-44b2-9dbe-92af1c8b324e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d6e1806-9fddr\" (UID: \"2f896c2a-12e2-44b2-9dbe-92af1c8b324e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6e1806-9fddr" Nov 01 20:52:15.473131 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:15.472978 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dgks\" (UniqueName: \"kubernetes.io/projected/2f896c2a-12e2-44b2-9dbe-92af1c8b324e-kube-api-access-6dgks\") pod \"maas-api-revoke-sub-9d6e1806-9fddr\" (UID: \"2f896c2a-12e2-44b2-9dbe-92af1c8b324e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6e1806-9fddr" Nov 01 20:52:15.480529 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:15.480463 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:52:15.480529 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:15.480480 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 20:52:15.574720 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:15.574676 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f896c2a-12e2-44b2-9dbe-92af1c8b324e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d6e1806-9fddr\" (UID: \"2f896c2a-12e2-44b2-9dbe-92af1c8b324e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6e1806-9fddr" Nov 01 20:52:15.575577 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:15.575546 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f896c2a-12e2-44b2-9dbe-92af1c8b324e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d6e1806-9fddr\" (UID: \"2f896c2a-12e2-44b2-9dbe-92af1c8b324e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6e1806-9fddr" Nov 01 20:52:15.575716 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:15.575650 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f896c2a-12e2-44b2-9dbe-92af1c8b324e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d6e1806-9fddr\" (UID: \"2f896c2a-12e2-44b2-9dbe-92af1c8b324e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6e1806-9fddr" Nov 01 20:52:15.575716 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:15.575691 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dgks\" (UniqueName: \"kubernetes.io/projected/2f896c2a-12e2-44b2-9dbe-92af1c8b324e-kube-api-access-6dgks\") pod \"maas-api-revoke-sub-9d6e1806-9fddr\" (UID: \"2f896c2a-12e2-44b2-9dbe-92af1c8b324e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6e1806-9fddr" Nov 01 20:52:15.578063 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:15.578038 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f896c2a-12e2-44b2-9dbe-92af1c8b324e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d6e1806-9fddr\" (UID: \"2f896c2a-12e2-44b2-9dbe-92af1c8b324e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6e1806-9fddr" Nov 01 20:52:15.652042 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:15.646083 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dgks\" (UniqueName: \"kubernetes.io/projected/2f896c2a-12e2-44b2-9dbe-92af1c8b324e-kube-api-access-6dgks\") pod \"maas-api-revoke-sub-9d6e1806-9fddr\" (UID: \"2f896c2a-12e2-44b2-9dbe-92af1c8b324e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6e1806-9fddr" Nov 01 20:52:15.693020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:15.678569 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:15.701053 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:15.699268 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6e1806-9fddr" Nov 01 20:52:15.852697 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:52:15.852663 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 20:52:15.967431 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:15.967398 4484 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 01 20:52:15.967611 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:15.967451 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5" podUID="ba0637f3-6fc9-4052-9808-ffc561da21ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 01 20:52:16.044254 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:16.044215 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a0ef08b4-7bd7876d5f-jcbw7" event={"ID":"e8ce7f55-690d-4fc8-944c-d7a7c82f6126","Type":"ContainerStarted","Data":"e3d22207e0b88ef10507c9c3a5f93e35d7095c8aab702b38e041baf9c6c9aa56"} Nov 01 20:52:16.044460 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:16.044268 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a0ef08b4-7bd7876d5f-jcbw7" event={"ID":"e8ce7f55-690d-4fc8-944c-d7a7c82f6126","Type":"ContainerStarted","Data":"87010bfd676d486d8a005a9668be5d6cc647da760fd6522c6c90ab10ad82a258"} Nov 01 20:52:16.044460 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:16.044322 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-a0ef08b4-7bd7876d5f-jcbw7" Nov 01 20:52:16.164362 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:16.164260 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-a0ef08b4-7bd7876d5f-jcbw7" podStartSLOduration=3.164245645 podStartE2EDuration="3.164245645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:52:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:52:16.164061222 +0000 UTC m=+1286.941701358" watchObservedRunningTime="2026-11-01 20:52:16.164245645 +0000 UTC m=+1286.941885780" Nov 01 20:52:16.235816 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:52:16.235783 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f896c2a_12e2_44b2_9dbe_92af1c8b324e.slice/crio-511234454a59b83ea3aff100f60264e65c2c99effc6d147271338be9179db27d WatchSource:0}: Error finding container 511234454a59b83ea3aff100f60264e65c2c99effc6d147271338be9179db27d: Status 404 returned error can't find the container with id 511234454a59b83ea3aff100f60264e65c2c99effc6d147271338be9179db27d Nov 01 20:52:16.237989 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:16.237915 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d6e1806-9fddr"] Nov 01 20:52:16.441440 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:16.441369 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67e82e0c-8zqjz"] Nov 01 20:52:16.447574 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:16.447501 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67e82e0c-8zqjz" Nov 01 20:52:16.494436 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:16.494402 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67e82e0c-8zqjz"] Nov 01 20:52:16.522042 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:16.519125 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5c0df19-a16b-4b18-98dd-c900627a705e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67e82e0c-8zqjz\" (UID: \"e5c0df19-a16b-4b18-98dd-c900627a705e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67e82e0c-8zqjz" Nov 01 20:52:16.522042 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:16.519251 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5c0df19-a16b-4b18-98dd-c900627a705e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67e82e0c-8zqjz\" (UID: \"e5c0df19-a16b-4b18-98dd-c900627a705e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67e82e0c-8zqjz" Nov 01 20:52:16.522042 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:16.519374 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnjlg\" (UniqueName: \"kubernetes.io/projected/e5c0df19-a16b-4b18-98dd-c900627a705e-kube-api-access-dnjlg\") pod \"maas-api-revoke-sub-67e82e0c-8zqjz\" (UID: \"e5c0df19-a16b-4b18-98dd-c900627a705e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67e82e0c-8zqjz" Nov 01 20:52:16.624770 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:16.624729 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5c0df19-a16b-4b18-98dd-c900627a705e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67e82e0c-8zqjz\" (UID: \"e5c0df19-a16b-4b18-98dd-c900627a705e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67e82e0c-8zqjz" Nov 01 20:52:16.625167 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:16.625129 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5c0df19-a16b-4b18-98dd-c900627a705e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67e82e0c-8zqjz\" (UID: \"e5c0df19-a16b-4b18-98dd-c900627a705e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67e82e0c-8zqjz" Nov 01 20:52:16.625467 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:16.625433 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnjlg\" (UniqueName: \"kubernetes.io/projected/e5c0df19-a16b-4b18-98dd-c900627a705e-kube-api-access-dnjlg\") pod \"maas-api-revoke-sub-67e82e0c-8zqjz\" (UID: \"e5c0df19-a16b-4b18-98dd-c900627a705e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67e82e0c-8zqjz" Nov 01 20:52:16.629326 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:16.629287 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5c0df19-a16b-4b18-98dd-c900627a705e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67e82e0c-8zqjz\" (UID: \"e5c0df19-a16b-4b18-98dd-c900627a705e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67e82e0c-8zqjz" Nov 01 20:52:16.635104 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:16.635048 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5c0df19-a16b-4b18-98dd-c900627a705e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67e82e0c-8zqjz\" (UID: \"e5c0df19-a16b-4b18-98dd-c900627a705e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67e82e0c-8zqjz" Nov 01 20:52:16.686742 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:16.686710 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnjlg\" (UniqueName: \"kubernetes.io/projected/e5c0df19-a16b-4b18-98dd-c900627a705e-kube-api-access-dnjlg\") pod \"maas-api-revoke-sub-67e82e0c-8zqjz\" (UID: \"e5c0df19-a16b-4b18-98dd-c900627a705e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67e82e0c-8zqjz" Nov 01 20:52:16.762715 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:16.762679 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67e82e0c-8zqjz" Nov 01 20:52:16.967169 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:16.967135 4484 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 01 20:52:16.967368 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:16.967242 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5" podUID="ba0637f3-6fc9-4052-9808-ffc561da21ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 01 20:52:17.056071 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:17.055956 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6e1806-9fddr" event={"ID":"2f896c2a-12e2-44b2-9dbe-92af1c8b324e","Type":"ContainerStarted","Data":"e79e50a51d320f3f3bb0913849f41f19841df17da62834a7bc24cf0dfc9ff40d"} Nov 01 20:52:17.056071 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:17.056036 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6e1806-9fddr" event={"ID":"2f896c2a-12e2-44b2-9dbe-92af1c8b324e","Type":"ContainerStarted","Data":"511234454a59b83ea3aff100f60264e65c2c99effc6d147271338be9179db27d"} Nov 01 20:52:17.178327 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:17.178222 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67e82e0c-8zqjz"] Nov 01 20:52:17.966859 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:17.966808 4484 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 01 20:52:17.967111 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:17.966868 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5" podUID="ba0637f3-6fc9-4052-9808-ffc561da21ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 01 20:52:18.067625 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:18.067525 4484 generic.go:356] "Generic (PLEG): container finished" podID="2f896c2a-12e2-44b2-9dbe-92af1c8b324e" containerID="e79e50a51d320f3f3bb0913849f41f19841df17da62834a7bc24cf0dfc9ff40d" exitCode=0 Nov 01 20:52:18.067625 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:18.067607 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6e1806-9fddr" event={"ID":"2f896c2a-12e2-44b2-9dbe-92af1c8b324e","Type":"ContainerDied","Data":"e79e50a51d320f3f3bb0913849f41f19841df17da62834a7bc24cf0dfc9ff40d"} Nov 01 20:52:18.069743 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:18.069718 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67e82e0c-8zqjz" event={"ID":"e5c0df19-a16b-4b18-98dd-c900627a705e","Type":"ContainerStarted","Data":"ce7d2b53ac68525c99d7d72c5b10bf655d9c0a765449a01a483aee28ba2a97c7"} Nov 01 20:52:18.069871 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:18.069760 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67e82e0c-8zqjz" event={"ID":"e5c0df19-a16b-4b18-98dd-c900627a705e","Type":"ContainerStarted","Data":"0fbb5b7838004d22321e2b962d81af528b3096820a5fcd6fe7b7dd4ab1a5ff57"} Nov 01 20:52:18.328839 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:18.328456 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67e82e0c-8zqjz" podStartSLOduration=2.328437434 podStartE2EDuration="2.328437434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:52:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:52:18.325824439 +0000 UTC m=+1289.103464573" watchObservedRunningTime="2026-11-01 20:52:18.328437434 +0000 UTC m=+1289.106077570" Nov 01 20:52:18.966830 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:18.966792 4484 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 01 20:52:18.967018 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:18.966847 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5" podUID="ba0637f3-6fc9-4052-9808-ffc561da21ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 01 20:52:19.081578 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:19.081547 4484 generic.go:356] "Generic (PLEG): container finished" podID="e5c0df19-a16b-4b18-98dd-c900627a705e" containerID="ce7d2b53ac68525c99d7d72c5b10bf655d9c0a765449a01a483aee28ba2a97c7" exitCode=0 Nov 01 20:52:19.081859 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:19.081629 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67e82e0c-8zqjz" event={"ID":"e5c0df19-a16b-4b18-98dd-c900627a705e","Type":"ContainerDied","Data":"ce7d2b53ac68525c99d7d72c5b10bf655d9c0a765449a01a483aee28ba2a97c7"} Nov 01 20:52:19.269083 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:19.269059 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6e1806-9fddr" Nov 01 20:52:19.312968 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:19.312937 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f955d283-6798d4f4dc-b22zs" Nov 01 20:52:19.313162 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:19.313106 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-f955d283-ddc87bc75-fdqxb" Nov 01 20:52:19.388866 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:19.388837 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f896c2a-12e2-44b2-9dbe-92af1c8b324e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f896c2a-12e2-44b2-9dbe-92af1c8b324e-maas-api-openshift-ca\") pod \"2f896c2a-12e2-44b2-9dbe-92af1c8b324e\" (UID: \"2f896c2a-12e2-44b2-9dbe-92af1c8b324e\") " Nov 01 20:52:19.389662 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:19.388967 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f896c2a-12e2-44b2-9dbe-92af1c8b324e-kube-api-access-6dgks\" (UniqueName: \"kubernetes.io/projected/2f896c2a-12e2-44b2-9dbe-92af1c8b324e-kube-api-access-6dgks\") pod \"2f896c2a-12e2-44b2-9dbe-92af1c8b324e\" (UID: \"2f896c2a-12e2-44b2-9dbe-92af1c8b324e\") " Nov 01 20:52:19.389662 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:19.389271 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f896c2a-12e2-44b2-9dbe-92af1c8b324e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f896c2a-12e2-44b2-9dbe-92af1c8b324e-maas-api-xks-ca\") pod \"2f896c2a-12e2-44b2-9dbe-92af1c8b324e\" (UID: \"2f896c2a-12e2-44b2-9dbe-92af1c8b324e\") " Nov 01 20:52:19.389662 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:19.389401 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f896c2a-12e2-44b2-9dbe-92af1c8b324e-maas-api-openshift-ca" pod "2f896c2a-12e2-44b2-9dbe-92af1c8b324e" (UID: "2f896c2a-12e2-44b2-9dbe-92af1c8b324e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:52:19.390103 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:19.390083 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f896c2a-12e2-44b2-9dbe-92af1c8b324e-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:52:19.392912 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:19.392110 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f896c2a-12e2-44b2-9dbe-92af1c8b324e-kube-api-access-6dgks" pod "2f896c2a-12e2-44b2-9dbe-92af1c8b324e" (UID: "2f896c2a-12e2-44b2-9dbe-92af1c8b324e"). InnerVolumeSpecName "kube-api-access-6dgks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:52:19.393258 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:19.393217 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f896c2a-12e2-44b2-9dbe-92af1c8b324e-maas-api-xks-ca" pod "2f896c2a-12e2-44b2-9dbe-92af1c8b324e" (UID: "2f896c2a-12e2-44b2-9dbe-92af1c8b324e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:52:19.476947 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:19.476906 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:19.492208 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:19.492180 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f896c2a-12e2-44b2-9dbe-92af1c8b324e-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:52:19.492344 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:19.492212 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dgks\" (UniqueName: \"kubernetes.io/projected/2f896c2a-12e2-44b2-9dbe-92af1c8b324e-kube-api-access-6dgks\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:52:20.102123 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:20.102077 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6e1806-9fddr" event={"ID":"2f896c2a-12e2-44b2-9dbe-92af1c8b324e","Type":"ContainerDied","Data":"511234454a59b83ea3aff100f60264e65c2c99effc6d147271338be9179db27d"} Nov 01 20:52:20.102320 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:20.102163 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="511234454a59b83ea3aff100f60264e65c2c99effc6d147271338be9179db27d" Nov 01 20:52:20.102320 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:20.102269 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6e1806-9fddr" Nov 01 20:52:20.377516 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:20.377487 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67e82e0c-8zqjz" Nov 01 20:52:20.424048 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:20.420625 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5c0df19-a16b-4b18-98dd-c900627a705e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5c0df19-a16b-4b18-98dd-c900627a705e-maas-api-openshift-ca\") pod \"e5c0df19-a16b-4b18-98dd-c900627a705e\" (UID: \"e5c0df19-a16b-4b18-98dd-c900627a705e\") " Nov 01 20:52:20.424048 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:20.420789 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5c0df19-a16b-4b18-98dd-c900627a705e-kube-api-access-dnjlg\" (UniqueName: \"kubernetes.io/projected/e5c0df19-a16b-4b18-98dd-c900627a705e-kube-api-access-dnjlg\") pod \"e5c0df19-a16b-4b18-98dd-c900627a705e\" (UID: \"e5c0df19-a16b-4b18-98dd-c900627a705e\") " Nov 01 20:52:20.428185 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:20.426817 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5c0df19-a16b-4b18-98dd-c900627a705e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5c0df19-a16b-4b18-98dd-c900627a705e-maas-api-xks-ca\") pod \"e5c0df19-a16b-4b18-98dd-c900627a705e\" (UID: \"e5c0df19-a16b-4b18-98dd-c900627a705e\") " Nov 01 20:52:20.444019 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:20.440816 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5c0df19-a16b-4b18-98dd-c900627a705e-maas-api-openshift-ca" pod "e5c0df19-a16b-4b18-98dd-c900627a705e" (UID: "e5c0df19-a16b-4b18-98dd-c900627a705e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:52:20.447640 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:20.447588 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5c0df19-a16b-4b18-98dd-c900627a705e-maas-api-xks-ca" pod "e5c0df19-a16b-4b18-98dd-c900627a705e" (UID: "e5c0df19-a16b-4b18-98dd-c900627a705e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:52:20.455496 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:20.455455 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5c0df19-a16b-4b18-98dd-c900627a705e-kube-api-access-dnjlg" pod "e5c0df19-a16b-4b18-98dd-c900627a705e" (UID: "e5c0df19-a16b-4b18-98dd-c900627a705e"). InnerVolumeSpecName "kube-api-access-dnjlg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:52:20.530043 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:20.529954 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5c0df19-a16b-4b18-98dd-c900627a705e-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:52:20.530043 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:20.530011 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5c0df19-a16b-4b18-98dd-c900627a705e-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:52:20.530043 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:20.530029 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnjlg\" (UniqueName: \"kubernetes.io/projected/e5c0df19-a16b-4b18-98dd-c900627a705e-kube-api-access-dnjlg\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:52:20.537597 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:20.537570 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5" Nov 01 20:52:20.538680 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:20.538653 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5" Nov 01 20:52:20.539737 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:20.539719 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5" Nov 01 20:52:21.112814 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:21.112778 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67e82e0c-8zqjz" event={"ID":"e5c0df19-a16b-4b18-98dd-c900627a705e","Type":"ContainerDied","Data":"0fbb5b7838004d22321e2b962d81af528b3096820a5fcd6fe7b7dd4ab1a5ff57"} Nov 01 20:52:21.112988 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:21.112836 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fbb5b7838004d22321e2b962d81af528b3096820a5fcd6fe7b7dd4ab1a5ff57" Nov 01 20:52:21.112988 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:21.112790 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67e82e0c-8zqjz" Nov 01 20:52:24.002225 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:24.002198 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-d0d57533-8598cb8b99-n7sv6" Nov 01 20:52:24.002614 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:24.002262 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-d0d57533-7b666f8685-dlwfv" Nov 01 20:52:24.853631 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:24.853592 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:25.317927 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:25.317083 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" podUID="9f19be29-2c84-4ae4-9674-acca4905f4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:25.824176 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:25.824115 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" podUID="e175b298-2c23-45d4-8f9c-0c9b9f6aab92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": context deadline exceeded" Nov 01 20:52:26.117680 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:26.116444 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" podUID="17a9ac24-19cb-48c4-99b5-d74ce8aac0a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:26.153547 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:26.153505 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" podUID="70b4dacf-b3ac-46ec-99e8-a44f05587e7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 01 20:52:26.190700 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:26.190648 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" podUID="a65d9fd8-6193-4d03-9a3c-123f48b5dcdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:26.325671 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:26.323111 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" podUID="9f19be29-2c84-4ae4-9674-acca4905f4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:26.325671 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:26.323925 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" Nov 01 20:52:26.899138 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:26.899091 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" podUID="e175b298-2c23-45d4-8f9c-0c9b9f6aab92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": context deadline exceeded" Nov 01 20:52:26.955019 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:26.936985 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" podUID="311f9699-6585-4a2c-8c20-e6b696b2f857" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:26.955019 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:26.942086 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" podUID="fe8307db-264d-4b8a-b097-f180bc255a9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 01 20:52:27.028213 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:27.028174 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" Nov 01 20:52:27.090279 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:27.089210 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-a0ef08b4-7bd7876d5f-jcbw7" Nov 01 20:52:27.123064 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:27.118138 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" podUID="17a9ac24-19cb-48c4-99b5-d74ce8aac0a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:27.139606 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:27.138309 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" Nov 01 20:52:27.198316 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:27.198213 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" podUID="a65d9fd8-6193-4d03-9a3c-123f48b5dcdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Nov 01 20:52:27.223059 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:27.205873 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" Nov 01 20:52:27.242693 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:27.242649 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" podUID="70b4dacf-b3ac-46ec-99e8-a44f05587e7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 01 20:52:27.250253 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:27.250216 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" Nov 01 20:52:27.338026 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:27.337433 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" podUID="9f19be29-2c84-4ae4-9674-acca4905f4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:27.429756 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:27.429694 4484 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" podUID="311f9699-6585-4a2c-8c20-e6b696b2f857" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:27.754017 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:27.729191 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:27.978154 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:27.978099 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" podUID="e175b298-2c23-45d4-8f9c-0c9b9f6aab92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": context deadline exceeded" Nov 01 20:52:28.639020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:28.000409 4484 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" podUID="e175b298-2c23-45d4-8f9c-0c9b9f6aab92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/health\": context deadline exceeded" Nov 01 20:52:28.639020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:28.073468 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" podUID="fe8307db-264d-4b8a-b097-f180bc255a9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:28.639020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:28.108738 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" podUID="311f9699-6585-4a2c-8c20-e6b696b2f857" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:28.639020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:28.208275 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" podUID="a65d9fd8-6193-4d03-9a3c-123f48b5dcdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Nov 01 20:52:28.639020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:28.344833 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" podUID="70b4dacf-b3ac-46ec-99e8-a44f05587e7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:28.673047 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:28.433116 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" podUID="9f19be29-2c84-4ae4-9674-acca4905f4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:29.132040 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:29.129103 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" podUID="17a9ac24-19cb-48c4-99b5-d74ce8aac0a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:29.142285 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:29.142165 4484 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" podUID="17a9ac24-19cb-48c4-99b5-d74ce8aac0a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": context deadline exceeded" Nov 01 20:52:29.585033 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:52:29.288205 4484 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.549s" Nov 01 20:52:29.585033 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:29.491255 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:29.585033 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:29.566560 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" podUID="e175b298-2c23-45d4-8f9c-0c9b9f6aab92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:29.600044 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:29.597369 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" podUID="fe8307db-264d-4b8a-b097-f180bc255a9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 01 20:52:29.608184 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:29.608124 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" podUID="311f9699-6585-4a2c-8c20-e6b696b2f857" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:29.647968 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:29.608448 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" podUID="a65d9fd8-6193-4d03-9a3c-123f48b5dcdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:29.649116 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:29.612081 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" podUID="70b4dacf-b3ac-46ec-99e8-a44f05587e7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:29.649291 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:29.647844 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" podUID="9f19be29-2c84-4ae4-9674-acca4905f4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 01 20:52:29.881081 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:29.863106 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" Nov 01 20:52:29.910022 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:29.889412 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" Nov 01 20:52:29.914783 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:29.914707 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:30.131199 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:30.131096 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" podUID="17a9ac24-19cb-48c4-99b5-d74ce8aac0a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:30.287022 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:30.284676 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:30.529481 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:30.529436 4484 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" podUID="9f19be29-2c84-4ae4-9674-acca4905f4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:30.870080 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:30.869184 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" podUID="fe8307db-264d-4b8a-b097-f180bc255a9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 01 20:52:30.870520 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:30.870490 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" podUID="e175b298-2c23-45d4-8f9c-0c9b9f6aab92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:30.911021 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:30.885321 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" podUID="9f19be29-2c84-4ae4-9674-acca4905f4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 01 20:52:30.911021 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:30.886827 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" podUID="a65d9fd8-6193-4d03-9a3c-123f48b5dcdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:30.911021 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:30.887312 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" podUID="70b4dacf-b3ac-46ec-99e8-a44f05587e7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 01 20:52:31.018176 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:31.018132 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" podUID="311f9699-6585-4a2c-8c20-e6b696b2f857" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:31.080223 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:31.080180 4484 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:52:31.090095 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:31.089571 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw" podUID="e9c302ea-168a-4fa6-885f-2a3263835c79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:31.212071 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:31.200943 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" podUID="17a9ac24-19cb-48c4-99b5-d74ce8aac0a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 01 20:52:31.232413 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:31.230574 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" Nov 01 20:52:31.273022 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:31.241555 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:31.633805 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:31.632802 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-dt84g" podUID="3060c0d1-44f0-48ec-843f-d6e8c0833982" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.22:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:32.098800 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:32.094844 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" podUID="fe8307db-264d-4b8a-b097-f180bc255a9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:32.098800 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:32.097105 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" podUID="70b4dacf-b3ac-46ec-99e8-a44f05587e7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:32.098800 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:32.097375 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" podUID="311f9699-6585-4a2c-8c20-e6b696b2f857" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 01 20:52:32.098800 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:32.097808 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" podUID="9f19be29-2c84-4ae4-9674-acca4905f4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 01 20:52:32.098800 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:32.097834 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" podUID="a65d9fd8-6193-4d03-9a3c-123f48b5dcdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Nov 01 20:52:32.335043 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:32.223930 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" podUID="17a9ac24-19cb-48c4-99b5-d74ce8aac0a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 01 20:52:32.335043 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:32.300554 4484 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" podUID="70b4dacf-b3ac-46ec-99e8-a44f05587e7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:32.380118 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:32.371189 4484 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" podUID="a65d9fd8-6193-4d03-9a3c-123f48b5dcdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:32.543416 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:32.543373 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" podUID="e175b298-2c23-45d4-8f9c-0c9b9f6aab92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:33.129022 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:33.103532 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" podUID="9f19be29-2c84-4ae4-9674-acca4905f4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 01 20:52:33.205023 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:33.105087 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" podUID="a65d9fd8-6193-4d03-9a3c-123f48b5dcdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:33.205023 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:33.163073 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" podUID="fe8307db-264d-4b8a-b097-f180bc255a9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 01 20:52:33.215332 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:33.215291 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" podUID="311f9699-6585-4a2c-8c20-e6b696b2f857" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 01 20:52:33.241806 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:33.105121 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" podUID="70b4dacf-b3ac-46ec-99e8-a44f05587e7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 01 20:52:33.251026 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:33.244362 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" podUID="17a9ac24-19cb-48c4-99b5-d74ce8aac0a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:33.261347 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:33.248359 4484 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:52:33.261347 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:33.261303 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj" podUID="8e7768b7-6fa1-4408-a1f5-2862a8a7d907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:33.557301 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:33.557253 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" podUID="e175b298-2c23-45d4-8f9c-0c9b9f6aab92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": context deadline exceeded" Nov 01 20:52:33.714210 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:33.714166 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" Nov 01 20:52:33.839676 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:33.839590 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" Nov 01 20:52:33.956382 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:33.956347 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" Nov 01 20:52:34.225076 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:34.192334 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" podUID="9f19be29-2c84-4ae4-9674-acca4905f4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 01 20:52:34.271076 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:34.203180 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" podUID="fe8307db-264d-4b8a-b097-f180bc255a9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:34.271076 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:34.203219 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" podUID="a65d9fd8-6193-4d03-9a3c-123f48b5dcdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:34.271076 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:34.216479 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" podUID="311f9699-6585-4a2c-8c20-e6b696b2f857" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:52:34.271076 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:34.252909 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:34.325900 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:34.325871 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" Nov 01 20:52:34.411746 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:34.411709 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" Nov 01 20:52:34.413281 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:34.413261 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" Nov 01 20:52:34.430673 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:34.422432 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" Nov 01 20:52:34.432130 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:34.432106 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" Nov 01 20:52:36.139896 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:36.139864 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:40.233549 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:40.233516 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:40.391576 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:40.391536 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:40.945662 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:40.945626 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:45.672440 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:45.672391 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:45.873960 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:45.873883 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:45.877975 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:52:45.877943 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 20:52:46.129802 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:46.129770 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:50.637014 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:50.636437 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:50.809497 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:50.809449 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:51.007330 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:51.007294 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:55.452310 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:55.452273 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z4sdv"] Nov 01 20:52:55.453114 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:55.453097 4484 cpu_manager.go:414] "containerMap: removing container" podUID="2f896c2a-12e2-44b2-9dbe-92af1c8b324e" containerName="revoke-keys" Nov 01 20:52:55.453114 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:55.453114 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f896c2a-12e2-44b2-9dbe-92af1c8b324e" containerName="revoke-keys" Nov 01 20:52:55.453232 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:55.453123 4484 cpu_manager.go:414] "containerMap: removing container" podUID="e5c0df19-a16b-4b18-98dd-c900627a705e" containerName="revoke-keys" Nov 01 20:52:55.453232 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:55.453129 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5c0df19-a16b-4b18-98dd-c900627a705e" containerName="revoke-keys" Nov 01 20:52:55.453299 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:55.453282 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f896c2a-12e2-44b2-9dbe-92af1c8b324e" containerName="revoke-keys" Nov 01 20:52:55.453299 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:55.453290 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5c0df19-a16b-4b18-98dd-c900627a705e" containerName="revoke-keys" Nov 01 20:52:55.458741 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:55.458724 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z4sdv" Nov 01 20:52:55.481705 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:55.481670 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:55.568543 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:55.557015 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/243b9a06-d07a-4d14-abe2-ad2239dd45fe-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-z4sdv\" (UID: \"243b9a06-d07a-4d14-abe2-ad2239dd45fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z4sdv" Nov 01 20:52:55.568543 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:55.557125 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/243b9a06-d07a-4d14-abe2-ad2239dd45fe-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-z4sdv\" (UID: \"243b9a06-d07a-4d14-abe2-ad2239dd45fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z4sdv" Nov 01 20:52:55.568543 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:55.557188 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/243b9a06-d07a-4d14-abe2-ad2239dd45fe-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-z4sdv\" (UID: \"243b9a06-d07a-4d14-abe2-ad2239dd45fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z4sdv" Nov 01 20:52:55.568543 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:55.557277 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtt7d\" (UniqueName: \"kubernetes.io/projected/243b9a06-d07a-4d14-abe2-ad2239dd45fe-kube-api-access-wtt7d\") pod \"maas-default-gateway-openshift-default-58c6585447-z4sdv\" (UID: \"243b9a06-d07a-4d14-abe2-ad2239dd45fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z4sdv" Nov 01 20:52:55.568543 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:55.557336 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/243b9a06-d07a-4d14-abe2-ad2239dd45fe-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-z4sdv\" (UID: \"243b9a06-d07a-4d14-abe2-ad2239dd45fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z4sdv" Nov 01 20:52:55.568543 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:55.557415 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/243b9a06-d07a-4d14-abe2-ad2239dd45fe-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-z4sdv\" (UID: \"243b9a06-d07a-4d14-abe2-ad2239dd45fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z4sdv" Nov 01 20:52:55.568543 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:55.557529 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/243b9a06-d07a-4d14-abe2-ad2239dd45fe-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-z4sdv\" (UID: \"243b9a06-d07a-4d14-abe2-ad2239dd45fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z4sdv" Nov 01 20:52:55.568543 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:55.557611 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/243b9a06-d07a-4d14-abe2-ad2239dd45fe-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-z4sdv\" (UID: \"243b9a06-d07a-4d14-abe2-ad2239dd45fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z4sdv" Nov 01 20:52:55.568543 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:55.557706 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/243b9a06-d07a-4d14-abe2-ad2239dd45fe-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-z4sdv\" (UID: \"243b9a06-d07a-4d14-abe2-ad2239dd45fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z4sdv" Nov 01 20:52:55.673686 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:55.673609 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:55.685143 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:55.685107 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z4sdv"] Nov 01 20:52:55.689840 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:55.689646 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/243b9a06-d07a-4d14-abe2-ad2239dd45fe-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-z4sdv\" (UID: \"243b9a06-d07a-4d14-abe2-ad2239dd45fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z4sdv" Nov 01 20:52:55.690171 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:55.690146 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/243b9a06-d07a-4d14-abe2-ad2239dd45fe-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-z4sdv\" (UID: \"243b9a06-d07a-4d14-abe2-ad2239dd45fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z4sdv" Nov 01 20:52:55.690495 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:55.690480 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtt7d\" (UniqueName: \"kubernetes.io/projected/243b9a06-d07a-4d14-abe2-ad2239dd45fe-kube-api-access-wtt7d\") pod \"maas-default-gateway-openshift-default-58c6585447-z4sdv\" (UID: \"243b9a06-d07a-4d14-abe2-ad2239dd45fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z4sdv" Nov 01 20:52:55.690850 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:55.690830 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/243b9a06-d07a-4d14-abe2-ad2239dd45fe-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-z4sdv\" (UID: \"243b9a06-d07a-4d14-abe2-ad2239dd45fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z4sdv" Nov 01 20:52:55.691261 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:55.691116 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/243b9a06-d07a-4d14-abe2-ad2239dd45fe-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-z4sdv\" (UID: \"243b9a06-d07a-4d14-abe2-ad2239dd45fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z4sdv" Nov 01 20:52:55.691647 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:55.691495 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/243b9a06-d07a-4d14-abe2-ad2239dd45fe-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-z4sdv\" (UID: \"243b9a06-d07a-4d14-abe2-ad2239dd45fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z4sdv" Nov 01 20:52:55.691810 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:55.691797 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/243b9a06-d07a-4d14-abe2-ad2239dd45fe-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-z4sdv\" (UID: \"243b9a06-d07a-4d14-abe2-ad2239dd45fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z4sdv" Nov 01 20:52:55.692134 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:55.692117 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/243b9a06-d07a-4d14-abe2-ad2239dd45fe-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-z4sdv\" (UID: \"243b9a06-d07a-4d14-abe2-ad2239dd45fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z4sdv" Nov 01 20:52:55.692383 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:55.692367 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/243b9a06-d07a-4d14-abe2-ad2239dd45fe-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-z4sdv\" (UID: \"243b9a06-d07a-4d14-abe2-ad2239dd45fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z4sdv" Nov 01 20:52:55.706498 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:55.706171 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/243b9a06-d07a-4d14-abe2-ad2239dd45fe-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-z4sdv\" (UID: \"243b9a06-d07a-4d14-abe2-ad2239dd45fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z4sdv" Nov 01 20:52:55.707767 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:55.707611 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/243b9a06-d07a-4d14-abe2-ad2239dd45fe-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-z4sdv\" (UID: \"243b9a06-d07a-4d14-abe2-ad2239dd45fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z4sdv" Nov 01 20:52:55.739378 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:55.730795 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/243b9a06-d07a-4d14-abe2-ad2239dd45fe-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-z4sdv\" (UID: \"243b9a06-d07a-4d14-abe2-ad2239dd45fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z4sdv" Nov 01 20:52:55.739378 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:55.731146 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/243b9a06-d07a-4d14-abe2-ad2239dd45fe-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-z4sdv\" (UID: \"243b9a06-d07a-4d14-abe2-ad2239dd45fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z4sdv" Nov 01 20:52:55.739622 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:55.739399 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/243b9a06-d07a-4d14-abe2-ad2239dd45fe-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-z4sdv\" (UID: \"243b9a06-d07a-4d14-abe2-ad2239dd45fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z4sdv" Nov 01 20:52:55.740551 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:55.740525 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/243b9a06-d07a-4d14-abe2-ad2239dd45fe-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-z4sdv\" (UID: \"243b9a06-d07a-4d14-abe2-ad2239dd45fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z4sdv" Nov 01 20:52:55.763193 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:55.763010 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/243b9a06-d07a-4d14-abe2-ad2239dd45fe-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-z4sdv\" (UID: \"243b9a06-d07a-4d14-abe2-ad2239dd45fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z4sdv" Nov 01 20:52:55.851247 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:55.851155 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/243b9a06-d07a-4d14-abe2-ad2239dd45fe-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-z4sdv\" (UID: \"243b9a06-d07a-4d14-abe2-ad2239dd45fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z4sdv" Nov 01 20:52:55.853222 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:55.853173 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtt7d\" (UniqueName: \"kubernetes.io/projected/243b9a06-d07a-4d14-abe2-ad2239dd45fe-kube-api-access-wtt7d\") pod \"maas-default-gateway-openshift-default-58c6585447-z4sdv\" (UID: \"243b9a06-d07a-4d14-abe2-ad2239dd45fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z4sdv" Nov 01 20:52:55.988582 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:55.988547 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:56.077888 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:56.077852 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z4sdv" Nov 01 20:52:56.520422 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:52:56.520391 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod243b9a06_d07a_4d14_abe2_ad2239dd45fe.slice/crio-1ffa4eda144f6e43e9b63d458f49d3822bf0dada7c76e541f7312405018a0b9f WatchSource:0}: Error finding container 1ffa4eda144f6e43e9b63d458f49d3822bf0dada7c76e541f7312405018a0b9f: Status 404 returned error can't find the container with id 1ffa4eda144f6e43e9b63d458f49d3822bf0dada7c76e541f7312405018a0b9f Nov 01 20:52:56.521980 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:56.521933 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z4sdv"] Nov 01 20:52:56.522942 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:56.522912 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:52:56.523029 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:56.522971 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:52:56.523074 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:56.523038 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:52:56.649370 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:56.649324 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:57.054118 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:57.054086 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z4sdv" event={"ID":"243b9a06-d07a-4d14-abe2-ad2239dd45fe","Type":"ContainerStarted","Data":"341a09a23ee439b432b557cc11272c45d10d5ae8721d87979f1fcfdd79c91bf9"} Nov 01 20:52:57.054305 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:57.054137 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z4sdv" event={"ID":"243b9a06-d07a-4d14-abe2-ad2239dd45fe","Type":"ContainerStarted","Data":"1ffa4eda144f6e43e9b63d458f49d3822bf0dada7c76e541f7312405018a0b9f"} Nov 01 20:52:57.078974 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:57.078933 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z4sdv" Nov 01 20:52:57.080542 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:57.080512 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-z4sdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 01 20:52:57.080674 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:57.080566 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z4sdv" podUID="243b9a06-d07a-4d14-abe2-ad2239dd45fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 01 20:52:57.175796 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:57.175766 4484 scope.go:122] "RemoveContainer" containerID="a242e4e528b1023c50568857d19fb85db06919e0b1c187e1f0d626ea9c4d1308" Nov 01 20:52:57.187542 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:57.187520 4484 scope.go:122] "RemoveContainer" containerID="6f9c877c16607bd0a26419238c37ec08a892a0ac81635cb94b3bf527112d52ed" Nov 01 20:52:57.197944 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:57.197925 4484 scope.go:122] "RemoveContainer" containerID="695f8ce87a7586ec8185af77047bd547bbf8bcac3241c1fcd7a9669091eebf95" Nov 01 20:52:57.208711 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:57.208688 4484 scope.go:122] "RemoveContainer" containerID="aa56e507833f602332e90ff8bf5ef525aee037a2f1a459b708e8f9e769e2099a" Nov 01 20:52:57.219699 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:57.219678 4484 scope.go:122] "RemoveContainer" containerID="71d1ec2ecd53dfb86bd46233a4add5b92659bec4f6462eb5952905fdfc767870" Nov 01 20:52:57.231647 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:57.231624 4484 scope.go:122] "RemoveContainer" containerID="0a6ae19c7585f2c47111f0bafb528a38cb47d422ab4f66b96f6eeb4b5634ead5" Nov 01 20:52:57.242954 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:57.242929 4484 scope.go:122] "RemoveContainer" containerID="c0c82dfbe9a056ee2ca0e95811cf9e84a9c9eaf1eacf5e61b927880b74dcff46" Nov 01 20:52:57.254454 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:57.254427 4484 scope.go:122] "RemoveContainer" containerID="e85f09c1a2b24dd350d7c3f710bc7d2d4e58a9068a6918438c51dc72b0a4e239" Nov 01 20:52:57.265159 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:57.265137 4484 scope.go:122] "RemoveContainer" containerID="3186ba9aef3a4c6eaf742b00585c9f27a3764744dec49851021eacd3dcb9e2b7" Nov 01 20:52:57.268200 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:57.268158 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z4sdv" podStartSLOduration=2.268144334 podStartE2EDuration="2.268144334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:52:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:52:57.223106866 +0000 UTC m=+1328.000747030" watchObservedRunningTime="2026-11-01 20:52:57.268144334 +0000 UTC m=+1328.045784469" Nov 01 20:52:57.278010 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:57.277979 4484 scope.go:122] "RemoveContainer" containerID="cf539a500adad9d6d35b61fa352fdbff05798df35d63d3d71fb118432c6e5ef4" Nov 01 20:52:57.288121 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:57.288100 4484 scope.go:122] "RemoveContainer" containerID="654837a9a3c05ed5f4a2a473193955a561b5c829a8bb07c49bca455acd95747a" Nov 01 20:52:58.097025 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:58.095226 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-z4sdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 01 20:52:58.097025 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:58.095300 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z4sdv" podUID="243b9a06-d07a-4d14-abe2-ad2239dd45fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 01 20:52:59.078894 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:59.078851 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-z4sdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 01 20:52:59.079136 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:52:59.078922 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z4sdv" podUID="243b9a06-d07a-4d14-abe2-ad2239dd45fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 01 20:53:00.079218 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:00.079183 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-z4sdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 01 20:53:00.079623 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:00.079252 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z4sdv" podUID="243b9a06-d07a-4d14-abe2-ad2239dd45fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 01 20:53:00.616025 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:00.607198 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:53:01.078367 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:01.078334 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-z4sdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 01 20:53:01.078532 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:01.078385 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z4sdv" podUID="243b9a06-d07a-4d14-abe2-ad2239dd45fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 01 20:53:02.078301 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:02.078244 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-z4sdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 01 20:53:02.078730 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:02.078315 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z4sdv" podUID="243b9a06-d07a-4d14-abe2-ad2239dd45fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 01 20:53:02.289777 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:02.289739 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:53:03.078950 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:03.078910 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-z4sdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 01 20:53:03.079380 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:03.078973 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z4sdv" podUID="243b9a06-d07a-4d14-abe2-ad2239dd45fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 01 20:53:04.079050 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:04.078971 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-z4sdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 01 20:53:04.079452 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:04.079059 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z4sdv" podUID="243b9a06-d07a-4d14-abe2-ad2239dd45fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 01 20:53:05.472741 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:05.471609 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:53:06.092019 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:06.083110 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-z4sdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:53:06.092019 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:06.083183 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z4sdv" podUID="243b9a06-d07a-4d14-abe2-ad2239dd45fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:53:06.406073 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:06.393290 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z4sdv" Nov 01 20:53:06.406073 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:06.395051 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z4sdv" Nov 01 20:53:06.406073 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:06.401279 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z4sdv" Nov 01 20:53:10.459900 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:10.459809 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:53:15.241756 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:15.241719 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:53:15.768466 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:53:15.768430 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 20:53:16.366919 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:16.366885 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-964f4a4a-x2lw6"] Nov 01 20:53:16.452598 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:16.452556 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-964f4a4a-x2lw6"] Nov 01 20:53:16.452803 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:16.452685 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-964f4a4a-x2lw6" Nov 01 20:53:16.506262 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:16.506216 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:53:16.506262 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:16.506216 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:53:16.522011 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:16.521971 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/529a9279-5840-45be-af91-529f4f5df586-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-964f4a4a-x2lw6\" (UID: \"529a9279-5840-45be-af91-529f4f5df586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-964f4a4a-x2lw6" Nov 01 20:53:16.522206 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:16.522186 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8lkq\" (UniqueName: \"kubernetes.io/projected/529a9279-5840-45be-af91-529f4f5df586-kube-api-access-q8lkq\") pod \"maas-api-revoke-sub-964f4a4a-x2lw6\" (UID: \"529a9279-5840-45be-af91-529f4f5df586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-964f4a4a-x2lw6" Nov 01 20:53:16.522284 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:16.522244 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/529a9279-5840-45be-af91-529f4f5df586-maas-api-xks-ca\") pod \"maas-api-revoke-sub-964f4a4a-x2lw6\" (UID: \"529a9279-5840-45be-af91-529f4f5df586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-964f4a4a-x2lw6" Nov 01 20:53:16.557210 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:16.557170 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:53:16.557210 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:16.557191 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 20:53:16.624176 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:16.624087 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8lkq\" (UniqueName: \"kubernetes.io/projected/529a9279-5840-45be-af91-529f4f5df586-kube-api-access-q8lkq\") pod \"maas-api-revoke-sub-964f4a4a-x2lw6\" (UID: \"529a9279-5840-45be-af91-529f4f5df586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-964f4a4a-x2lw6" Nov 01 20:53:16.624176 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:16.624151 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/529a9279-5840-45be-af91-529f4f5df586-maas-api-xks-ca\") pod \"maas-api-revoke-sub-964f4a4a-x2lw6\" (UID: \"529a9279-5840-45be-af91-529f4f5df586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-964f4a4a-x2lw6" Nov 01 20:53:16.624423 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:16.624306 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/529a9279-5840-45be-af91-529f4f5df586-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-964f4a4a-x2lw6\" (UID: \"529a9279-5840-45be-af91-529f4f5df586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-964f4a4a-x2lw6" Nov 01 20:53:16.624918 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:16.624895 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/529a9279-5840-45be-af91-529f4f5df586-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-964f4a4a-x2lw6\" (UID: \"529a9279-5840-45be-af91-529f4f5df586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-964f4a4a-x2lw6" Nov 01 20:53:16.626565 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:16.626537 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/529a9279-5840-45be-af91-529f4f5df586-maas-api-xks-ca\") pod \"maas-api-revoke-sub-964f4a4a-x2lw6\" (UID: \"529a9279-5840-45be-af91-529f4f5df586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-964f4a4a-x2lw6" Nov 01 20:53:16.682311 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:16.682271 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8lkq\" (UniqueName: \"kubernetes.io/projected/529a9279-5840-45be-af91-529f4f5df586-kube-api-access-q8lkq\") pod \"maas-api-revoke-sub-964f4a4a-x2lw6\" (UID: \"529a9279-5840-45be-af91-529f4f5df586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-964f4a4a-x2lw6" Nov 01 20:53:16.791582 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:16.791541 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-964f4a4a-x2lw6" Nov 01 20:53:17.305788 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:53:17.305739 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod529a9279_5840_45be_af91_529f4f5df586.slice/crio-f14d91d33ee1e78819209e4521a0b3eaa3a79a02ee5861fefe07e942df9111ef WatchSource:0}: Error finding container f14d91d33ee1e78819209e4521a0b3eaa3a79a02ee5861fefe07e942df9111ef: Status 404 returned error can't find the container with id f14d91d33ee1e78819209e4521a0b3eaa3a79a02ee5861fefe07e942df9111ef Nov 01 20:53:17.309256 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:17.308757 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-964f4a4a-x2lw6"] Nov 01 20:53:17.648375 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:17.648295 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-964f4a4a-x2lw6" event={"ID":"529a9279-5840-45be-af91-529f4f5df586","Type":"ContainerStarted","Data":"f14d91d33ee1e78819209e4521a0b3eaa3a79a02ee5861fefe07e942df9111ef"} Nov 01 20:53:18.078193 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:18.078154 4484 patch_prober.go:29] interesting pod/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:53:18.078506 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:18.078220 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv" podUID="dff8b3d7-cade-4cba-9e07-3e912f0cabc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:53:18.662871 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:18.662836 4484 generic.go:356] "Generic (PLEG): container finished" podID="529a9279-5840-45be-af91-529f4f5df586" containerID="83954ada3f526b92f36d0356b7542c9b7e69f7ea4c19c317a98ef0615dc0f662" exitCode=0 Nov 01 20:53:18.665095 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:18.662947 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-964f4a4a-x2lw6" event={"ID":"529a9279-5840-45be-af91-529f4f5df586","Type":"ContainerDied","Data":"83954ada3f526b92f36d0356b7542c9b7e69f7ea4c19c317a98ef0615dc0f662"} Nov 01 20:53:19.870270 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:19.870238 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-964f4a4a-x2lw6" Nov 01 20:53:19.977399 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:19.977314 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/529a9279-5840-45be-af91-529f4f5df586-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/529a9279-5840-45be-af91-529f4f5df586-maas-api-openshift-ca\") pod \"529a9279-5840-45be-af91-529f4f5df586\" (UID: \"529a9279-5840-45be-af91-529f4f5df586\") " Nov 01 20:53:19.977675 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:19.977651 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/529a9279-5840-45be-af91-529f4f5df586-maas-api-openshift-ca" pod "529a9279-5840-45be-af91-529f4f5df586" (UID: "529a9279-5840-45be-af91-529f4f5df586"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:53:19.977778 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:19.977761 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/529a9279-5840-45be-af91-529f4f5df586-kube-api-access-q8lkq\" (UniqueName: \"kubernetes.io/projected/529a9279-5840-45be-af91-529f4f5df586-kube-api-access-q8lkq\") pod \"529a9279-5840-45be-af91-529f4f5df586\" (UID: \"529a9279-5840-45be-af91-529f4f5df586\") " Nov 01 20:53:19.977839 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:19.977825 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/529a9279-5840-45be-af91-529f4f5df586-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/529a9279-5840-45be-af91-529f4f5df586-maas-api-xks-ca\") pod \"529a9279-5840-45be-af91-529f4f5df586\" (UID: \"529a9279-5840-45be-af91-529f4f5df586\") " Nov 01 20:53:19.978554 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:19.978521 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/529a9279-5840-45be-af91-529f4f5df586-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:53:19.979887 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:19.979854 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/529a9279-5840-45be-af91-529f4f5df586-kube-api-access-q8lkq" pod "529a9279-5840-45be-af91-529f4f5df586" (UID: "529a9279-5840-45be-af91-529f4f5df586"). InnerVolumeSpecName "kube-api-access-q8lkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:53:19.979988 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:19.979895 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/529a9279-5840-45be-af91-529f4f5df586-maas-api-xks-ca" pod "529a9279-5840-45be-af91-529f4f5df586" (UID: "529a9279-5840-45be-af91-529f4f5df586"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:53:20.079810 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:20.079772 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8lkq\" (UniqueName: \"kubernetes.io/projected/529a9279-5840-45be-af91-529f4f5df586-kube-api-access-q8lkq\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:53:20.079810 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:20.079809 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/529a9279-5840-45be-af91-529f4f5df586-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:53:20.169793 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:20.169747 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:53:20.683962 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:20.683929 4484 generic.go:356] "Generic (PLEG): container finished" podID="dff8b3d7-cade-4cba-9e07-3e912f0cabc8" containerID="2a690d6c47f89babe853ff1cbb0cba0cae8655a7b71cf5f7e282e35e76f4e95d" exitCode=0 Nov 01 20:53:20.684199 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:20.684016 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv" event={"ID":"dff8b3d7-cade-4cba-9e07-3e912f0cabc8","Type":"ContainerDied","Data":"2a690d6c47f89babe853ff1cbb0cba0cae8655a7b71cf5f7e282e35e76f4e95d"} Nov 01 20:53:20.684557 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:20.684523 4484 scope.go:122] "RemoveContainer" containerID="2a690d6c47f89babe853ff1cbb0cba0cae8655a7b71cf5f7e282e35e76f4e95d" Nov 01 20:53:20.685752 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:20.685731 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-964f4a4a-x2lw6" event={"ID":"529a9279-5840-45be-af91-529f4f5df586","Type":"ContainerDied","Data":"f14d91d33ee1e78819209e4521a0b3eaa3a79a02ee5861fefe07e942df9111ef"} Nov 01 20:53:20.685832 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:20.685757 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-964f4a4a-x2lw6" Nov 01 20:53:20.685832 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:20.685773 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f14d91d33ee1e78819209e4521a0b3eaa3a79a02ee5861fefe07e942df9111ef" Nov 01 20:53:20.687127 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:20.687092 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:53:20.687240 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:20.687164 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:53:20.687240 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:20.687198 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:53:21.697685 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:21.697653 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv" event={"ID":"dff8b3d7-cade-4cba-9e07-3e912f0cabc8","Type":"ContainerStarted","Data":"578a8ddb4623e84b75b7c4fb4aa39ed74914603c2f27dff3ccc1556c9a09d137"} Nov 01 20:53:22.169199 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:22.169166 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv" Nov 01 20:53:22.169387 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:22.169363 4484 patch_prober.go:29] interesting pod/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Nov 01 20:53:22.169450 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:22.169431 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv" podUID="dff8b3d7-cade-4cba-9e07-3e912f0cabc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Nov 01 20:53:23.168933 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:23.168892 4484 patch_prober.go:29] interesting pod/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Nov 01 20:53:23.169377 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:23.168951 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv" podUID="dff8b3d7-cade-4cba-9e07-3e912f0cabc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Nov 01 20:53:24.169157 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:24.169116 4484 patch_prober.go:29] interesting pod/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Nov 01 20:53:24.169665 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:24.169188 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv" podUID="dff8b3d7-cade-4cba-9e07-3e912f0cabc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Nov 01 20:53:25.173020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:25.170435 4484 patch_prober.go:29] interesting pod/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Nov 01 20:53:25.173020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:25.170516 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv" podUID="dff8b3d7-cade-4cba-9e07-3e912f0cabc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Nov 01 20:53:25.338782 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:25.338737 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:53:25.496054 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:25.496024 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:53:26.168557 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:26.168519 4484 patch_prober.go:29] interesting pod/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Nov 01 20:53:26.168736 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:26.168575 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv" podUID="dff8b3d7-cade-4cba-9e07-3e912f0cabc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Nov 01 20:53:27.039783 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:27.039734 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:53:27.173519 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:27.173485 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv" Nov 01 20:53:27.173915 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:27.173894 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv" Nov 01 20:53:27.174588 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:27.174565 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv" Nov 01 20:53:29.364657 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:29.364624 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:53:29.837762 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:29.837725 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:53:32.681724 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:32.681694 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:53:34.623677 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:34.623630 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:53:37.038220 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:37.038188 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:53:39.425698 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:39.425660 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:53:40.571793 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:40.571754 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:53:44.526204 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:44.526132 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:53:44.627710 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:44.627670 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-wwpvf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:53:44.628089 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:44.627958 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wwpvf" podUID="f411ea1f-0db8-4748-87e9-4ae94a7a12c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:53:45.768649 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:53:45.768605 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 20:53:49.758138 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:49.758105 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:53:50.360300 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:50.360266 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f955d283-k7gzw"] Nov 01 20:53:50.361064 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:50.361049 4484 cpu_manager.go:414] "containerMap: removing container" podUID="529a9279-5840-45be-af91-529f4f5df586" containerName="revoke-keys" Nov 01 20:53:50.361114 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:50.361065 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="529a9279-5840-45be-af91-529f4f5df586" containerName="revoke-keys" Nov 01 20:53:50.361201 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:50.361193 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="529a9279-5840-45be-af91-529f4f5df586" containerName="revoke-keys" Nov 01 20:53:50.363832 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:50.363807 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f955d283-k7gzw" Nov 01 20:53:50.421072 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:50.421038 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f955d283-k7gzw"] Nov 01 20:53:50.464329 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:50.464295 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:53:50.464520 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:50.464361 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:53:50.464520 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:50.464376 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:53:50.467265 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:50.467241 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 20:53:50.534111 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:50.534059 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsmmh\" (UniqueName: \"kubernetes.io/projected/732dc849-a68c-4415-85ca-594a86f70898-kube-api-access-qsmmh\") pod \"maas-api-revoke-keys-e2e-ait-f955d283-k7gzw\" (UID: \"732dc849-a68c-4415-85ca-594a86f70898\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f955d283-k7gzw" Nov 01 20:53:50.534307 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:50.534179 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/732dc849-a68c-4415-85ca-594a86f70898-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f955d283-k7gzw\" (UID: \"732dc849-a68c-4415-85ca-594a86f70898\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f955d283-k7gzw" Nov 01 20:53:50.534362 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:50.534313 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/732dc849-a68c-4415-85ca-594a86f70898-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f955d283-k7gzw\" (UID: \"732dc849-a68c-4415-85ca-594a86f70898\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f955d283-k7gzw" Nov 01 20:53:50.636510 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:50.636420 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/732dc849-a68c-4415-85ca-594a86f70898-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f955d283-k7gzw\" (UID: \"732dc849-a68c-4415-85ca-594a86f70898\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f955d283-k7gzw" Nov 01 20:53:50.636510 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:50.636507 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/732dc849-a68c-4415-85ca-594a86f70898-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f955d283-k7gzw\" (UID: \"732dc849-a68c-4415-85ca-594a86f70898\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f955d283-k7gzw" Nov 01 20:53:50.636821 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:50.636800 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsmmh\" (UniqueName: \"kubernetes.io/projected/732dc849-a68c-4415-85ca-594a86f70898-kube-api-access-qsmmh\") pod \"maas-api-revoke-keys-e2e-ait-f955d283-k7gzw\" (UID: \"732dc849-a68c-4415-85ca-594a86f70898\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f955d283-k7gzw" Nov 01 20:53:50.637183 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:50.637162 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/732dc849-a68c-4415-85ca-594a86f70898-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f955d283-k7gzw\" (UID: \"732dc849-a68c-4415-85ca-594a86f70898\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f955d283-k7gzw" Nov 01 20:53:50.638882 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:50.638860 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/732dc849-a68c-4415-85ca-594a86f70898-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f955d283-k7gzw\" (UID: \"732dc849-a68c-4415-85ca-594a86f70898\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f955d283-k7gzw" Nov 01 20:53:50.694984 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:50.694948 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsmmh\" (UniqueName: \"kubernetes.io/projected/732dc849-a68c-4415-85ca-594a86f70898-kube-api-access-qsmmh\") pod \"maas-api-revoke-keys-e2e-ait-f955d283-k7gzw\" (UID: \"732dc849-a68c-4415-85ca-594a86f70898\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f955d283-k7gzw" Nov 01 20:53:50.977505 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:50.977410 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f955d283-k7gzw" Nov 01 20:53:51.360744 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:53:51.360710 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod732dc849_a68c_4415_85ca_594a86f70898.slice/crio-7ae6779b35e0914e160b5635e3d3b21765e8a6a6cbac7a0b808a565704a2aff3 WatchSource:0}: Error finding container 7ae6779b35e0914e160b5635e3d3b21765e8a6a6cbac7a0b808a565704a2aff3: Status 404 returned error can't find the container with id 7ae6779b35e0914e160b5635e3d3b21765e8a6a6cbac7a0b808a565704a2aff3 Nov 01 20:53:51.362114 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:51.362083 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f955d283-k7gzw"] Nov 01 20:53:51.644119 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:51.644086 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0ef08b4-mshhs"] Nov 01 20:53:51.647773 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:51.647750 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0ef08b4-mshhs" Nov 01 20:53:51.701532 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:51.701496 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0ef08b4-mshhs"] Nov 01 20:53:51.756223 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:51.756187 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ntxt\" (UniqueName: \"kubernetes.io/projected/d89f74fe-7a5b-411d-b82d-d77ff26b92f3-kube-api-access-5ntxt\") pod \"maas-api-revoke-keys-e2e-ait-a0ef08b4-mshhs\" (UID: \"d89f74fe-7a5b-411d-b82d-d77ff26b92f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0ef08b4-mshhs" Nov 01 20:53:51.756402 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:51.756243 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d89f74fe-7a5b-411d-b82d-d77ff26b92f3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a0ef08b4-mshhs\" (UID: \"d89f74fe-7a5b-411d-b82d-d77ff26b92f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0ef08b4-mshhs" Nov 01 20:53:51.756483 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:51.756462 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d89f74fe-7a5b-411d-b82d-d77ff26b92f3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a0ef08b4-mshhs\" (UID: \"d89f74fe-7a5b-411d-b82d-d77ff26b92f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0ef08b4-mshhs" Nov 01 20:53:51.858389 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:51.858350 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ntxt\" (UniqueName: \"kubernetes.io/projected/d89f74fe-7a5b-411d-b82d-d77ff26b92f3-kube-api-access-5ntxt\") pod \"maas-api-revoke-keys-e2e-ait-a0ef08b4-mshhs\" (UID: \"d89f74fe-7a5b-411d-b82d-d77ff26b92f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0ef08b4-mshhs" Nov 01 20:53:51.858547 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:51.858411 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d89f74fe-7a5b-411d-b82d-d77ff26b92f3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a0ef08b4-mshhs\" (UID: \"d89f74fe-7a5b-411d-b82d-d77ff26b92f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0ef08b4-mshhs" Nov 01 20:53:51.858634 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:51.858603 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d89f74fe-7a5b-411d-b82d-d77ff26b92f3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a0ef08b4-mshhs\" (UID: \"d89f74fe-7a5b-411d-b82d-d77ff26b92f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0ef08b4-mshhs" Nov 01 20:53:51.859158 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:51.859135 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d89f74fe-7a5b-411d-b82d-d77ff26b92f3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a0ef08b4-mshhs\" (UID: \"d89f74fe-7a5b-411d-b82d-d77ff26b92f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0ef08b4-mshhs" Nov 01 20:53:51.860950 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:51.860928 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d89f74fe-7a5b-411d-b82d-d77ff26b92f3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a0ef08b4-mshhs\" (UID: \"d89f74fe-7a5b-411d-b82d-d77ff26b92f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0ef08b4-mshhs" Nov 01 20:53:51.919918 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:51.919846 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ntxt\" (UniqueName: \"kubernetes.io/projected/d89f74fe-7a5b-411d-b82d-d77ff26b92f3-kube-api-access-5ntxt\") pod \"maas-api-revoke-keys-e2e-ait-a0ef08b4-mshhs\" (UID: \"d89f74fe-7a5b-411d-b82d-d77ff26b92f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0ef08b4-mshhs" Nov 01 20:53:51.967080 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:51.967041 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0ef08b4-mshhs" Nov 01 20:53:52.062498 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:52.062458 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40cb3cb1-75zv6"] Nov 01 20:53:52.071504 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:52.071439 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40cb3cb1-75zv6"] Nov 01 20:53:52.134092 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:52.134046 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f955d283-k7gzw" event={"ID":"732dc849-a68c-4415-85ca-594a86f70898","Type":"ContainerStarted","Data":"0b4eb635a8d3d9ca5f23d94d184f613aecef032c401f7e3fb2741f6da7b56af0"} Nov 01 20:53:52.134269 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:52.134124 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f955d283-k7gzw" event={"ID":"732dc849-a68c-4415-85ca-594a86f70898","Type":"ContainerStarted","Data":"7ae6779b35e0914e160b5635e3d3b21765e8a6a6cbac7a0b808a565704a2aff3"} Nov 01 20:53:52.184824 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:52.184750 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:53:52.252537 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:52.252422 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f955d283-k7gzw" podStartSLOduration=2.252407136 podStartE2EDuration="2.252407136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:53:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:53:52.249139441 +0000 UTC m=+1383.026779595" watchObservedRunningTime="2026-11-01 20:53:52.252407136 +0000 UTC m=+1383.030047270" Nov 01 20:53:52.332324 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:52.332289 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:53:52.355414 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:52.355385 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0ef08b4-mshhs"] Nov 01 20:53:52.360293 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:53:52.360264 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd89f74fe_7a5b_411d_b82d_d77ff26b92f3.slice/crio-89fcef8f92341fab42d2fa9049f2d2b1ef22ea2b62deb10c19ed9267fd708cb5 WatchSource:0}: Error finding container 89fcef8f92341fab42d2fa9049f2d2b1ef22ea2b62deb10c19ed9267fd708cb5: Status 404 returned error can't find the container with id 89fcef8f92341fab42d2fa9049f2d2b1ef22ea2b62deb10c19ed9267fd708cb5 Nov 01 20:53:53.145152 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:53.145115 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0ef08b4-mshhs" event={"ID":"d89f74fe-7a5b-411d-b82d-d77ff26b92f3","Type":"ContainerStarted","Data":"b690fad52de89a4fbdbec6d5759ff6399cfa995b855663743c7f7791f67a7907"} Nov 01 20:53:53.145616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:53.145181 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0ef08b4-mshhs" event={"ID":"d89f74fe-7a5b-411d-b82d-d77ff26b92f3","Type":"ContainerStarted","Data":"89fcef8f92341fab42d2fa9049f2d2b1ef22ea2b62deb10c19ed9267fd708cb5"} Nov 01 20:53:53.147039 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:53.146980 4484 generic.go:356] "Generic (PLEG): container finished" podID="732dc849-a68c-4415-85ca-594a86f70898" containerID="0b4eb635a8d3d9ca5f23d94d184f613aecef032c401f7e3fb2741f6da7b56af0" exitCode=0 Nov 01 20:53:53.147186 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:53.147027 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f955d283-k7gzw" event={"ID":"732dc849-a68c-4415-85ca-594a86f70898","Type":"ContainerDied","Data":"0b4eb635a8d3d9ca5f23d94d184f613aecef032c401f7e3fb2741f6da7b56af0"} Nov 01 20:53:53.265886 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:53.265830 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0ef08b4-mshhs" podStartSLOduration=2.265812129 podStartE2EDuration="2.265812129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:53:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:53:53.260048667 +0000 UTC m=+1384.037688804" watchObservedRunningTime="2026-11-01 20:53:53.265812129 +0000 UTC m=+1384.043452267" Nov 01 20:53:53.373827 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:53.373790 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="096f520b-a10b-4f4c-afcc-0ea5d143fbd2" path="/var/lib/kubelet/pods/096f520b-a10b-4f4c-afcc-0ea5d143fbd2/volumes" Nov 01 20:53:53.401067 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:53.400942 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:53:54.247020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:54.226136 4484 generic.go:356] "Generic (PLEG): container finished" podID="d89f74fe-7a5b-411d-b82d-d77ff26b92f3" containerID="b690fad52de89a4fbdbec6d5759ff6399cfa995b855663743c7f7791f67a7907" exitCode=0 Nov 01 20:53:54.247020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:54.226451 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0ef08b4-mshhs" event={"ID":"d89f74fe-7a5b-411d-b82d-d77ff26b92f3","Type":"ContainerDied","Data":"b690fad52de89a4fbdbec6d5759ff6399cfa995b855663743c7f7791f67a7907"} Nov 01 20:53:54.677268 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:54.677242 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f955d283-k7gzw" Nov 01 20:53:54.782612 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:54.782571 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/732dc849-a68c-4415-85ca-594a86f70898-kube-api-access-qsmmh\" (UniqueName: \"kubernetes.io/projected/732dc849-a68c-4415-85ca-594a86f70898-kube-api-access-qsmmh\") pod \"732dc849-a68c-4415-85ca-594a86f70898\" (UID: \"732dc849-a68c-4415-85ca-594a86f70898\") " Nov 01 20:53:54.782802 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:54.782714 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/732dc849-a68c-4415-85ca-594a86f70898-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/732dc849-a68c-4415-85ca-594a86f70898-maas-api-openshift-ca\") pod \"732dc849-a68c-4415-85ca-594a86f70898\" (UID: \"732dc849-a68c-4415-85ca-594a86f70898\") " Nov 01 20:53:54.782875 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:54.782858 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/732dc849-a68c-4415-85ca-594a86f70898-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/732dc849-a68c-4415-85ca-594a86f70898-maas-api-xks-ca\") pod \"732dc849-a68c-4415-85ca-594a86f70898\" (UID: \"732dc849-a68c-4415-85ca-594a86f70898\") " Nov 01 20:53:54.783124 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:54.783099 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/732dc849-a68c-4415-85ca-594a86f70898-maas-api-openshift-ca" pod "732dc849-a68c-4415-85ca-594a86f70898" (UID: "732dc849-a68c-4415-85ca-594a86f70898"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:53:54.783564 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:54.783538 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/732dc849-a68c-4415-85ca-594a86f70898-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:53:54.784772 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:54.784744 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/732dc849-a68c-4415-85ca-594a86f70898-kube-api-access-qsmmh" pod "732dc849-a68c-4415-85ca-594a86f70898" (UID: "732dc849-a68c-4415-85ca-594a86f70898"). InnerVolumeSpecName "kube-api-access-qsmmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:53:54.784892 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:54.784852 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/732dc849-a68c-4415-85ca-594a86f70898-maas-api-xks-ca" pod "732dc849-a68c-4415-85ca-594a86f70898" (UID: "732dc849-a68c-4415-85ca-594a86f70898"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:53:54.885675 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:54.885638 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/732dc849-a68c-4415-85ca-594a86f70898-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:53:54.885675 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:54.885669 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsmmh\" (UniqueName: \"kubernetes.io/projected/732dc849-a68c-4415-85ca-594a86f70898-kube-api-access-qsmmh\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:53:55.171586 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:55.171489 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:53:55.275016 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:55.273132 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f955d283-k7gzw" Nov 01 20:53:55.275016 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:55.274653 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f955d283-k7gzw" event={"ID":"732dc849-a68c-4415-85ca-594a86f70898","Type":"ContainerDied","Data":"7ae6779b35e0914e160b5635e3d3b21765e8a6a6cbac7a0b808a565704a2aff3"} Nov 01 20:53:55.275016 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:55.274796 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ae6779b35e0914e160b5635e3d3b21765e8a6a6cbac7a0b808a565704a2aff3" Nov 01 20:53:55.491633 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:55.491610 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0ef08b4-mshhs" Nov 01 20:53:55.609912 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:55.609879 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d89f74fe-7a5b-411d-b82d-d77ff26b92f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d89f74fe-7a5b-411d-b82d-d77ff26b92f3-maas-api-xks-ca\") pod \"d89f74fe-7a5b-411d-b82d-d77ff26b92f3\" (UID: \"d89f74fe-7a5b-411d-b82d-d77ff26b92f3\") " Nov 01 20:53:55.610119 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:55.610011 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d89f74fe-7a5b-411d-b82d-d77ff26b92f3-kube-api-access-5ntxt\" (UniqueName: \"kubernetes.io/projected/d89f74fe-7a5b-411d-b82d-d77ff26b92f3-kube-api-access-5ntxt\") pod \"d89f74fe-7a5b-411d-b82d-d77ff26b92f3\" (UID: \"d89f74fe-7a5b-411d-b82d-d77ff26b92f3\") " Nov 01 20:53:55.610251 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:55.610234 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d89f74fe-7a5b-411d-b82d-d77ff26b92f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d89f74fe-7a5b-411d-b82d-d77ff26b92f3-maas-api-openshift-ca\") pod \"d89f74fe-7a5b-411d-b82d-d77ff26b92f3\" (UID: \"d89f74fe-7a5b-411d-b82d-d77ff26b92f3\") " Nov 01 20:53:55.611495 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:55.611462 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d89f74fe-7a5b-411d-b82d-d77ff26b92f3-maas-api-openshift-ca" pod "d89f74fe-7a5b-411d-b82d-d77ff26b92f3" (UID: "d89f74fe-7a5b-411d-b82d-d77ff26b92f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:53:55.613517 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:55.613490 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d89f74fe-7a5b-411d-b82d-d77ff26b92f3-kube-api-access-5ntxt" pod "d89f74fe-7a5b-411d-b82d-d77ff26b92f3" (UID: "d89f74fe-7a5b-411d-b82d-d77ff26b92f3"). InnerVolumeSpecName "kube-api-access-5ntxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:53:55.614442 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:55.614414 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d89f74fe-7a5b-411d-b82d-d77ff26b92f3-maas-api-xks-ca" pod "d89f74fe-7a5b-411d-b82d-d77ff26b92f3" (UID: "d89f74fe-7a5b-411d-b82d-d77ff26b92f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:53:55.713285 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:55.713246 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d89f74fe-7a5b-411d-b82d-d77ff26b92f3-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:53:55.713285 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:55.713284 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ntxt\" (UniqueName: \"kubernetes.io/projected/d89f74fe-7a5b-411d-b82d-d77ff26b92f3-kube-api-access-5ntxt\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:53:55.713488 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:55.713301 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d89f74fe-7a5b-411d-b82d-d77ff26b92f3-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:53:56.284502 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:56.284465 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0ef08b4-mshhs" event={"ID":"d89f74fe-7a5b-411d-b82d-d77ff26b92f3","Type":"ContainerDied","Data":"89fcef8f92341fab42d2fa9049f2d2b1ef22ea2b62deb10c19ed9267fd708cb5"} Nov 01 20:53:56.285024 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:56.284523 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89fcef8f92341fab42d2fa9049f2d2b1ef22ea2b62deb10c19ed9267fd708cb5" Nov 01 20:53:56.285024 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:56.284476 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0ef08b4-mshhs" Nov 01 20:53:56.419263 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:56.419234 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d0d57533-mxw8f"] Nov 01 20:53:56.420050 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:56.420031 4484 cpu_manager.go:414] "containerMap: removing container" podUID="d89f74fe-7a5b-411d-b82d-d77ff26b92f3" containerName="revoke-keys" Nov 01 20:53:56.420050 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:56.420048 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d89f74fe-7a5b-411d-b82d-d77ff26b92f3" containerName="revoke-keys" Nov 01 20:53:56.420151 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:56.420087 4484 cpu_manager.go:414] "containerMap: removing container" podUID="732dc849-a68c-4415-85ca-594a86f70898" containerName="revoke-keys" Nov 01 20:53:56.420151 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:56.420093 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="732dc849-a68c-4415-85ca-594a86f70898" containerName="revoke-keys" Nov 01 20:53:56.420237 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:56.420226 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="732dc849-a68c-4415-85ca-594a86f70898" containerName="revoke-keys" Nov 01 20:53:56.420273 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:56.420243 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="d89f74fe-7a5b-411d-b82d-d77ff26b92f3" containerName="revoke-keys" Nov 01 20:53:56.423087 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:56.423066 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d0d57533-mxw8f" Nov 01 20:53:56.476587 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:56.476548 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:53:56.476787 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:56.476588 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:53:56.476787 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:56.476663 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:53:56.476902 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:56.476863 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d0d57533-mxw8f"] Nov 01 20:53:56.520758 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:56.520729 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 20:53:56.525687 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:56.525657 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22fpp\" (UniqueName: \"kubernetes.io/projected/fd3626f4-8f4c-4466-beb5-b50487013481-kube-api-access-22fpp\") pod \"maas-api-revoke-keys-e2e-derive-d0d57533-mxw8f\" (UID: \"fd3626f4-8f4c-4466-beb5-b50487013481\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d0d57533-mxw8f" Nov 01 20:53:56.526005 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:56.525972 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd3626f4-8f4c-4466-beb5-b50487013481-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-d0d57533-mxw8f\" (UID: \"fd3626f4-8f4c-4466-beb5-b50487013481\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d0d57533-mxw8f" Nov 01 20:53:56.526126 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:56.526017 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd3626f4-8f4c-4466-beb5-b50487013481-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-d0d57533-mxw8f\" (UID: \"fd3626f4-8f4c-4466-beb5-b50487013481\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d0d57533-mxw8f" Nov 01 20:53:56.627820 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:56.627730 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22fpp\" (UniqueName: \"kubernetes.io/projected/fd3626f4-8f4c-4466-beb5-b50487013481-kube-api-access-22fpp\") pod \"maas-api-revoke-keys-e2e-derive-d0d57533-mxw8f\" (UID: \"fd3626f4-8f4c-4466-beb5-b50487013481\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d0d57533-mxw8f" Nov 01 20:53:56.628032 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:56.627983 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd3626f4-8f4c-4466-beb5-b50487013481-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-d0d57533-mxw8f\" (UID: \"fd3626f4-8f4c-4466-beb5-b50487013481\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d0d57533-mxw8f" Nov 01 20:53:56.628032 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:56.628022 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd3626f4-8f4c-4466-beb5-b50487013481-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-d0d57533-mxw8f\" (UID: \"fd3626f4-8f4c-4466-beb5-b50487013481\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d0d57533-mxw8f" Nov 01 20:53:56.628662 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:56.628639 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd3626f4-8f4c-4466-beb5-b50487013481-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-d0d57533-mxw8f\" (UID: \"fd3626f4-8f4c-4466-beb5-b50487013481\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d0d57533-mxw8f" Nov 01 20:53:56.630716 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:56.630691 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd3626f4-8f4c-4466-beb5-b50487013481-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-d0d57533-mxw8f\" (UID: \"fd3626f4-8f4c-4466-beb5-b50487013481\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d0d57533-mxw8f" Nov 01 20:53:56.685426 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:56.685392 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22fpp\" (UniqueName: \"kubernetes.io/projected/fd3626f4-8f4c-4466-beb5-b50487013481-kube-api-access-22fpp\") pod \"maas-api-revoke-keys-e2e-derive-d0d57533-mxw8f\" (UID: \"fd3626f4-8f4c-4466-beb5-b50487013481\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d0d57533-mxw8f" Nov 01 20:53:56.740320 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:56.740286 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d0d57533-mxw8f" Nov 01 20:53:57.118677 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:53:57.118646 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd3626f4_8f4c_4466_beb5_b50487013481.slice/crio-221b0d27d666b607615ee18098d4c1ae8a7755e78e0a71e77661c541b84e8ce8 WatchSource:0}: Error finding container 221b0d27d666b607615ee18098d4c1ae8a7755e78e0a71e77661c541b84e8ce8: Status 404 returned error can't find the container with id 221b0d27d666b607615ee18098d4c1ae8a7755e78e0a71e77661c541b84e8ce8 Nov 01 20:53:57.120043 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:57.119989 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d0d57533-mxw8f"] Nov 01 20:53:57.294737 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:57.294704 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d0d57533-mxw8f" event={"ID":"fd3626f4-8f4c-4466-beb5-b50487013481","Type":"ContainerStarted","Data":"221b0d27d666b607615ee18098d4c1ae8a7755e78e0a71e77661c541b84e8ce8"} Nov 01 20:53:58.260327 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:58.260292 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c232c5f-b4r48"] Nov 01 20:53:58.263888 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:58.263862 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c232c5f-b4r48" Nov 01 20:53:58.305144 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:58.305105 4484 generic.go:356] "Generic (PLEG): container finished" podID="fd3626f4-8f4c-4466-beb5-b50487013481" containerID="4de535e1e7b350ec7714fa65a8d9400089829e5350b3a0b91c82dd2462f0b1c7" exitCode=0 Nov 01 20:53:58.305526 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:58.305188 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d0d57533-mxw8f" event={"ID":"fd3626f4-8f4c-4466-beb5-b50487013481","Type":"ContainerDied","Data":"4de535e1e7b350ec7714fa65a8d9400089829e5350b3a0b91c82dd2462f0b1c7"} Nov 01 20:53:58.321842 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:58.321756 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c232c5f-b4r48"] Nov 01 20:53:58.362413 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:58.362369 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/743ddb5c-ae93-4e30-bd06-f44401ea0d68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c232c5f-b4r48\" (UID: \"743ddb5c-ae93-4e30-bd06-f44401ea0d68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c232c5f-b4r48" Nov 01 20:53:58.362636 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:58.362575 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/743ddb5c-ae93-4e30-bd06-f44401ea0d68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c232c5f-b4r48\" (UID: \"743ddb5c-ae93-4e30-bd06-f44401ea0d68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c232c5f-b4r48" Nov 01 20:53:58.362793 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:58.362761 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck6rq\" (UniqueName: \"kubernetes.io/projected/743ddb5c-ae93-4e30-bd06-f44401ea0d68-kube-api-access-ck6rq\") pod \"maas-api-revoke-sub-5c232c5f-b4r48\" (UID: \"743ddb5c-ae93-4e30-bd06-f44401ea0d68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c232c5f-b4r48" Nov 01 20:53:58.464614 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:58.464572 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/743ddb5c-ae93-4e30-bd06-f44401ea0d68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c232c5f-b4r48\" (UID: \"743ddb5c-ae93-4e30-bd06-f44401ea0d68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c232c5f-b4r48" Nov 01 20:53:58.464884 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:58.464847 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/743ddb5c-ae93-4e30-bd06-f44401ea0d68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c232c5f-b4r48\" (UID: \"743ddb5c-ae93-4e30-bd06-f44401ea0d68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c232c5f-b4r48" Nov 01 20:53:58.465119 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:58.465100 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck6rq\" (UniqueName: \"kubernetes.io/projected/743ddb5c-ae93-4e30-bd06-f44401ea0d68-kube-api-access-ck6rq\") pod \"maas-api-revoke-sub-5c232c5f-b4r48\" (UID: \"743ddb5c-ae93-4e30-bd06-f44401ea0d68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c232c5f-b4r48" Nov 01 20:53:58.465636 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:58.465613 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/743ddb5c-ae93-4e30-bd06-f44401ea0d68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c232c5f-b4r48\" (UID: \"743ddb5c-ae93-4e30-bd06-f44401ea0d68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c232c5f-b4r48" Nov 01 20:53:58.467119 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:58.467099 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/743ddb5c-ae93-4e30-bd06-f44401ea0d68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c232c5f-b4r48\" (UID: \"743ddb5c-ae93-4e30-bd06-f44401ea0d68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c232c5f-b4r48" Nov 01 20:53:58.523983 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:58.523946 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck6rq\" (UniqueName: \"kubernetes.io/projected/743ddb5c-ae93-4e30-bd06-f44401ea0d68-kube-api-access-ck6rq\") pod \"maas-api-revoke-sub-5c232c5f-b4r48\" (UID: \"743ddb5c-ae93-4e30-bd06-f44401ea0d68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c232c5f-b4r48" Nov 01 20:53:58.578794 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:58.578708 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c232c5f-b4r48" Nov 01 20:53:58.961788 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:53:58.961753 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod743ddb5c_ae93_4e30_bd06_f44401ea0d68.slice/crio-2bced953427cb32dab8987d4fa37131f197adf9215eb0de18f0858a72609d524 WatchSource:0}: Error finding container 2bced953427cb32dab8987d4fa37131f197adf9215eb0de18f0858a72609d524: Status 404 returned error can't find the container with id 2bced953427cb32dab8987d4fa37131f197adf9215eb0de18f0858a72609d524 Nov 01 20:53:58.963253 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:58.963227 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c232c5f-b4r48"] Nov 01 20:53:59.316422 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:59.316386 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c232c5f-b4r48" event={"ID":"743ddb5c-ae93-4e30-bd06-f44401ea0d68","Type":"ContainerStarted","Data":"a6f957f8225f7bb299168a28f2fec5525d8ed9e444c8b3ba5dc3fe9c36e692d4"} Nov 01 20:53:59.316907 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:59.316452 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c232c5f-b4r48" event={"ID":"743ddb5c-ae93-4e30-bd06-f44401ea0d68","Type":"ContainerStarted","Data":"2bced953427cb32dab8987d4fa37131f197adf9215eb0de18f0858a72609d524"} Nov 01 20:53:59.435015 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:59.431656 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c232c5f-b4r48" podStartSLOduration=1.4316348620000001 podStartE2EDuration="1.431634862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:53:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:53:59.431204786 +0000 UTC m=+1390.208844927" watchObservedRunningTime="2026-11-01 20:53:59.431634862 +0000 UTC m=+1390.209274998" Nov 01 20:53:59.489685 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:59.489661 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d0d57533-mxw8f" Nov 01 20:53:59.588241 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:59.588200 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd3626f4-8f4c-4466-beb5-b50487013481-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd3626f4-8f4c-4466-beb5-b50487013481-maas-api-xks-ca\") pod \"fd3626f4-8f4c-4466-beb5-b50487013481\" (UID: \"fd3626f4-8f4c-4466-beb5-b50487013481\") " Nov 01 20:53:59.588457 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:59.588421 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd3626f4-8f4c-4466-beb5-b50487013481-kube-api-access-22fpp\" (UniqueName: \"kubernetes.io/projected/fd3626f4-8f4c-4466-beb5-b50487013481-kube-api-access-22fpp\") pod \"fd3626f4-8f4c-4466-beb5-b50487013481\" (UID: \"fd3626f4-8f4c-4466-beb5-b50487013481\") " Nov 01 20:53:59.588538 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:59.588498 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd3626f4-8f4c-4466-beb5-b50487013481-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd3626f4-8f4c-4466-beb5-b50487013481-maas-api-openshift-ca\") pod \"fd3626f4-8f4c-4466-beb5-b50487013481\" (UID: \"fd3626f4-8f4c-4466-beb5-b50487013481\") " Nov 01 20:53:59.588897 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:59.588869 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd3626f4-8f4c-4466-beb5-b50487013481-maas-api-openshift-ca" pod "fd3626f4-8f4c-4466-beb5-b50487013481" (UID: "fd3626f4-8f4c-4466-beb5-b50487013481"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:53:59.589247 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:59.589229 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd3626f4-8f4c-4466-beb5-b50487013481-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:53:59.590434 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:59.590396 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd3626f4-8f4c-4466-beb5-b50487013481-maas-api-xks-ca" pod "fd3626f4-8f4c-4466-beb5-b50487013481" (UID: "fd3626f4-8f4c-4466-beb5-b50487013481"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:53:59.590540 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:59.590507 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd3626f4-8f4c-4466-beb5-b50487013481-kube-api-access-22fpp" pod "fd3626f4-8f4c-4466-beb5-b50487013481" (UID: "fd3626f4-8f4c-4466-beb5-b50487013481"). InnerVolumeSpecName "kube-api-access-22fpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:53:59.691052 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:59.691011 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd3626f4-8f4c-4466-beb5-b50487013481-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:53:59.691052 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:53:59.691038 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22fpp\" (UniqueName: \"kubernetes.io/projected/fd3626f4-8f4c-4466-beb5-b50487013481-kube-api-access-22fpp\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:54:00.326645 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:00.326610 4484 generic.go:356] "Generic (PLEG): container finished" podID="743ddb5c-ae93-4e30-bd06-f44401ea0d68" containerID="a6f957f8225f7bb299168a28f2fec5525d8ed9e444c8b3ba5dc3fe9c36e692d4" exitCode=0 Nov 01 20:54:00.327135 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:00.326701 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c232c5f-b4r48" event={"ID":"743ddb5c-ae93-4e30-bd06-f44401ea0d68","Type":"ContainerDied","Data":"a6f957f8225f7bb299168a28f2fec5525d8ed9e444c8b3ba5dc3fe9c36e692d4"} Nov 01 20:54:00.328373 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:00.328310 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d0d57533-mxw8f" Nov 01 20:54:00.328373 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:00.328328 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d0d57533-mxw8f" event={"ID":"fd3626f4-8f4c-4466-beb5-b50487013481","Type":"ContainerDied","Data":"221b0d27d666b607615ee18098d4c1ae8a7755e78e0a71e77661c541b84e8ce8"} Nov 01 20:54:00.328373 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:00.328369 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="221b0d27d666b607615ee18098d4c1ae8a7755e78e0a71e77661c541b84e8ce8" Nov 01 20:54:00.465834 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:00.465796 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:01.514335 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:01.514308 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c232c5f-b4r48" Nov 01 20:54:01.554485 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:01.554443 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/743ddb5c-ae93-4e30-bd06-f44401ea0d68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/743ddb5c-ae93-4e30-bd06-f44401ea0d68-maas-api-xks-ca\") pod \"743ddb5c-ae93-4e30-bd06-f44401ea0d68\" (UID: \"743ddb5c-ae93-4e30-bd06-f44401ea0d68\") " Nov 01 20:54:01.554698 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:01.554650 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/743ddb5c-ae93-4e30-bd06-f44401ea0d68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/743ddb5c-ae93-4e30-bd06-f44401ea0d68-maas-api-openshift-ca\") pod \"743ddb5c-ae93-4e30-bd06-f44401ea0d68\" (UID: \"743ddb5c-ae93-4e30-bd06-f44401ea0d68\") " Nov 01 20:54:01.554698 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:01.554684 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/743ddb5c-ae93-4e30-bd06-f44401ea0d68-kube-api-access-ck6rq\" (UniqueName: \"kubernetes.io/projected/743ddb5c-ae93-4e30-bd06-f44401ea0d68-kube-api-access-ck6rq\") pod \"743ddb5c-ae93-4e30-bd06-f44401ea0d68\" (UID: \"743ddb5c-ae93-4e30-bd06-f44401ea0d68\") " Nov 01 20:54:01.555062 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:01.554988 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/743ddb5c-ae93-4e30-bd06-f44401ea0d68-maas-api-openshift-ca" pod "743ddb5c-ae93-4e30-bd06-f44401ea0d68" (UID: "743ddb5c-ae93-4e30-bd06-f44401ea0d68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:54:01.555539 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:01.555520 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/743ddb5c-ae93-4e30-bd06-f44401ea0d68-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:54:01.556681 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:01.556656 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/743ddb5c-ae93-4e30-bd06-f44401ea0d68-maas-api-xks-ca" pod "743ddb5c-ae93-4e30-bd06-f44401ea0d68" (UID: "743ddb5c-ae93-4e30-bd06-f44401ea0d68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:54:01.556811 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:01.556793 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/743ddb5c-ae93-4e30-bd06-f44401ea0d68-kube-api-access-ck6rq" pod "743ddb5c-ae93-4e30-bd06-f44401ea0d68" (UID: "743ddb5c-ae93-4e30-bd06-f44401ea0d68"). InnerVolumeSpecName "kube-api-access-ck6rq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:54:01.656948 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:01.656854 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/743ddb5c-ae93-4e30-bd06-f44401ea0d68-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:54:01.656948 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:01.656887 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck6rq\" (UniqueName: \"kubernetes.io/projected/743ddb5c-ae93-4e30-bd06-f44401ea0d68-kube-api-access-ck6rq\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:54:01.875174 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:01.875131 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f955d283-ddc87bc75-fdqxb"] Nov 01 20:54:02.002008 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:02.001969 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f955d283-6798d4f4dc-b22zs"] Nov 01 20:54:02.272380 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:02.272285 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:02.528086 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:02.523812 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:02.679645 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:02.679614 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c232c5f-b4r48" Nov 01 20:54:02.679645 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:02.679626 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c232c5f-b4r48" event={"ID":"743ddb5c-ae93-4e30-bd06-f44401ea0d68","Type":"ContainerDied","Data":"2bced953427cb32dab8987d4fa37131f197adf9215eb0de18f0858a72609d524"} Nov 01 20:54:02.679847 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:02.679669 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bced953427cb32dab8987d4fa37131f197adf9215eb0de18f0858a72609d524" Nov 01 20:54:04.544269 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:04.544231 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a0ef08b4-7bd7876d5f-jcbw7"] Nov 01 20:54:04.942647 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:04.942557 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:05.109052 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:05.094617 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:05.472041 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:05.472008 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:06.876473 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:06.876425 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-f955d283-ddc87bc75-fdqxb" podUID="b15235c4-9944-48a8-b4b2-5cd176532f0c" containerName="payload-processing" containerID="cri-o://643e26895373ab05406e3e925f23690bd7cb814fd3e59d174145400aa1beefe7" gracePeriod=25 Nov 01 20:54:07.003333 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:07.003284 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-f955d283-6798d4f4dc-b22zs" podUID="d597fefb-c575-4838-87ef-bced5e8f82f5" containerName="payload-pre-processing" containerID="cri-o://c0dd97eb6cdfd6230ea1d3113c8987a8da47eb15cbf055b6d6c737fcf1b52343" gracePeriod=25 Nov 01 20:54:07.017452 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:07.017418 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d0d57533-7b666f8685-dlwfv"] Nov 01 20:54:07.093273 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:07.093236 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48a2ffe7-8nlnq"] Nov 01 20:54:07.093464 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:07.093289 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48a2ffe7-8nlnq"] Nov 01 20:54:07.151877 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:07.151838 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-d0d57533-8598cb8b99-n7sv6"] Nov 01 20:54:07.214235 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:07.214209 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f955d283-ddc87bc75-fdqxb" Nov 01 20:54:07.273488 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:07.273443 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b15235c4-9944-48a8-b4b2-5cd176532f0c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b15235c4-9944-48a8-b4b2-5cd176532f0c-plugins-config-volume\") pod \"b15235c4-9944-48a8-b4b2-5cd176532f0c\" (UID: \"b15235c4-9944-48a8-b4b2-5cd176532f0c\") " Nov 01 20:54:07.273644 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:07.273565 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b15235c4-9944-48a8-b4b2-5cd176532f0c-kube-api-access-pkssl\" (UniqueName: \"kubernetes.io/projected/b15235c4-9944-48a8-b4b2-5cd176532f0c-kube-api-access-pkssl\") pod \"b15235c4-9944-48a8-b4b2-5cd176532f0c\" (UID: \"b15235c4-9944-48a8-b4b2-5cd176532f0c\") " Nov 01 20:54:07.273968 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:07.273943 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b15235c4-9944-48a8-b4b2-5cd176532f0c-plugins-config-volume" pod "b15235c4-9944-48a8-b4b2-5cd176532f0c" (UID: "b15235c4-9944-48a8-b4b2-5cd176532f0c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:54:07.274599 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:07.274569 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b15235c4-9944-48a8-b4b2-5cd176532f0c-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:54:07.276039 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:07.276011 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b15235c4-9944-48a8-b4b2-5cd176532f0c-kube-api-access-pkssl" pod "b15235c4-9944-48a8-b4b2-5cd176532f0c" (UID: "b15235c4-9944-48a8-b4b2-5cd176532f0c"). InnerVolumeSpecName "kube-api-access-pkssl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:54:07.376352 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:07.376312 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkssl\" (UniqueName: \"kubernetes.io/projected/b15235c4-9944-48a8-b4b2-5cd176532f0c-kube-api-access-pkssl\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:54:07.384773 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:07.384739 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1479472c-ffe5-49cc-98b9-5fba410f0c4c" path="/var/lib/kubelet/pods/1479472c-ffe5-49cc-98b9-5fba410f0c4c/volumes" Nov 01 20:54:07.397462 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:07.397433 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:07.406388 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:07.406324 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f955d283-6798d4f4dc-b22zs" Nov 01 20:54:07.487338 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:07.487301 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d597fefb-c575-4838-87ef-bced5e8f82f5-kube-api-access-bpfsj\" (UniqueName: \"kubernetes.io/projected/d597fefb-c575-4838-87ef-bced5e8f82f5-kube-api-access-bpfsj\") pod \"d597fefb-c575-4838-87ef-bced5e8f82f5\" (UID: \"d597fefb-c575-4838-87ef-bced5e8f82f5\") " Nov 01 20:54:07.503941 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:07.503905 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d597fefb-c575-4838-87ef-bced5e8f82f5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d597fefb-c575-4838-87ef-bced5e8f82f5-plugins-config-volume\") pod \"d597fefb-c575-4838-87ef-bced5e8f82f5\" (UID: \"d597fefb-c575-4838-87ef-bced5e8f82f5\") " Nov 01 20:54:07.553255 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:07.553174 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d597fefb-c575-4838-87ef-bced5e8f82f5-plugins-config-volume" pod "d597fefb-c575-4838-87ef-bced5e8f82f5" (UID: "d597fefb-c575-4838-87ef-bced5e8f82f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:54:07.584020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:07.583429 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d597fefb-c575-4838-87ef-bced5e8f82f5-kube-api-access-bpfsj" pod "d597fefb-c575-4838-87ef-bced5e8f82f5" (UID: "d597fefb-c575-4838-87ef-bced5e8f82f5"). InnerVolumeSpecName "kube-api-access-bpfsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:54:07.621731 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:07.621691 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:07.622248 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:07.621864 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d597fefb-c575-4838-87ef-bced5e8f82f5-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:54:07.622422 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:07.622404 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpfsj\" (UniqueName: \"kubernetes.io/projected/d597fefb-c575-4838-87ef-bced5e8f82f5-kube-api-access-bpfsj\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:54:07.827836 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:07.827804 4484 generic.go:356] "Generic (PLEG): container finished" podID="b15235c4-9944-48a8-b4b2-5cd176532f0c" containerID="643e26895373ab05406e3e925f23690bd7cb814fd3e59d174145400aa1beefe7" exitCode=0 Nov 01 20:54:07.828047 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:07.827924 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f955d283-ddc87bc75-fdqxb" event={"ID":"b15235c4-9944-48a8-b4b2-5cd176532f0c","Type":"ContainerDied","Data":"643e26895373ab05406e3e925f23690bd7cb814fd3e59d174145400aa1beefe7"} Nov 01 20:54:07.828047 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:07.827964 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f955d283-ddc87bc75-fdqxb" event={"ID":"b15235c4-9944-48a8-b4b2-5cd176532f0c","Type":"ContainerDied","Data":"c7f8f1d803a94f640167b694a5a28090f8d9af2ea72b8ccc2a59680ec7969da8"} Nov 01 20:54:07.828047 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:07.828030 4484 scope.go:122] "RemoveContainer" containerID="643e26895373ab05406e3e925f23690bd7cb814fd3e59d174145400aa1beefe7" Nov 01 20:54:07.828165 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:07.828127 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f955d283-ddc87bc75-fdqxb" Nov 01 20:54:07.829802 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:07.829776 4484 generic.go:356] "Generic (PLEG): container finished" podID="d597fefb-c575-4838-87ef-bced5e8f82f5" containerID="c0dd97eb6cdfd6230ea1d3113c8987a8da47eb15cbf055b6d6c737fcf1b52343" exitCode=0 Nov 01 20:54:07.829904 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:07.829825 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f955d283-6798d4f4dc-b22zs" Nov 01 20:54:07.830027 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:07.829971 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f955d283-6798d4f4dc-b22zs" event={"ID":"d597fefb-c575-4838-87ef-bced5e8f82f5","Type":"ContainerDied","Data":"c0dd97eb6cdfd6230ea1d3113c8987a8da47eb15cbf055b6d6c737fcf1b52343"} Nov 01 20:54:07.830092 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:07.830055 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f955d283-6798d4f4dc-b22zs" event={"ID":"d597fefb-c575-4838-87ef-bced5e8f82f5","Type":"ContainerDied","Data":"f463ca1fe6255c79d2e426ff0b55c93a51b7f21c919a28259a16db7c95f175a5"} Nov 01 20:54:07.841649 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:07.841627 4484 scope.go:122] "RemoveContainer" containerID="643e26895373ab05406e3e925f23690bd7cb814fd3e59d174145400aa1beefe7" Nov 01 20:54:07.841974 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:54:07.841953 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"643e26895373ab05406e3e925f23690bd7cb814fd3e59d174145400aa1beefe7\": container with ID starting with 643e26895373ab05406e3e925f23690bd7cb814fd3e59d174145400aa1beefe7 not found: ID does not exist" containerID="643e26895373ab05406e3e925f23690bd7cb814fd3e59d174145400aa1beefe7" Nov 01 20:54:07.842129 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:07.841988 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"643e26895373ab05406e3e925f23690bd7cb814fd3e59d174145400aa1beefe7"} err="failed to get container status \"643e26895373ab05406e3e925f23690bd7cb814fd3e59d174145400aa1beefe7\": rpc error: code = NotFound desc = could not find container \"643e26895373ab05406e3e925f23690bd7cb814fd3e59d174145400aa1beefe7\": container with ID starting with 643e26895373ab05406e3e925f23690bd7cb814fd3e59d174145400aa1beefe7 not found: ID does not exist" Nov 01 20:54:07.842129 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:07.842038 4484 scope.go:122] "RemoveContainer" containerID="c0dd97eb6cdfd6230ea1d3113c8987a8da47eb15cbf055b6d6c737fcf1b52343" Nov 01 20:54:07.854248 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:07.853203 4484 scope.go:122] "RemoveContainer" containerID="c0dd97eb6cdfd6230ea1d3113c8987a8da47eb15cbf055b6d6c737fcf1b52343" Nov 01 20:54:07.854248 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:54:07.853581 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0dd97eb6cdfd6230ea1d3113c8987a8da47eb15cbf055b6d6c737fcf1b52343\": container with ID starting with c0dd97eb6cdfd6230ea1d3113c8987a8da47eb15cbf055b6d6c737fcf1b52343 not found: ID does not exist" containerID="c0dd97eb6cdfd6230ea1d3113c8987a8da47eb15cbf055b6d6c737fcf1b52343" Nov 01 20:54:07.854248 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:07.853614 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0dd97eb6cdfd6230ea1d3113c8987a8da47eb15cbf055b6d6c737fcf1b52343"} err="failed to get container status \"c0dd97eb6cdfd6230ea1d3113c8987a8da47eb15cbf055b6d6c737fcf1b52343\": rpc error: code = NotFound desc = could not find container \"c0dd97eb6cdfd6230ea1d3113c8987a8da47eb15cbf055b6d6c737fcf1b52343\": container with ID starting with c0dd97eb6cdfd6230ea1d3113c8987a8da47eb15cbf055b6d6c737fcf1b52343 not found: ID does not exist" Nov 01 20:54:08.008516 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:08.008467 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f955d283-ddc87bc75-fdqxb"] Nov 01 20:54:08.014007 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:08.013967 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f955d283-ddc87bc75-fdqxb"] Nov 01 20:54:08.123805 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:08.123725 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f955d283-6798d4f4dc-b22zs"] Nov 01 20:54:08.129614 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:08.129582 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f955d283-6798d4f4dc-b22zs"] Nov 01 20:54:09.029201 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:09.029166 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:09.373520 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:09.373444 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b15235c4-9944-48a8-b4b2-5cd176532f0c" path="/var/lib/kubelet/pods/b15235c4-9944-48a8-b4b2-5cd176532f0c/volumes" Nov 01 20:54:09.373927 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:09.373913 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d597fefb-c575-4838-87ef-bced5e8f82f5" path="/var/lib/kubelet/pods/d597fefb-c575-4838-87ef-bced5e8f82f5/volumes" Nov 01 20:54:09.544653 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:09.544608 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-a0ef08b4-7bd7876d5f-jcbw7" podUID="e8ce7f55-690d-4fc8-944c-d7a7c82f6126" containerName="payload-processing" containerID="cri-o://e3d22207e0b88ef10507c9c3a5f93e35d7095c8aab702b38e041baf9c6c9aa56" gracePeriod=25 Nov 01 20:54:09.874961 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:09.874937 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a0ef08b4-7bd7876d5f-jcbw7" Nov 01 20:54:09.936234 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:09.936146 4484 generic.go:356] "Generic (PLEG): container finished" podID="e8ce7f55-690d-4fc8-944c-d7a7c82f6126" containerID="e3d22207e0b88ef10507c9c3a5f93e35d7095c8aab702b38e041baf9c6c9aa56" exitCode=0 Nov 01 20:54:09.936234 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:09.936188 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a0ef08b4-7bd7876d5f-jcbw7" Nov 01 20:54:09.936234 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:09.936224 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a0ef08b4-7bd7876d5f-jcbw7" event={"ID":"e8ce7f55-690d-4fc8-944c-d7a7c82f6126","Type":"ContainerDied","Data":"e3d22207e0b88ef10507c9c3a5f93e35d7095c8aab702b38e041baf9c6c9aa56"} Nov 01 20:54:09.936496 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:09.936274 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a0ef08b4-7bd7876d5f-jcbw7" event={"ID":"e8ce7f55-690d-4fc8-944c-d7a7c82f6126","Type":"ContainerDied","Data":"87010bfd676d486d8a005a9668be5d6cc647da760fd6522c6c90ab10ad82a258"} Nov 01 20:54:09.936496 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:09.936298 4484 scope.go:122] "RemoveContainer" containerID="e3d22207e0b88ef10507c9c3a5f93e35d7095c8aab702b38e041baf9c6c9aa56" Nov 01 20:54:09.947706 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:09.947688 4484 scope.go:122] "RemoveContainer" containerID="e3d22207e0b88ef10507c9c3a5f93e35d7095c8aab702b38e041baf9c6c9aa56" Nov 01 20:54:09.948023 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:54:09.947986 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3d22207e0b88ef10507c9c3a5f93e35d7095c8aab702b38e041baf9c6c9aa56\": container with ID starting with e3d22207e0b88ef10507c9c3a5f93e35d7095c8aab702b38e041baf9c6c9aa56 not found: ID does not exist" containerID="e3d22207e0b88ef10507c9c3a5f93e35d7095c8aab702b38e041baf9c6c9aa56" Nov 01 20:54:09.948084 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:09.948037 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3d22207e0b88ef10507c9c3a5f93e35d7095c8aab702b38e041baf9c6c9aa56"} err="failed to get container status \"e3d22207e0b88ef10507c9c3a5f93e35d7095c8aab702b38e041baf9c6c9aa56\": rpc error: code = NotFound desc = could not find container \"e3d22207e0b88ef10507c9c3a5f93e35d7095c8aab702b38e041baf9c6c9aa56\": container with ID starting with e3d22207e0b88ef10507c9c3a5f93e35d7095c8aab702b38e041baf9c6c9aa56 not found: ID does not exist" Nov 01 20:54:10.026191 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:10.026153 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8ce7f55-690d-4fc8-944c-d7a7c82f6126-kube-api-access-njzjl\" (UniqueName: \"kubernetes.io/projected/e8ce7f55-690d-4fc8-944c-d7a7c82f6126-kube-api-access-njzjl\") pod \"e8ce7f55-690d-4fc8-944c-d7a7c82f6126\" (UID: \"e8ce7f55-690d-4fc8-944c-d7a7c82f6126\") " Nov 01 20:54:10.026338 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:10.026313 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8ce7f55-690d-4fc8-944c-d7a7c82f6126-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8ce7f55-690d-4fc8-944c-d7a7c82f6126-plugins-config-volume\") pod \"e8ce7f55-690d-4fc8-944c-d7a7c82f6126\" (UID: \"e8ce7f55-690d-4fc8-944c-d7a7c82f6126\") " Nov 01 20:54:10.026683 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:10.026653 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8ce7f55-690d-4fc8-944c-d7a7c82f6126-plugins-config-volume" pod "e8ce7f55-690d-4fc8-944c-d7a7c82f6126" (UID: "e8ce7f55-690d-4fc8-944c-d7a7c82f6126"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:54:10.027261 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:10.027238 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8ce7f55-690d-4fc8-944c-d7a7c82f6126-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:54:10.028296 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:10.028273 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8ce7f55-690d-4fc8-944c-d7a7c82f6126-kube-api-access-njzjl" pod "e8ce7f55-690d-4fc8-944c-d7a7c82f6126" (UID: "e8ce7f55-690d-4fc8-944c-d7a7c82f6126"). InnerVolumeSpecName "kube-api-access-njzjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:54:10.128762 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:10.128722 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njzjl\" (UniqueName: \"kubernetes.io/projected/e8ce7f55-690d-4fc8-944c-d7a7c82f6126-kube-api-access-njzjl\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:54:10.221263 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:10.221159 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:10.419120 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:10.419082 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a0ef08b4-7bd7876d5f-jcbw7"] Nov 01 20:54:10.424668 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:10.424637 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a0ef08b4-7bd7876d5f-jcbw7"] Nov 01 20:54:11.373896 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:11.373862 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8ce7f55-690d-4fc8-944c-d7a7c82f6126" path="/var/lib/kubelet/pods/e8ce7f55-690d-4fc8-944c-d7a7c82f6126/volumes" Nov 01 20:54:12.018748 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:12.018704 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-d0d57533-7b666f8685-dlwfv" podUID="b95260f8-613a-4ab5-9780-4e8c271a2bbe" containerName="payload-processing" containerID="cri-o://b1c070c5ee5a88062051ba0e1ee3d346eb7d9ab66741e7bd1b202b2a8b48efd2" gracePeriod=25 Nov 01 20:54:12.153058 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:12.153014 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-d0d57533-8598cb8b99-n7sv6" podUID="59b0a447-bbb7-441e-977f-0fa8fe7696d0" containerName="payload-pre-processing" containerID="cri-o://9395ed24f40cd1e58d431811f321bca45f3887c0f574a23c12910f044ac36f8e" gracePeriod=25 Nov 01 20:54:12.326169 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:12.326138 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-d0d57533-7b666f8685-dlwfv" Nov 01 20:54:12.465974 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:12.465942 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-d0d57533-8598cb8b99-n7sv6" Nov 01 20:54:12.470558 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:12.470524 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b95260f8-613a-4ab5-9780-4e8c271a2bbe-kube-api-access-9j5pv\" (UniqueName: \"kubernetes.io/projected/b95260f8-613a-4ab5-9780-4e8c271a2bbe-kube-api-access-9j5pv\") pod \"b95260f8-613a-4ab5-9780-4e8c271a2bbe\" (UID: \"b95260f8-613a-4ab5-9780-4e8c271a2bbe\") " Nov 01 20:54:12.470827 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:12.470806 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b95260f8-613a-4ab5-9780-4e8c271a2bbe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b95260f8-613a-4ab5-9780-4e8c271a2bbe-plugins-config-volume\") pod \"b95260f8-613a-4ab5-9780-4e8c271a2bbe\" (UID: \"b95260f8-613a-4ab5-9780-4e8c271a2bbe\") " Nov 01 20:54:12.471118 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:12.471095 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b95260f8-613a-4ab5-9780-4e8c271a2bbe-plugins-config-volume" pod "b95260f8-613a-4ab5-9780-4e8c271a2bbe" (UID: "b95260f8-613a-4ab5-9780-4e8c271a2bbe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:54:12.471552 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:12.471529 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b95260f8-613a-4ab5-9780-4e8c271a2bbe-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:54:12.473311 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:12.473281 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b95260f8-613a-4ab5-9780-4e8c271a2bbe-kube-api-access-9j5pv" pod "b95260f8-613a-4ab5-9780-4e8c271a2bbe" (UID: "b95260f8-613a-4ab5-9780-4e8c271a2bbe"). InnerVolumeSpecName "kube-api-access-9j5pv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:54:12.572724 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:12.572636 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59b0a447-bbb7-441e-977f-0fa8fe7696d0-kube-api-access-42x8w\" (UniqueName: \"kubernetes.io/projected/59b0a447-bbb7-441e-977f-0fa8fe7696d0-kube-api-access-42x8w\") pod \"59b0a447-bbb7-441e-977f-0fa8fe7696d0\" (UID: \"59b0a447-bbb7-441e-977f-0fa8fe7696d0\") " Nov 01 20:54:12.572871 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:12.572834 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59b0a447-bbb7-441e-977f-0fa8fe7696d0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59b0a447-bbb7-441e-977f-0fa8fe7696d0-plugins-config-volume\") pod \"59b0a447-bbb7-441e-977f-0fa8fe7696d0\" (UID: \"59b0a447-bbb7-441e-977f-0fa8fe7696d0\") " Nov 01 20:54:12.573260 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:12.573232 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59b0a447-bbb7-441e-977f-0fa8fe7696d0-plugins-config-volume" pod "59b0a447-bbb7-441e-977f-0fa8fe7696d0" (UID: "59b0a447-bbb7-441e-977f-0fa8fe7696d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:54:12.573502 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:12.573485 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59b0a447-bbb7-441e-977f-0fa8fe7696d0-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:54:12.573571 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:12.573504 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9j5pv\" (UniqueName: \"kubernetes.io/projected/b95260f8-613a-4ab5-9780-4e8c271a2bbe-kube-api-access-9j5pv\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:54:12.574889 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:12.574868 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59b0a447-bbb7-441e-977f-0fa8fe7696d0-kube-api-access-42x8w" pod "59b0a447-bbb7-441e-977f-0fa8fe7696d0" (UID: "59b0a447-bbb7-441e-977f-0fa8fe7696d0"). InnerVolumeSpecName "kube-api-access-42x8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:54:12.675068 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:12.675033 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42x8w\" (UniqueName: \"kubernetes.io/projected/59b0a447-bbb7-441e-977f-0fa8fe7696d0-kube-api-access-42x8w\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:54:12.966745 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:12.966654 4484 generic.go:356] "Generic (PLEG): container finished" podID="b95260f8-613a-4ab5-9780-4e8c271a2bbe" containerID="b1c070c5ee5a88062051ba0e1ee3d346eb7d9ab66741e7bd1b202b2a8b48efd2" exitCode=0 Nov 01 20:54:12.966745 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:12.966699 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-d0d57533-7b666f8685-dlwfv" Nov 01 20:54:12.966965 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:12.966740 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-d0d57533-7b666f8685-dlwfv" event={"ID":"b95260f8-613a-4ab5-9780-4e8c271a2bbe","Type":"ContainerDied","Data":"b1c070c5ee5a88062051ba0e1ee3d346eb7d9ab66741e7bd1b202b2a8b48efd2"} Nov 01 20:54:12.966965 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:12.966792 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-d0d57533-7b666f8685-dlwfv" event={"ID":"b95260f8-613a-4ab5-9780-4e8c271a2bbe","Type":"ContainerDied","Data":"b0a7ef1bfd27e3c69fa8b5705443a03b168466106ef85a0ce09693a232334baf"} Nov 01 20:54:12.966965 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:12.966825 4484 scope.go:122] "RemoveContainer" containerID="b1c070c5ee5a88062051ba0e1ee3d346eb7d9ab66741e7bd1b202b2a8b48efd2" Nov 01 20:54:12.968407 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:12.968381 4484 generic.go:356] "Generic (PLEG): container finished" podID="59b0a447-bbb7-441e-977f-0fa8fe7696d0" containerID="9395ed24f40cd1e58d431811f321bca45f3887c0f574a23c12910f044ac36f8e" exitCode=0 Nov 01 20:54:12.968533 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:12.968423 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-d0d57533-8598cb8b99-n7sv6" Nov 01 20:54:12.968533 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:12.968473 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-d0d57533-8598cb8b99-n7sv6" event={"ID":"59b0a447-bbb7-441e-977f-0fa8fe7696d0","Type":"ContainerDied","Data":"9395ed24f40cd1e58d431811f321bca45f3887c0f574a23c12910f044ac36f8e"} Nov 01 20:54:12.968533 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:12.968523 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-d0d57533-8598cb8b99-n7sv6" event={"ID":"59b0a447-bbb7-441e-977f-0fa8fe7696d0","Type":"ContainerDied","Data":"edde11fcd074e34c19a00259ed0b6568edbdc8a5cd7e7e69391c5b45ce7c864a"} Nov 01 20:54:12.987397 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:12.987376 4484 scope.go:122] "RemoveContainer" containerID="b1c070c5ee5a88062051ba0e1ee3d346eb7d9ab66741e7bd1b202b2a8b48efd2" Nov 01 20:54:12.987979 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:54:12.987945 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1c070c5ee5a88062051ba0e1ee3d346eb7d9ab66741e7bd1b202b2a8b48efd2\": container with ID starting with b1c070c5ee5a88062051ba0e1ee3d346eb7d9ab66741e7bd1b202b2a8b48efd2 not found: ID does not exist" containerID="b1c070c5ee5a88062051ba0e1ee3d346eb7d9ab66741e7bd1b202b2a8b48efd2" Nov 01 20:54:12.988090 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:12.988009 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1c070c5ee5a88062051ba0e1ee3d346eb7d9ab66741e7bd1b202b2a8b48efd2"} err="failed to get container status \"b1c070c5ee5a88062051ba0e1ee3d346eb7d9ab66741e7bd1b202b2a8b48efd2\": rpc error: code = NotFound desc = could not find container \"b1c070c5ee5a88062051ba0e1ee3d346eb7d9ab66741e7bd1b202b2a8b48efd2\": container with ID starting with b1c070c5ee5a88062051ba0e1ee3d346eb7d9ab66741e7bd1b202b2a8b48efd2 not found: ID does not exist" Nov 01 20:54:12.988090 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:12.988038 4484 scope.go:122] "RemoveContainer" containerID="9395ed24f40cd1e58d431811f321bca45f3887c0f574a23c12910f044ac36f8e" Nov 01 20:54:13.001833 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:13.001810 4484 scope.go:122] "RemoveContainer" containerID="9395ed24f40cd1e58d431811f321bca45f3887c0f574a23c12910f044ac36f8e" Nov 01 20:54:13.002221 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:54:13.002195 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9395ed24f40cd1e58d431811f321bca45f3887c0f574a23c12910f044ac36f8e\": container with ID starting with 9395ed24f40cd1e58d431811f321bca45f3887c0f574a23c12910f044ac36f8e not found: ID does not exist" containerID="9395ed24f40cd1e58d431811f321bca45f3887c0f574a23c12910f044ac36f8e" Nov 01 20:54:13.002286 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:13.002233 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9395ed24f40cd1e58d431811f321bca45f3887c0f574a23c12910f044ac36f8e"} err="failed to get container status \"9395ed24f40cd1e58d431811f321bca45f3887c0f574a23c12910f044ac36f8e\": rpc error: code = NotFound desc = could not find container \"9395ed24f40cd1e58d431811f321bca45f3887c0f574a23c12910f044ac36f8e\": container with ID starting with 9395ed24f40cd1e58d431811f321bca45f3887c0f574a23c12910f044ac36f8e not found: ID does not exist" Nov 01 20:54:13.136137 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:13.136101 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-d0d57533-8598cb8b99-n7sv6"] Nov 01 20:54:13.186938 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:13.186899 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-d0d57533-8598cb8b99-n7sv6"] Nov 01 20:54:13.298071 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:13.298036 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d0d57533-7b666f8685-dlwfv"] Nov 01 20:54:13.303491 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:13.303464 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d0d57533-7b666f8685-dlwfv"] Nov 01 20:54:13.374850 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:13.374812 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59b0a447-bbb7-441e-977f-0fa8fe7696d0" path="/var/lib/kubelet/pods/59b0a447-bbb7-441e-977f-0fa8fe7696d0/volumes" Nov 01 20:54:13.375375 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:13.375358 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b95260f8-613a-4ab5-9780-4e8c271a2bbe" path="/var/lib/kubelet/pods/b95260f8-613a-4ab5-9780-4e8c271a2bbe/volumes" Nov 01 20:54:14.991464 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:14.991434 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:15.180776 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:15.180739 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:15.768280 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:54:15.768244 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 20:54:19.831533 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:19.831494 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:24.545459 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:24.545425 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:26.806635 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:26.806587 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:26.985004 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:26.984952 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:27.373649 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:27.373618 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:29.277252 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:29.277215 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:31.052703 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:31.052671 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b23881c3-9m222"] Nov 01 20:54:31.103710 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:31.103672 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b23881c3-9m222"] Nov 01 20:54:31.376277 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:31.376192 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2704b115-3839-4088-bc5b-fb6fc97760ee" path="/var/lib/kubelet/pods/2704b115-3839-4088-bc5b-fb6fc97760ee/volumes" Nov 01 20:54:33.089076 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:33.089045 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:33.252928 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:33.252872 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:33.511353 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:33.511312 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:34.666461 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:34.666363 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:35.038905 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:35.038865 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:35.300756 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:35.300656 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:35.818015 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:35.813636 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f955d283-k7gzw"] Nov 01 20:54:35.823129 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:35.821836 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:35.887127 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:35.887093 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f955d283-k7gzw"] Nov 01 20:54:35.895927 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:35.895889 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv"] Nov 01 20:54:35.896241 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:35.896180 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv" podUID="dff8b3d7-cade-4cba-9e07-3e912f0cabc8" containerName="istio-proxy" containerID="cri-o://578a8ddb4623e84b75b7c4fb4aa39ed74914603c2f27dff3ccc1556c9a09d137" gracePeriod=30 Nov 01 20:54:36.002789 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:36.002753 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:36.370608 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:36.370559 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:36.578555 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:36.578385 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:36.753051 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:36.745754 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:37.215345 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.215263 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:37.254869 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.254833 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr"] Nov 01 20:54:37.255581 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.255566 4484 cpu_manager.go:414] "containerMap: removing container" podUID="b15235c4-9944-48a8-b4b2-5cd176532f0c" containerName="payload-processing" Nov 01 20:54:37.255638 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.255583 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b15235c4-9944-48a8-b4b2-5cd176532f0c" containerName="payload-processing" Nov 01 20:54:37.255638 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.255599 4484 cpu_manager.go:414] "containerMap: removing container" podUID="e8ce7f55-690d-4fc8-944c-d7a7c82f6126" containerName="payload-processing" Nov 01 20:54:37.255638 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.255604 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8ce7f55-690d-4fc8-944c-d7a7c82f6126" containerName="payload-processing" Nov 01 20:54:37.255638 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.255620 4484 cpu_manager.go:414] "containerMap: removing container" podUID="59b0a447-bbb7-441e-977f-0fa8fe7696d0" containerName="payload-pre-processing" Nov 01 20:54:37.255638 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.255626 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59b0a447-bbb7-441e-977f-0fa8fe7696d0" containerName="payload-pre-processing" Nov 01 20:54:37.255638 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.255635 4484 cpu_manager.go:414] "containerMap: removing container" podUID="743ddb5c-ae93-4e30-bd06-f44401ea0d68" containerName="revoke-keys" Nov 01 20:54:37.255638 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.255640 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="743ddb5c-ae93-4e30-bd06-f44401ea0d68" containerName="revoke-keys" Nov 01 20:54:37.255858 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.255657 4484 cpu_manager.go:414] "containerMap: removing container" podUID="d597fefb-c575-4838-87ef-bced5e8f82f5" containerName="payload-pre-processing" Nov 01 20:54:37.255858 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.255662 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d597fefb-c575-4838-87ef-bced5e8f82f5" containerName="payload-pre-processing" Nov 01 20:54:37.255858 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.255672 4484 cpu_manager.go:414] "containerMap: removing container" podUID="b95260f8-613a-4ab5-9780-4e8c271a2bbe" containerName="payload-processing" Nov 01 20:54:37.255858 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.255677 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b95260f8-613a-4ab5-9780-4e8c271a2bbe" containerName="payload-processing" Nov 01 20:54:37.255858 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.255683 4484 cpu_manager.go:414] "containerMap: removing container" podUID="fd3626f4-8f4c-4466-beb5-b50487013481" containerName="revoke-keys" Nov 01 20:54:37.255858 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.255687 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd3626f4-8f4c-4466-beb5-b50487013481" containerName="revoke-keys" Nov 01 20:54:37.255858 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.255779 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="b15235c4-9944-48a8-b4b2-5cd176532f0c" containerName="payload-processing" Nov 01 20:54:37.255858 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.255787 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8ce7f55-690d-4fc8-944c-d7a7c82f6126" containerName="payload-processing" Nov 01 20:54:37.255858 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.255796 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="59b0a447-bbb7-441e-977f-0fa8fe7696d0" containerName="payload-pre-processing" Nov 01 20:54:37.255858 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.255801 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="743ddb5c-ae93-4e30-bd06-f44401ea0d68" containerName="revoke-keys" Nov 01 20:54:37.255858 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.255809 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="d597fefb-c575-4838-87ef-bced5e8f82f5" containerName="payload-pre-processing" Nov 01 20:54:37.255858 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.255814 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="b95260f8-613a-4ab5-9780-4e8c271a2bbe" containerName="payload-processing" Nov 01 20:54:37.255858 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.255819 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd3626f4-8f4c-4466-beb5-b50487013481" containerName="revoke-keys" Nov 01 20:54:37.259749 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.259727 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr" Nov 01 20:54:37.320635 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.320592 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr"] Nov 01 20:54:37.361741 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.361707 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-8ddfb06d-gw-openshift-default-dockercfg-b9ztt\"" Nov 01 20:54:37.362496 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.362468 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-workload-certs\") pod \"e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr\" (UID: \"ce125f0b-43ab-40bf-ae39-5dc6e195ce39\") " pod="openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr" Nov 01 20:54:37.362633 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.362615 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-istio-envoy\") pod \"e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr\" (UID: \"ce125f0b-43ab-40bf-ae39-5dc6e195ce39\") " pod="openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr" Nov 01 20:54:37.362706 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.362688 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4mg4\" (UniqueName: \"kubernetes.io/projected/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-kube-api-access-r4mg4\") pod \"e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr\" (UID: \"ce125f0b-43ab-40bf-ae39-5dc6e195ce39\") " pod="openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr" Nov 01 20:54:37.362805 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.362785 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-istiod-ca-cert\") pod \"e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr\" (UID: \"ce125f0b-43ab-40bf-ae39-5dc6e195ce39\") " pod="openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr" Nov 01 20:54:37.362845 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.362823 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-istio-token\") pod \"e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr\" (UID: \"ce125f0b-43ab-40bf-ae39-5dc6e195ce39\") " pod="openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr" Nov 01 20:54:37.362903 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.362890 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-workload-socket\") pod \"e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr\" (UID: \"ce125f0b-43ab-40bf-ae39-5dc6e195ce39\") " pod="openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr" Nov 01 20:54:37.363032 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.362983 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-credential-socket\") pod \"e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr\" (UID: \"ce125f0b-43ab-40bf-ae39-5dc6e195ce39\") " pod="openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr" Nov 01 20:54:37.363073 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.363045 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-istio-podinfo\") pod \"e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr\" (UID: \"ce125f0b-43ab-40bf-ae39-5dc6e195ce39\") " pod="openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr" Nov 01 20:54:37.363123 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.363112 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-istio-data\") pod \"e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr\" (UID: \"ce125f0b-43ab-40bf-ae39-5dc6e195ce39\") " pod="openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr" Nov 01 20:54:37.375537 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.375508 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="732dc849-a68c-4415-85ca-594a86f70898" path="/var/lib/kubelet/pods/732dc849-a68c-4415-85ca-594a86f70898/volumes" Nov 01 20:54:37.396102 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.396064 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0ef08b4-mshhs"] Nov 01 20:54:37.423265 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.423206 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0ef08b4-mshhs"] Nov 01 20:54:37.451189 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.451151 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:37.468350 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.466445 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-istio-envoy\") pod \"e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr\" (UID: \"ce125f0b-43ab-40bf-ae39-5dc6e195ce39\") " pod="openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr" Nov 01 20:54:37.468350 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.466563 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4mg4\" (UniqueName: \"kubernetes.io/projected/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-kube-api-access-r4mg4\") pod \"e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr\" (UID: \"ce125f0b-43ab-40bf-ae39-5dc6e195ce39\") " pod="openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr" Nov 01 20:54:37.468350 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.466664 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-istiod-ca-cert\") pod \"e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr\" (UID: \"ce125f0b-43ab-40bf-ae39-5dc6e195ce39\") " pod="openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr" Nov 01 20:54:37.468350 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.466700 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-istio-token\") pod \"e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr\" (UID: \"ce125f0b-43ab-40bf-ae39-5dc6e195ce39\") " pod="openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr" Nov 01 20:54:37.491095 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.472125 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-workload-socket\") pod \"e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr\" (UID: \"ce125f0b-43ab-40bf-ae39-5dc6e195ce39\") " pod="openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr" Nov 01 20:54:37.491095 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.472260 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-credential-socket\") pod \"e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr\" (UID: \"ce125f0b-43ab-40bf-ae39-5dc6e195ce39\") " pod="openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr" Nov 01 20:54:37.491095 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.472297 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-istio-podinfo\") pod \"e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr\" (UID: \"ce125f0b-43ab-40bf-ae39-5dc6e195ce39\") " pod="openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr" Nov 01 20:54:37.491095 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.472356 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-istio-data\") pod \"e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr\" (UID: \"ce125f0b-43ab-40bf-ae39-5dc6e195ce39\") " pod="openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr" Nov 01 20:54:37.491095 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.472502 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-workload-certs\") pod \"e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr\" (UID: \"ce125f0b-43ab-40bf-ae39-5dc6e195ce39\") " pod="openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr" Nov 01 20:54:37.491095 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.473108 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-workload-certs\") pod \"e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr\" (UID: \"ce125f0b-43ab-40bf-ae39-5dc6e195ce39\") " pod="openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr" Nov 01 20:54:37.495613 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.495579 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-workload-socket\") pod \"e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr\" (UID: \"ce125f0b-43ab-40bf-ae39-5dc6e195ce39\") " pod="openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr" Nov 01 20:54:37.506167 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.496586 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-istiod-ca-cert\") pod \"e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr\" (UID: \"ce125f0b-43ab-40bf-ae39-5dc6e195ce39\") " pod="openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr" Nov 01 20:54:37.541019 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.498859 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-istio-data\") pod \"e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr\" (UID: \"ce125f0b-43ab-40bf-ae39-5dc6e195ce39\") " pod="openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr" Nov 01 20:54:37.541019 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.499124 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-credential-socket\") pod \"e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr\" (UID: \"ce125f0b-43ab-40bf-ae39-5dc6e195ce39\") " pod="openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr" Nov 01 20:54:37.541019 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.512419 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-istio-podinfo\") pod \"e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr\" (UID: \"ce125f0b-43ab-40bf-ae39-5dc6e195ce39\") " pod="openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr" Nov 01 20:54:37.552700 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.552624 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-istio-envoy\") pod \"e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr\" (UID: \"ce125f0b-43ab-40bf-ae39-5dc6e195ce39\") " pod="openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr" Nov 01 20:54:37.590014 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.583980 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4mg4\" (UniqueName: \"kubernetes.io/projected/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-kube-api-access-r4mg4\") pod \"e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr\" (UID: \"ce125f0b-43ab-40bf-ae39-5dc6e195ce39\") " pod="openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr" Nov 01 20:54:37.591639 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.591609 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-istio-token\") pod \"e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr\" (UID: \"ce125f0b-43ab-40bf-ae39-5dc6e195ce39\") " pod="openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr" Nov 01 20:54:37.635091 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.635053 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:37.797028 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.796762 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:37.880578 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.880535 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr" Nov 01 20:54:37.986773 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:37.986699 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:38.301785 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:38.301740 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:38.462293 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:54:38.462255 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce125f0b_43ab_40bf_ae39_5dc6e195ce39.slice/crio-a8814ff4889808a6ff123242c98a49b1a8f37fda57358a2c60f4d2024887ce5d WatchSource:0}: Error finding container a8814ff4889808a6ff123242c98a49b1a8f37fda57358a2c60f4d2024887ce5d: Status 404 returned error can't find the container with id a8814ff4889808a6ff123242c98a49b1a8f37fda57358a2c60f4d2024887ce5d Nov 01 20:54:38.464182 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:38.464155 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr"] Nov 01 20:54:38.464939 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:38.464906 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:54:38.464987 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:38.464971 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:54:38.465063 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:38.465016 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:54:38.515494 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:38.515451 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d0d57533-mxw8f"] Nov 01 20:54:38.521389 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:38.521358 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d0d57533-mxw8f"] Nov 01 20:54:38.624388 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:38.624347 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:38.876543 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:38.873833 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:39.328270 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:39.328220 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr" event={"ID":"ce125f0b-43ab-40bf-ae39-5dc6e195ce39","Type":"ContainerStarted","Data":"4207d5f3773e1840c07c8df01b337d98418cd7d3b1813cc705a5e1673dfa7760"} Nov 01 20:54:39.328863 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:39.328837 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr" event={"ID":"ce125f0b-43ab-40bf-ae39-5dc6e195ce39","Type":"ContainerStarted","Data":"a8814ff4889808a6ff123242c98a49b1a8f37fda57358a2c60f4d2024887ce5d"} Nov 01 20:54:39.486017 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:39.485374 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d89f74fe-7a5b-411d-b82d-d77ff26b92f3" path="/var/lib/kubelet/pods/d89f74fe-7a5b-411d-b82d-d77ff26b92f3/volumes" Nov 01 20:54:39.486652 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:39.486089 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd3626f4-8f4c-4466-beb5-b50487013481" path="/var/lib/kubelet/pods/fd3626f4-8f4c-4466-beb5-b50487013481/volumes" Nov 01 20:54:39.497298 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:39.497265 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:39.573669 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:39.573616 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr" podStartSLOduration=2.573602974 podStartE2EDuration="2.573602974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:54:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:54:39.521364947 +0000 UTC m=+1430.299005084" watchObservedRunningTime="2026-11-01 20:54:39.573602974 +0000 UTC m=+1430.351243109" Nov 01 20:54:39.809021 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:39.804233 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:39.888050 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:39.885289 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr" Nov 01 20:54:39.929023 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:39.928311 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr" Nov 01 20:54:40.059019 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:40.058174 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0393ace-lv62v"] Nov 01 20:54:40.091751 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:40.089860 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0393ace-lv62v"] Nov 01 20:54:40.190336 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:40.190292 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg"] Nov 01 20:54:40.195380 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:40.195356 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg" Nov 01 20:54:40.555401 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:40.548107 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8137e500-e107-49d5-81ba-948c7ae986f1-workload-socket\") pod \"e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg\" (UID: \"8137e500-e107-49d5-81ba-948c7ae986f1\") " pod="openshift-ingress/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg" Nov 01 20:54:40.555401 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:40.548211 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8137e500-e107-49d5-81ba-948c7ae986f1-istio-envoy\") pod \"e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg\" (UID: \"8137e500-e107-49d5-81ba-948c7ae986f1\") " pod="openshift-ingress/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg" Nov 01 20:54:40.555401 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:40.548482 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8137e500-e107-49d5-81ba-948c7ae986f1-istiod-ca-cert\") pod \"e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg\" (UID: \"8137e500-e107-49d5-81ba-948c7ae986f1\") " pod="openshift-ingress/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg" Nov 01 20:54:40.555401 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:40.548532 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8137e500-e107-49d5-81ba-948c7ae986f1-credential-socket\") pod \"e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg\" (UID: \"8137e500-e107-49d5-81ba-948c7ae986f1\") " pod="openshift-ingress/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg" Nov 01 20:54:40.555401 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:40.548617 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8137e500-e107-49d5-81ba-948c7ae986f1-istio-data\") pod \"e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg\" (UID: \"8137e500-e107-49d5-81ba-948c7ae986f1\") " pod="openshift-ingress/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg" Nov 01 20:54:40.555401 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:40.548647 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8137e500-e107-49d5-81ba-948c7ae986f1-istio-podinfo\") pod \"e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg\" (UID: \"8137e500-e107-49d5-81ba-948c7ae986f1\") " pod="openshift-ingress/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg" Nov 01 20:54:40.555401 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:40.548689 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8137e500-e107-49d5-81ba-948c7ae986f1-istio-token\") pod \"e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg\" (UID: \"8137e500-e107-49d5-81ba-948c7ae986f1\") " pod="openshift-ingress/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg" Nov 01 20:54:40.555401 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:40.548721 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppj26\" (UniqueName: \"kubernetes.io/projected/8137e500-e107-49d5-81ba-948c7ae986f1-kube-api-access-ppj26\") pod \"e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg\" (UID: \"8137e500-e107-49d5-81ba-948c7ae986f1\") " pod="openshift-ingress/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg" Nov 01 20:54:40.555401 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:40.548770 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8137e500-e107-49d5-81ba-948c7ae986f1-workload-certs\") pod \"e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg\" (UID: \"8137e500-e107-49d5-81ba-948c7ae986f1\") " pod="openshift-ingress/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg" Nov 01 20:54:40.601338 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:40.600861 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-74ab02fe-openshift-default-dockercfg-8g2tk\"" Nov 01 20:54:40.601975 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:40.601761 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:40.641951 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:40.641923 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr" Nov 01 20:54:40.654120 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:40.650329 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8137e500-e107-49d5-81ba-948c7ae986f1-istiod-ca-cert\") pod \"e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg\" (UID: \"8137e500-e107-49d5-81ba-948c7ae986f1\") " pod="openshift-ingress/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg" Nov 01 20:54:40.654120 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:40.650407 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8137e500-e107-49d5-81ba-948c7ae986f1-credential-socket\") pod \"e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg\" (UID: \"8137e500-e107-49d5-81ba-948c7ae986f1\") " pod="openshift-ingress/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg" Nov 01 20:54:40.654120 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:40.650529 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8137e500-e107-49d5-81ba-948c7ae986f1-istio-data\") pod \"e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg\" (UID: \"8137e500-e107-49d5-81ba-948c7ae986f1\") " pod="openshift-ingress/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg" Nov 01 20:54:40.654120 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:40.650566 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8137e500-e107-49d5-81ba-948c7ae986f1-istio-podinfo\") pod \"e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg\" (UID: \"8137e500-e107-49d5-81ba-948c7ae986f1\") " pod="openshift-ingress/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg" Nov 01 20:54:40.654120 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:40.650639 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8137e500-e107-49d5-81ba-948c7ae986f1-istio-token\") pod \"e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg\" (UID: \"8137e500-e107-49d5-81ba-948c7ae986f1\") " pod="openshift-ingress/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg" Nov 01 20:54:40.654120 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:40.650674 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppj26\" (UniqueName: \"kubernetes.io/projected/8137e500-e107-49d5-81ba-948c7ae986f1-kube-api-access-ppj26\") pod \"e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg\" (UID: \"8137e500-e107-49d5-81ba-948c7ae986f1\") " pod="openshift-ingress/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg" Nov 01 20:54:40.654120 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:40.650741 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8137e500-e107-49d5-81ba-948c7ae986f1-workload-certs\") pod \"e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg\" (UID: \"8137e500-e107-49d5-81ba-948c7ae986f1\") " pod="openshift-ingress/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg" Nov 01 20:54:40.654120 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:40.650800 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8137e500-e107-49d5-81ba-948c7ae986f1-workload-socket\") pod \"e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg\" (UID: \"8137e500-e107-49d5-81ba-948c7ae986f1\") " pod="openshift-ingress/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg" Nov 01 20:54:40.655722 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:40.654883 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8137e500-e107-49d5-81ba-948c7ae986f1-istio-envoy\") pod \"e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg\" (UID: \"8137e500-e107-49d5-81ba-948c7ae986f1\") " pod="openshift-ingress/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg" Nov 01 20:54:40.657392 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:40.657366 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8137e500-e107-49d5-81ba-948c7ae986f1-istio-data\") pod \"e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg\" (UID: \"8137e500-e107-49d5-81ba-948c7ae986f1\") " pod="openshift-ingress/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg" Nov 01 20:54:40.657675 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:40.657655 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr" Nov 01 20:54:40.659782 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:40.659758 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8137e500-e107-49d5-81ba-948c7ae986f1-istiod-ca-cert\") pod \"e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg\" (UID: \"8137e500-e107-49d5-81ba-948c7ae986f1\") " pod="openshift-ingress/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg" Nov 01 20:54:40.732728 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:40.661711 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8137e500-e107-49d5-81ba-948c7ae986f1-workload-certs\") pod \"e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg\" (UID: \"8137e500-e107-49d5-81ba-948c7ae986f1\") " pod="openshift-ingress/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg" Nov 01 20:54:40.732963 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:40.669200 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg"] Nov 01 20:54:40.733136 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:40.685456 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8137e500-e107-49d5-81ba-948c7ae986f1-credential-socket\") pod \"e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg\" (UID: \"8137e500-e107-49d5-81ba-948c7ae986f1\") " pod="openshift-ingress/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg" Nov 01 20:54:40.733243 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:40.686798 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8137e500-e107-49d5-81ba-948c7ae986f1-workload-socket\") pod \"e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg\" (UID: \"8137e500-e107-49d5-81ba-948c7ae986f1\") " pod="openshift-ingress/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg" Nov 01 20:54:40.753953 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:40.753878 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8137e500-e107-49d5-81ba-948c7ae986f1-istio-envoy\") pod \"e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg\" (UID: \"8137e500-e107-49d5-81ba-948c7ae986f1\") " pod="openshift-ingress/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg" Nov 01 20:54:40.769970 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:40.769938 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8137e500-e107-49d5-81ba-948c7ae986f1-istio-token\") pod \"e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg\" (UID: \"8137e500-e107-49d5-81ba-948c7ae986f1\") " pod="openshift-ingress/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg" Nov 01 20:54:40.770764 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:40.770717 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8137e500-e107-49d5-81ba-948c7ae986f1-istio-podinfo\") pod \"e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg\" (UID: \"8137e500-e107-49d5-81ba-948c7ae986f1\") " pod="openshift-ingress/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg" Nov 01 20:54:40.775456 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:40.774671 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:40.775672 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:40.775261 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppj26\" (UniqueName: \"kubernetes.io/projected/8137e500-e107-49d5-81ba-948c7ae986f1-kube-api-access-ppj26\") pod \"e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg\" (UID: \"8137e500-e107-49d5-81ba-948c7ae986f1\") " pod="openshift-ingress/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg" Nov 01 20:54:40.853121 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:40.847107 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg" Nov 01 20:54:40.945357 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:40.934026 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:41.195349 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.195324 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv" Nov 01 20:54:41.276371 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.276333 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-istio-podinfo\") pod \"dff8b3d7-cade-4cba-9e07-3e912f0cabc8\" (UID: \"dff8b3d7-cade-4cba-9e07-3e912f0cabc8\") " Nov 01 20:54:41.276582 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.276377 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-istiod-ca-cert\") pod \"dff8b3d7-cade-4cba-9e07-3e912f0cabc8\" (UID: \"dff8b3d7-cade-4cba-9e07-3e912f0cabc8\") " Nov 01 20:54:41.276582 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.276458 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-istio-token\" (UniqueName: \"kubernetes.io/projected/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-istio-token\") pod \"dff8b3d7-cade-4cba-9e07-3e912f0cabc8\" (UID: \"dff8b3d7-cade-4cba-9e07-3e912f0cabc8\") " Nov 01 20:54:41.276582 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.276524 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-kube-api-access-s5v2b\" (UniqueName: \"kubernetes.io/projected/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-kube-api-access-s5v2b\") pod \"dff8b3d7-cade-4cba-9e07-3e912f0cabc8\" (UID: \"dff8b3d7-cade-4cba-9e07-3e912f0cabc8\") " Nov 01 20:54:41.276582 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.276574 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-workload-certs\") pod \"dff8b3d7-cade-4cba-9e07-3e912f0cabc8\" (UID: \"dff8b3d7-cade-4cba-9e07-3e912f0cabc8\") " Nov 01 20:54:41.276795 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.276626 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-credential-socket\") pod \"dff8b3d7-cade-4cba-9e07-3e912f0cabc8\" (UID: \"dff8b3d7-cade-4cba-9e07-3e912f0cabc8\") " Nov 01 20:54:41.276795 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.276695 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-workload-socket\") pod \"dff8b3d7-cade-4cba-9e07-3e912f0cabc8\" (UID: \"dff8b3d7-cade-4cba-9e07-3e912f0cabc8\") " Nov 01 20:54:41.276795 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.276765 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-istio-envoy\") pod \"dff8b3d7-cade-4cba-9e07-3e912f0cabc8\" (UID: \"dff8b3d7-cade-4cba-9e07-3e912f0cabc8\") " Nov 01 20:54:41.276950 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.276851 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-istio-data\") pod \"dff8b3d7-cade-4cba-9e07-3e912f0cabc8\" (UID: \"dff8b3d7-cade-4cba-9e07-3e912f0cabc8\") " Nov 01 20:54:41.277017 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.276967 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-credential-socket" pod "dff8b3d7-cade-4cba-9e07-3e912f0cabc8" (UID: "dff8b3d7-cade-4cba-9e07-3e912f0cabc8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:54:41.277075 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.276978 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-workload-certs" pod "dff8b3d7-cade-4cba-9e07-3e912f0cabc8" (UID: "dff8b3d7-cade-4cba-9e07-3e912f0cabc8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:54:41.277225 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.277204 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-workload-socket" pod "dff8b3d7-cade-4cba-9e07-3e912f0cabc8" (UID: "dff8b3d7-cade-4cba-9e07-3e912f0cabc8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:54:41.277281 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.277213 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-istiod-ca-cert" pod "dff8b3d7-cade-4cba-9e07-3e912f0cabc8" (UID: "dff8b3d7-cade-4cba-9e07-3e912f0cabc8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:54:41.277428 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.277410 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-istio-data" pod "dff8b3d7-cade-4cba-9e07-3e912f0cabc8" (UID: "dff8b3d7-cade-4cba-9e07-3e912f0cabc8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:54:41.277601 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.277582 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:54:41.277601 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.277601 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:54:41.277817 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.277612 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:54:41.277817 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.277621 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:54:41.277817 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.277630 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:54:41.278920 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.278894 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-istio-podinfo" pod "dff8b3d7-cade-4cba-9e07-3e912f0cabc8" (UID: "dff8b3d7-cade-4cba-9e07-3e912f0cabc8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:54:41.279358 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.279337 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-kube-api-access-s5v2b" pod "dff8b3d7-cade-4cba-9e07-3e912f0cabc8" (UID: "dff8b3d7-cade-4cba-9e07-3e912f0cabc8"). InnerVolumeSpecName "kube-api-access-s5v2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:54:41.279441 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.279418 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-istio-envoy" pod "dff8b3d7-cade-4cba-9e07-3e912f0cabc8" (UID: "dff8b3d7-cade-4cba-9e07-3e912f0cabc8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:54:41.279490 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.279440 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-istio-token" pod "dff8b3d7-cade-4cba-9e07-3e912f0cabc8" (UID: "dff8b3d7-cade-4cba-9e07-3e912f0cabc8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:54:41.288133 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.288109 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7b8v8"] Nov 01 20:54:41.288831 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.288815 4484 cpu_manager.go:414] "containerMap: removing container" podUID="dff8b3d7-cade-4cba-9e07-3e912f0cabc8" containerName="istio-proxy" Nov 01 20:54:41.288888 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.288833 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dff8b3d7-cade-4cba-9e07-3e912f0cabc8" containerName="istio-proxy" Nov 01 20:54:41.289003 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.288987 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="dff8b3d7-cade-4cba-9e07-3e912f0cabc8" containerName="istio-proxy" Nov 01 20:54:41.293238 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.293223 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7b8v8" Nov 01 20:54:41.377374 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.377337 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d235567c-ae00-4a30-ac29-3ef4cc1e4bbb" path="/var/lib/kubelet/pods/d235567c-ae00-4a30-ac29-3ef4cc1e4bbb/volumes" Nov 01 20:54:41.378771 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.378740 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e23f5f45-a08b-4a20-b61e-d33119765592-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-7b8v8\" (UID: \"e23f5f45-a08b-4a20-b61e-d33119765592\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7b8v8" Nov 01 20:54:41.379026 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.378990 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e23f5f45-a08b-4a20-b61e-d33119765592-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-7b8v8\" (UID: \"e23f5f45-a08b-4a20-b61e-d33119765592\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7b8v8" Nov 01 20:54:41.379301 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.379264 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e23f5f45-a08b-4a20-b61e-d33119765592-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-7b8v8\" (UID: \"e23f5f45-a08b-4a20-b61e-d33119765592\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7b8v8" Nov 01 20:54:41.379659 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.379616 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e23f5f45-a08b-4a20-b61e-d33119765592-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-7b8v8\" (UID: \"e23f5f45-a08b-4a20-b61e-d33119765592\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7b8v8" Nov 01 20:54:41.379777 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.379698 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e23f5f45-a08b-4a20-b61e-d33119765592-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-7b8v8\" (UID: \"e23f5f45-a08b-4a20-b61e-d33119765592\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7b8v8" Nov 01 20:54:41.379777 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.379727 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e23f5f45-a08b-4a20-b61e-d33119765592-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-7b8v8\" (UID: \"e23f5f45-a08b-4a20-b61e-d33119765592\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7b8v8" Nov 01 20:54:41.379777 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.379757 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e23f5f45-a08b-4a20-b61e-d33119765592-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-7b8v8\" (UID: \"e23f5f45-a08b-4a20-b61e-d33119765592\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7b8v8" Nov 01 20:54:41.380085 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.380057 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9kqj\" (UniqueName: \"kubernetes.io/projected/e23f5f45-a08b-4a20-b61e-d33119765592-kube-api-access-c9kqj\") pod \"maas-default-gateway-openshift-default-58c6585447-7b8v8\" (UID: \"e23f5f45-a08b-4a20-b61e-d33119765592\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7b8v8" Nov 01 20:54:41.380165 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.380098 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e23f5f45-a08b-4a20-b61e-d33119765592-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-7b8v8\" (UID: \"e23f5f45-a08b-4a20-b61e-d33119765592\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7b8v8" Nov 01 20:54:41.380217 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.380202 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:54:41.380266 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.380219 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5v2b\" (UniqueName: \"kubernetes.io/projected/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-kube-api-access-s5v2b\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:54:41.380266 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.380235 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:54:41.380266 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.380249 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dff8b3d7-cade-4cba-9e07-3e912f0cabc8-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:54:41.389976 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.389951 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7b8v8"] Nov 01 20:54:41.450375 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:54:41.450345 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8137e500_e107_49d5_81ba_948c7ae986f1.slice/crio-fcbba7613da663d22034a2e494d85e47714b5884583ceee80803773ecc8f95d6 WatchSource:0}: Error finding container fcbba7613da663d22034a2e494d85e47714b5884583ceee80803773ecc8f95d6: Status 404 returned error can't find the container with id fcbba7613da663d22034a2e494d85e47714b5884583ceee80803773ecc8f95d6 Nov 01 20:54:41.452053 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.452024 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg"] Nov 01 20:54:41.453387 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.453344 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:54:41.453495 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.453449 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:54:41.453555 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.453496 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:54:41.481795 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.481765 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e23f5f45-a08b-4a20-b61e-d33119765592-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-7b8v8\" (UID: \"e23f5f45-a08b-4a20-b61e-d33119765592\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7b8v8" Nov 01 20:54:41.481923 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.481852 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e23f5f45-a08b-4a20-b61e-d33119765592-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-7b8v8\" (UID: \"e23f5f45-a08b-4a20-b61e-d33119765592\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7b8v8" Nov 01 20:54:41.481923 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.481877 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e23f5f45-a08b-4a20-b61e-d33119765592-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-7b8v8\" (UID: \"e23f5f45-a08b-4a20-b61e-d33119765592\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7b8v8" Nov 01 20:54:41.481923 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.481907 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e23f5f45-a08b-4a20-b61e-d33119765592-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-7b8v8\" (UID: \"e23f5f45-a08b-4a20-b61e-d33119765592\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7b8v8" Nov 01 20:54:41.482089 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.482015 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e23f5f45-a08b-4a20-b61e-d33119765592-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-7b8v8\" (UID: \"e23f5f45-a08b-4a20-b61e-d33119765592\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7b8v8" Nov 01 20:54:41.482089 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.482080 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e23f5f45-a08b-4a20-b61e-d33119765592-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-7b8v8\" (UID: \"e23f5f45-a08b-4a20-b61e-d33119765592\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7b8v8" Nov 01 20:54:41.482197 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.482107 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e23f5f45-a08b-4a20-b61e-d33119765592-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-7b8v8\" (UID: \"e23f5f45-a08b-4a20-b61e-d33119765592\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7b8v8" Nov 01 20:54:41.482197 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.482141 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e23f5f45-a08b-4a20-b61e-d33119765592-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-7b8v8\" (UID: \"e23f5f45-a08b-4a20-b61e-d33119765592\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7b8v8" Nov 01 20:54:41.482361 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.482330 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9kqj\" (UniqueName: \"kubernetes.io/projected/e23f5f45-a08b-4a20-b61e-d33119765592-kube-api-access-c9kqj\") pod \"maas-default-gateway-openshift-default-58c6585447-7b8v8\" (UID: \"e23f5f45-a08b-4a20-b61e-d33119765592\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7b8v8" Nov 01 20:54:41.482477 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.482451 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e23f5f45-a08b-4a20-b61e-d33119765592-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-7b8v8\" (UID: \"e23f5f45-a08b-4a20-b61e-d33119765592\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7b8v8" Nov 01 20:54:41.482477 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.482471 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e23f5f45-a08b-4a20-b61e-d33119765592-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-7b8v8\" (UID: \"e23f5f45-a08b-4a20-b61e-d33119765592\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7b8v8" Nov 01 20:54:41.482615 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.482343 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e23f5f45-a08b-4a20-b61e-d33119765592-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-7b8v8\" (UID: \"e23f5f45-a08b-4a20-b61e-d33119765592\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7b8v8" Nov 01 20:54:41.482713 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.482696 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e23f5f45-a08b-4a20-b61e-d33119765592-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-7b8v8\" (UID: \"e23f5f45-a08b-4a20-b61e-d33119765592\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7b8v8" Nov 01 20:54:41.482918 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.482900 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e23f5f45-a08b-4a20-b61e-d33119765592-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-7b8v8\" (UID: \"e23f5f45-a08b-4a20-b61e-d33119765592\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7b8v8" Nov 01 20:54:41.484684 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.484666 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e23f5f45-a08b-4a20-b61e-d33119765592-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-7b8v8\" (UID: \"e23f5f45-a08b-4a20-b61e-d33119765592\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7b8v8" Nov 01 20:54:41.484939 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.484920 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e23f5f45-a08b-4a20-b61e-d33119765592-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-7b8v8\" (UID: \"e23f5f45-a08b-4a20-b61e-d33119765592\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7b8v8" Nov 01 20:54:41.539555 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.539523 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e23f5f45-a08b-4a20-b61e-d33119765592-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-7b8v8\" (UID: \"e23f5f45-a08b-4a20-b61e-d33119765592\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7b8v8" Nov 01 20:54:41.539681 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.539611 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9kqj\" (UniqueName: \"kubernetes.io/projected/e23f5f45-a08b-4a20-b61e-d33119765592-kube-api-access-c9kqj\") pod \"maas-default-gateway-openshift-default-58c6585447-7b8v8\" (UID: \"e23f5f45-a08b-4a20-b61e-d33119765592\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7b8v8" Nov 01 20:54:41.609873 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.609834 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7b8v8" Nov 01 20:54:41.638108 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.638074 4484 generic.go:356] "Generic (PLEG): container finished" podID="dff8b3d7-cade-4cba-9e07-3e912f0cabc8" containerID="578a8ddb4623e84b75b7c4fb4aa39ed74914603c2f27dff3ccc1556c9a09d137" exitCode=0 Nov 01 20:54:41.638292 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.638165 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv" Nov 01 20:54:41.638292 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.638160 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv" event={"ID":"dff8b3d7-cade-4cba-9e07-3e912f0cabc8","Type":"ContainerDied","Data":"578a8ddb4623e84b75b7c4fb4aa39ed74914603c2f27dff3ccc1556c9a09d137"} Nov 01 20:54:41.638411 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.638336 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv" event={"ID":"dff8b3d7-cade-4cba-9e07-3e912f0cabc8","Type":"ContainerDied","Data":"5233362a6e2f749f2302627d73be858707c9334139689fb2dc75ce99d072c8fe"} Nov 01 20:54:41.638411 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.638382 4484 scope.go:122] "RemoveContainer" containerID="578a8ddb4623e84b75b7c4fb4aa39ed74914603c2f27dff3ccc1556c9a09d137" Nov 01 20:54:41.642179 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.642154 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg" event={"ID":"8137e500-e107-49d5-81ba-948c7ae986f1","Type":"ContainerStarted","Data":"c76773bf6f98025744a313005c3c15dc5dc2e8f3e252a02daf73199676275fb0"} Nov 01 20:54:41.642308 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.642203 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg" event={"ID":"8137e500-e107-49d5-81ba-948c7ae986f1","Type":"ContainerStarted","Data":"fcbba7613da663d22034a2e494d85e47714b5884583ceee80803773ecc8f95d6"} Nov 01 20:54:41.654207 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.654186 4484 scope.go:122] "RemoveContainer" containerID="2a690d6c47f89babe853ff1cbb0cba0cae8655a7b71cf5f7e282e35e76f4e95d" Nov 01 20:54:41.671942 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.666967 4484 scope.go:122] "RemoveContainer" containerID="578a8ddb4623e84b75b7c4fb4aa39ed74914603c2f27dff3ccc1556c9a09d137" Nov 01 20:54:41.672365 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:54:41.672339 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"578a8ddb4623e84b75b7c4fb4aa39ed74914603c2f27dff3ccc1556c9a09d137\": container with ID starting with 578a8ddb4623e84b75b7c4fb4aa39ed74914603c2f27dff3ccc1556c9a09d137 not found: ID does not exist" containerID="578a8ddb4623e84b75b7c4fb4aa39ed74914603c2f27dff3ccc1556c9a09d137" Nov 01 20:54:41.672442 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.672373 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"578a8ddb4623e84b75b7c4fb4aa39ed74914603c2f27dff3ccc1556c9a09d137"} err="failed to get container status \"578a8ddb4623e84b75b7c4fb4aa39ed74914603c2f27dff3ccc1556c9a09d137\": rpc error: code = NotFound desc = could not find container \"578a8ddb4623e84b75b7c4fb4aa39ed74914603c2f27dff3ccc1556c9a09d137\": container with ID starting with 578a8ddb4623e84b75b7c4fb4aa39ed74914603c2f27dff3ccc1556c9a09d137 not found: ID does not exist" Nov 01 20:54:41.672442 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.672392 4484 scope.go:122] "RemoveContainer" containerID="2a690d6c47f89babe853ff1cbb0cba0cae8655a7b71cf5f7e282e35e76f4e95d" Nov 01 20:54:41.672758 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:54:41.672736 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a690d6c47f89babe853ff1cbb0cba0cae8655a7b71cf5f7e282e35e76f4e95d\": container with ID starting with 2a690d6c47f89babe853ff1cbb0cba0cae8655a7b71cf5f7e282e35e76f4e95d not found: ID does not exist" containerID="2a690d6c47f89babe853ff1cbb0cba0cae8655a7b71cf5f7e282e35e76f4e95d" Nov 01 20:54:41.672814 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.672770 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a690d6c47f89babe853ff1cbb0cba0cae8655a7b71cf5f7e282e35e76f4e95d"} err="failed to get container status \"2a690d6c47f89babe853ff1cbb0cba0cae8655a7b71cf5f7e282e35e76f4e95d\": rpc error: code = NotFound desc = could not find container \"2a690d6c47f89babe853ff1cbb0cba0cae8655a7b71cf5f7e282e35e76f4e95d\": container with ID starting with 2a690d6c47f89babe853ff1cbb0cba0cae8655a7b71cf5f7e282e35e76f4e95d not found: ID does not exist" Nov 01 20:54:41.808921 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.808882 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv"] Nov 01 20:54:41.814193 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.814164 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-f955d283-openshift-default-775bfb5f57-x79dv"] Nov 01 20:54:41.847440 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.847410 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg" Nov 01 20:54:41.929829 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.929773 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg" podStartSLOduration=1.929757916 podStartE2EDuration="1.929757916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:54:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:54:41.925193124 +0000 UTC m=+1432.702833262" watchObservedRunningTime="2026-11-01 20:54:41.929757916 +0000 UTC m=+1432.707398051" Nov 01 20:54:41.983338 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:41.983302 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:42.075021 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:42.065278 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7b8v8"] Nov 01 20:54:42.135036 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:42.132429 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:54:42.135036 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:42.132537 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:54:42.135036 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:42.132579 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:54:42.199963 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:42.199688 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:42.653352 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:42.653299 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7b8v8" event={"ID":"e23f5f45-a08b-4a20-b61e-d33119765592","Type":"ContainerStarted","Data":"a76d9d911356c21a33bd508e2d2212fbd9d35990e35e1a3bc496ef66dbef17ed"} Nov 01 20:54:42.653859 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:42.653383 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7b8v8" event={"ID":"e23f5f45-a08b-4a20-b61e-d33119765592","Type":"ContainerStarted","Data":"cbac34cee4a4a8f9275f2f0398e3d862bca4ffde59ecd683e865237f0e287582"} Nov 01 20:54:42.778578 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:42.778512 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7b8v8" podStartSLOduration=1.778497413 podStartE2EDuration="1.778497413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:54:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:54:42.772497757 +0000 UTC m=+1433.550137905" watchObservedRunningTime="2026-11-01 20:54:42.778497413 +0000 UTC m=+1433.556137633" Nov 01 20:54:42.853289 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:42.853259 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg" Nov 01 20:54:43.100073 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:43.100039 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:43.374428 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:43.374350 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dff8b3d7-cade-4cba-9e07-3e912f0cabc8" path="/var/lib/kubelet/pods/dff8b3d7-cade-4cba-9e07-3e912f0cabc8/volumes" Nov 01 20:54:43.625032 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:43.624922 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7b8v8" Nov 01 20:54:43.628174 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:43.627164 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-7b8v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 01 20:54:43.628174 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:43.627226 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7b8v8" podUID="e23f5f45-a08b-4a20-b61e-d33119765592" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 01 20:54:43.661881 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:43.661841 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg" Nov 01 20:54:43.663059 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:43.663032 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg" Nov 01 20:54:44.531346 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:44.530983 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:44.623020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:44.621480 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-7b8v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 01 20:54:44.623020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:44.621542 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7b8v8" podUID="e23f5f45-a08b-4a20-b61e-d33119765592" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 01 20:54:44.905093 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:44.900863 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:45.709020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:45.677258 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:45.709020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:45.678762 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" podUID="a65d9fd8-6193-4d03-9a3c-123f48b5dcdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Nov 01 20:54:45.709020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:45.688892 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" podUID="9f19be29-2c84-4ae4-9674-acca4905f4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 01 20:54:45.738023 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:45.712623 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" podUID="70b4dacf-b3ac-46ec-99e8-a44f05587e7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:54:45.776359 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:45.747574 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-7b8v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 01 20:54:45.776359 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:45.772163 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7b8v8" podUID="e23f5f45-a08b-4a20-b61e-d33119765592" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 01 20:54:45.776359 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:45.763845 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8ddfb06d-66db94bbb7-2fxht"] Nov 01 20:54:45.776359 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:45.773297 4484 cpu_manager.go:414] "containerMap: removing container" podUID="dff8b3d7-cade-4cba-9e07-3e912f0cabc8" containerName="istio-proxy" Nov 01 20:54:45.776359 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:45.773316 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dff8b3d7-cade-4cba-9e07-3e912f0cabc8" containerName="istio-proxy" Nov 01 20:54:45.776359 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:45.773505 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="dff8b3d7-cade-4cba-9e07-3e912f0cabc8" containerName="istio-proxy" Nov 01 20:54:45.782589 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:45.782549 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8ddfb06d-66db94bbb7-2fxht" Nov 01 20:54:45.863018 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:45.855105 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-8ddfb06d\"" Nov 01 20:54:45.863018 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:45.858843 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15651aae-6296-488c-b5b3-fb525dbb5a68-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-8ddfb06d-66db94bbb7-2fxht\" (UID: \"15651aae-6296-488c-b5b3-fb525dbb5a68\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8ddfb06d-66db94bbb7-2fxht" Nov 01 20:54:45.863018 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:45.858897 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwzxg\" (UniqueName: \"kubernetes.io/projected/15651aae-6296-488c-b5b3-fb525dbb5a68-kube-api-access-rwzxg\") pod \"payload-pre-processing-e2e-migrate-8ddfb06d-66db94bbb7-2fxht\" (UID: \"15651aae-6296-488c-b5b3-fb525dbb5a68\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8ddfb06d-66db94bbb7-2fxht" Nov 01 20:54:45.863018 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:45.859410 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8ddfb06d-66db94bbb7-2fxht"] Nov 01 20:54:45.863791 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:45.863762 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:45.865566 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:45.865548 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8ddfb06d-54d7b96ff7-flgn4"] Nov 01 20:54:45.872333 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:45.872316 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8ddfb06d-54d7b96ff7-flgn4"] Nov 01 20:54:45.872569 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:45.872555 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-8ddfb06d-54d7b96ff7-flgn4" Nov 01 20:54:45.890301 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:54:45.890273 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 20:54:45.960243 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:45.960147 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5vss\" (UniqueName: \"kubernetes.io/projected/497bee83-44f6-4eee-88c9-e62f051c8a28-kube-api-access-t5vss\") pod \"payload-processing-e2e-migrate-8ddfb06d-54d7b96ff7-flgn4\" (UID: \"497bee83-44f6-4eee-88c9-e62f051c8a28\") " pod="openshift-ingress/payload-processing-e2e-migrate-8ddfb06d-54d7b96ff7-flgn4" Nov 01 20:54:45.960243 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:45.960217 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/497bee83-44f6-4eee-88c9-e62f051c8a28-plugins-config-volume\") pod \"payload-processing-e2e-migrate-8ddfb06d-54d7b96ff7-flgn4\" (UID: \"497bee83-44f6-4eee-88c9-e62f051c8a28\") " pod="openshift-ingress/payload-processing-e2e-migrate-8ddfb06d-54d7b96ff7-flgn4" Nov 01 20:54:45.960772 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:45.960343 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15651aae-6296-488c-b5b3-fb525dbb5a68-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-8ddfb06d-66db94bbb7-2fxht\" (UID: \"15651aae-6296-488c-b5b3-fb525dbb5a68\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8ddfb06d-66db94bbb7-2fxht" Nov 01 20:54:45.960772 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:45.960381 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwzxg\" (UniqueName: \"kubernetes.io/projected/15651aae-6296-488c-b5b3-fb525dbb5a68-kube-api-access-rwzxg\") pod \"payload-pre-processing-e2e-migrate-8ddfb06d-66db94bbb7-2fxht\" (UID: \"15651aae-6296-488c-b5b3-fb525dbb5a68\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8ddfb06d-66db94bbb7-2fxht" Nov 01 20:54:45.961243 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:45.961222 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15651aae-6296-488c-b5b3-fb525dbb5a68-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-8ddfb06d-66db94bbb7-2fxht\" (UID: \"15651aae-6296-488c-b5b3-fb525dbb5a68\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8ddfb06d-66db94bbb7-2fxht" Nov 01 20:54:45.996084 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:45.996044 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-8ddfb06d-dockercfg-2b8ww\"" Nov 01 20:54:46.028834 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:46.028799 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwzxg\" (UniqueName: \"kubernetes.io/projected/15651aae-6296-488c-b5b3-fb525dbb5a68-kube-api-access-rwzxg\") pod \"payload-pre-processing-e2e-migrate-8ddfb06d-66db94bbb7-2fxht\" (UID: \"15651aae-6296-488c-b5b3-fb525dbb5a68\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8ddfb06d-66db94bbb7-2fxht" Nov 01 20:54:46.061783 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:46.061746 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5vss\" (UniqueName: \"kubernetes.io/projected/497bee83-44f6-4eee-88c9-e62f051c8a28-kube-api-access-t5vss\") pod \"payload-processing-e2e-migrate-8ddfb06d-54d7b96ff7-flgn4\" (UID: \"497bee83-44f6-4eee-88c9-e62f051c8a28\") " pod="openshift-ingress/payload-processing-e2e-migrate-8ddfb06d-54d7b96ff7-flgn4" Nov 01 20:54:46.061970 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:46.061834 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/497bee83-44f6-4eee-88c9-e62f051c8a28-plugins-config-volume\") pod \"payload-processing-e2e-migrate-8ddfb06d-54d7b96ff7-flgn4\" (UID: \"497bee83-44f6-4eee-88c9-e62f051c8a28\") " pod="openshift-ingress/payload-processing-e2e-migrate-8ddfb06d-54d7b96ff7-flgn4" Nov 01 20:54:46.062586 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:46.062560 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/497bee83-44f6-4eee-88c9-e62f051c8a28-plugins-config-volume\") pod \"payload-processing-e2e-migrate-8ddfb06d-54d7b96ff7-flgn4\" (UID: \"497bee83-44f6-4eee-88c9-e62f051c8a28\") " pod="openshift-ingress/payload-processing-e2e-migrate-8ddfb06d-54d7b96ff7-flgn4" Nov 01 20:54:46.129790 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:46.129758 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5vss\" (UniqueName: \"kubernetes.io/projected/497bee83-44f6-4eee-88c9-e62f051c8a28-kube-api-access-t5vss\") pod \"payload-processing-e2e-migrate-8ddfb06d-54d7b96ff7-flgn4\" (UID: \"497bee83-44f6-4eee-88c9-e62f051c8a28\") " pod="openshift-ingress/payload-processing-e2e-migrate-8ddfb06d-54d7b96ff7-flgn4" Nov 01 20:54:46.181617 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:46.181579 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8ddfb06d-66db94bbb7-2fxht" Nov 01 20:54:46.193210 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:46.193184 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-8ddfb06d-54d7b96ff7-flgn4" Nov 01 20:54:46.628202 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:46.628158 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-7b8v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 01 20:54:46.628426 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:46.628224 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7b8v8" podUID="e23f5f45-a08b-4a20-b61e-d33119765592" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 01 20:54:46.701974 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:54:46.701943 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod497bee83_44f6_4eee_88c9_e62f051c8a28.slice/crio-21edefba1902b8d8bbfcc655ef403635ee9e3a7ab1ef41ddfe27ae4e06f7be1f WatchSource:0}: Error finding container 21edefba1902b8d8bbfcc655ef403635ee9e3a7ab1ef41ddfe27ae4e06f7be1f: Status 404 returned error can't find the container with id 21edefba1902b8d8bbfcc655ef403635ee9e3a7ab1ef41ddfe27ae4e06f7be1f Nov 01 20:54:46.729036 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:46.703348 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8ddfb06d-54d7b96ff7-flgn4"] Nov 01 20:54:46.738436 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:46.738406 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8ddfb06d-66db94bbb7-2fxht"] Nov 01 20:54:46.751617 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:54:46.751574 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15651aae_6296_488c_b5b3_fb525dbb5a68.slice/crio-0306a97af5a97990311c95f0dc3ff2047c296fea076e0614884093c9a62cd515 WatchSource:0}: Error finding container 0306a97af5a97990311c95f0dc3ff2047c296fea076e0614884093c9a62cd515: Status 404 returned error can't find the container with id 0306a97af5a97990311c95f0dc3ff2047c296fea076e0614884093c9a62cd515 Nov 01 20:54:47.625573 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:47.625114 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-7b8v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 01 20:54:47.625573 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:47.625181 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7b8v8" podUID="e23f5f45-a08b-4a20-b61e-d33119765592" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 01 20:54:47.660980 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:47.651471 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-8ddfb06d-54d7b96ff7-flgn4" event={"ID":"497bee83-44f6-4eee-88c9-e62f051c8a28","Type":"ContainerStarted","Data":"18fa80bcf8ab16d987952008ece3f248a6c76c04874936346b60c65504950ac2"} Nov 01 20:54:47.660980 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:47.651590 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-8ddfb06d-54d7b96ff7-flgn4" Nov 01 20:54:47.660980 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:47.651626 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-8ddfb06d-54d7b96ff7-flgn4" event={"ID":"497bee83-44f6-4eee-88c9-e62f051c8a28","Type":"ContainerStarted","Data":"21edefba1902b8d8bbfcc655ef403635ee9e3a7ab1ef41ddfe27ae4e06f7be1f"} Nov 01 20:54:47.660980 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:47.653671 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8ddfb06d-66db94bbb7-2fxht" event={"ID":"15651aae-6296-488c-b5b3-fb525dbb5a68","Type":"ContainerStarted","Data":"2c760fa0c73e46ca90c0b02f93b85562a900185aa05a3f8d46979205413c30c0"} Nov 01 20:54:47.660980 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:47.653720 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8ddfb06d-66db94bbb7-2fxht" event={"ID":"15651aae-6296-488c-b5b3-fb525dbb5a68","Type":"ContainerStarted","Data":"0306a97af5a97990311c95f0dc3ff2047c296fea076e0614884093c9a62cd515"} Nov 01 20:54:47.660980 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:47.653758 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8ddfb06d-66db94bbb7-2fxht" Nov 01 20:54:47.792045 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:47.791965 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-8ddfb06d-54d7b96ff7-flgn4" podStartSLOduration=3.7919457100000002 podStartE2EDuration="3.79194571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:54:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:54:47.790049135 +0000 UTC m=+1438.567689270" watchObservedRunningTime="2026-11-01 20:54:47.79194571 +0000 UTC m=+1438.569585846" Nov 01 20:54:47.942078 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:47.937151 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8ddfb06d-66db94bbb7-2fxht" podStartSLOduration=3.937132952 podStartE2EDuration="3.937132952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:54:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:54:47.936406214 +0000 UTC m=+1438.714046350" watchObservedRunningTime="2026-11-01 20:54:47.937132952 +0000 UTC m=+1438.714773104" Nov 01 20:54:48.480925 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:48.480881 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:48.611031 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:48.610198 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-7b8v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 01 20:54:48.611031 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:48.610259 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7b8v8" podUID="e23f5f45-a08b-4a20-b61e-d33119765592" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 01 20:54:49.610683 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:49.610635 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-7b8v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 01 20:54:49.611208 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:49.610694 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7b8v8" podUID="e23f5f45-a08b-4a20-b61e-d33119765592" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 01 20:54:49.622576 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:49.622545 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:49.813087 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:49.810616 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:50.077197 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:50.020093 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-74ab02fe-78888c8555-t9shv"] Nov 01 20:54:50.105338 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:50.105309 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-74ab02fe-78888c8555-t9shv" Nov 01 20:54:50.137196 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:50.133716 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-74ab02fe-78888c8555-t9shv"] Nov 01 20:54:50.180174 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:50.180141 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-74ab02fe-7c85d47c89-2fpz8"] Nov 01 20:54:50.234017 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:50.218135 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5ld9\" (UniqueName: \"kubernetes.io/projected/f51e345c-11ff-428a-a715-8e0ec3ff98f8-kube-api-access-g5ld9\") pod \"payload-pre-processing-e2e-ait-74ab02fe-78888c8555-t9shv\" (UID: \"f51e345c-11ff-428a-a715-8e0ec3ff98f8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-74ab02fe-78888c8555-t9shv" Nov 01 20:54:50.234925 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:50.234886 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f51e345c-11ff-428a-a715-8e0ec3ff98f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-74ab02fe-78888c8555-t9shv\" (UID: \"f51e345c-11ff-428a-a715-8e0ec3ff98f8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-74ab02fe-78888c8555-t9shv" Nov 01 20:54:50.243706 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:50.243674 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-74ab02fe-7c85d47c89-2fpz8" Nov 01 20:54:50.244429 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:50.244400 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-74ab02fe-dockercfg-v8w2c\"" Nov 01 20:54:50.244701 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:50.244683 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-74ab02fe\"" Nov 01 20:54:50.252909 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:50.252872 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-74ab02fe-7c85d47c89-2fpz8"] Nov 01 20:54:50.465133 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:50.465097 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bcd38da-1bd5-4293-9d1b-531907031b36-plugins-config-volume\") pod \"payload-processing-e2e-ait-74ab02fe-7c85d47c89-2fpz8\" (UID: \"7bcd38da-1bd5-4293-9d1b-531907031b36\") " pod="openshift-ingress/payload-processing-e2e-ait-74ab02fe-7c85d47c89-2fpz8" Nov 01 20:54:50.466919 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:50.466884 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5ld9\" (UniqueName: \"kubernetes.io/projected/f51e345c-11ff-428a-a715-8e0ec3ff98f8-kube-api-access-g5ld9\") pod \"payload-pre-processing-e2e-ait-74ab02fe-78888c8555-t9shv\" (UID: \"f51e345c-11ff-428a-a715-8e0ec3ff98f8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-74ab02fe-78888c8555-t9shv" Nov 01 20:54:50.468426 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:50.468398 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sp89\" (UniqueName: \"kubernetes.io/projected/7bcd38da-1bd5-4293-9d1b-531907031b36-kube-api-access-2sp89\") pod \"payload-processing-e2e-ait-74ab02fe-7c85d47c89-2fpz8\" (UID: \"7bcd38da-1bd5-4293-9d1b-531907031b36\") " pod="openshift-ingress/payload-processing-e2e-ait-74ab02fe-7c85d47c89-2fpz8" Nov 01 20:54:50.468861 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:50.468837 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f51e345c-11ff-428a-a715-8e0ec3ff98f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-74ab02fe-78888c8555-t9shv\" (UID: \"f51e345c-11ff-428a-a715-8e0ec3ff98f8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-74ab02fe-78888c8555-t9shv" Nov 01 20:54:50.487043 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:50.472768 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f51e345c-11ff-428a-a715-8e0ec3ff98f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-74ab02fe-78888c8555-t9shv\" (UID: \"f51e345c-11ff-428a-a715-8e0ec3ff98f8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-74ab02fe-78888c8555-t9shv" Nov 01 20:54:50.548096 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:50.548028 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5ld9\" (UniqueName: \"kubernetes.io/projected/f51e345c-11ff-428a-a715-8e0ec3ff98f8-kube-api-access-g5ld9\") pod \"payload-pre-processing-e2e-ait-74ab02fe-78888c8555-t9shv\" (UID: \"f51e345c-11ff-428a-a715-8e0ec3ff98f8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-74ab02fe-78888c8555-t9shv" Nov 01 20:54:50.578019 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:50.572869 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bcd38da-1bd5-4293-9d1b-531907031b36-plugins-config-volume\") pod \"payload-processing-e2e-ait-74ab02fe-7c85d47c89-2fpz8\" (UID: \"7bcd38da-1bd5-4293-9d1b-531907031b36\") " pod="openshift-ingress/payload-processing-e2e-ait-74ab02fe-7c85d47c89-2fpz8" Nov 01 20:54:50.578019 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:50.573129 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sp89\" (UniqueName: \"kubernetes.io/projected/7bcd38da-1bd5-4293-9d1b-531907031b36-kube-api-access-2sp89\") pod \"payload-processing-e2e-ait-74ab02fe-7c85d47c89-2fpz8\" (UID: \"7bcd38da-1bd5-4293-9d1b-531907031b36\") " pod="openshift-ingress/payload-processing-e2e-ait-74ab02fe-7c85d47c89-2fpz8" Nov 01 20:54:50.578019 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:50.574856 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bcd38da-1bd5-4293-9d1b-531907031b36-plugins-config-volume\") pod \"payload-processing-e2e-ait-74ab02fe-7c85d47c89-2fpz8\" (UID: \"7bcd38da-1bd5-4293-9d1b-531907031b36\") " pod="openshift-ingress/payload-processing-e2e-ait-74ab02fe-7c85d47c89-2fpz8" Nov 01 20:54:50.615137 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:50.611318 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-7b8v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 01 20:54:50.615137 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:50.611381 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7b8v8" podUID="e23f5f45-a08b-4a20-b61e-d33119765592" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 01 20:54:50.668017 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:50.656885 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sp89\" (UniqueName: \"kubernetes.io/projected/7bcd38da-1bd5-4293-9d1b-531907031b36-kube-api-access-2sp89\") pod \"payload-processing-e2e-ait-74ab02fe-7c85d47c89-2fpz8\" (UID: \"7bcd38da-1bd5-4293-9d1b-531907031b36\") " pod="openshift-ingress/payload-processing-e2e-ait-74ab02fe-7c85d47c89-2fpz8" Nov 01 20:54:50.763017 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:50.755333 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-74ab02fe-7c85d47c89-2fpz8" Nov 01 20:54:50.773082 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:50.771799 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-74ab02fe-78888c8555-t9shv" Nov 01 20:54:51.178816 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:54:51.178784 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf51e345c_11ff_428a_a715_8e0ec3ff98f8.slice/crio-08d5366797c9568e444effa964ef81d6988bf003fe59c83e3e93b1c6095715da WatchSource:0}: Error finding container 08d5366797c9568e444effa964ef81d6988bf003fe59c83e3e93b1c6095715da: Status 404 returned error can't find the container with id 08d5366797c9568e444effa964ef81d6988bf003fe59c83e3e93b1c6095715da Nov 01 20:54:51.183841 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:51.183809 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-74ab02fe-78888c8555-t9shv"] Nov 01 20:54:51.229140 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:54:51.229106 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bcd38da_1bd5_4293_9d1b_531907031b36.slice/crio-51c9e0ef7c0f15622f78fb21edc61b033e0a7e102327aeca94ae5f70dab8cc0b WatchSource:0}: Error finding container 51c9e0ef7c0f15622f78fb21edc61b033e0a7e102327aeca94ae5f70dab8cc0b: Status 404 returned error can't find the container with id 51c9e0ef7c0f15622f78fb21edc61b033e0a7e102327aeca94ae5f70dab8cc0b Nov 01 20:54:51.229726 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:51.229699 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-74ab02fe-7c85d47c89-2fpz8"] Nov 01 20:54:51.891056 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:51.890976 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-74ab02fe-7c85d47c89-2fpz8" event={"ID":"7bcd38da-1bd5-4293-9d1b-531907031b36","Type":"ContainerStarted","Data":"ebc20707f8f6c44e9c12ddc51bcf2891caf32d30e455b426093537b537b386a0"} Nov 01 20:54:51.891056 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:51.891046 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-74ab02fe-7c85d47c89-2fpz8" event={"ID":"7bcd38da-1bd5-4293-9d1b-531907031b36","Type":"ContainerStarted","Data":"51c9e0ef7c0f15622f78fb21edc61b033e0a7e102327aeca94ae5f70dab8cc0b"} Nov 01 20:54:51.891645 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:51.891091 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-74ab02fe-7c85d47c89-2fpz8" Nov 01 20:54:51.893050 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:51.893021 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-74ab02fe-78888c8555-t9shv" event={"ID":"f51e345c-11ff-428a-a715-8e0ec3ff98f8","Type":"ContainerStarted","Data":"183229c1ab5dd344b6f4a1c9ceb59d9e924858f8915be2d47257d289e68b4d8b"} Nov 01 20:54:51.893230 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:51.893077 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-74ab02fe-78888c8555-t9shv" event={"ID":"f51e345c-11ff-428a-a715-8e0ec3ff98f8","Type":"ContainerStarted","Data":"08d5366797c9568e444effa964ef81d6988bf003fe59c83e3e93b1c6095715da"} Nov 01 20:54:51.893230 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:51.893106 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-74ab02fe-78888c8555-t9shv" Nov 01 20:54:52.006773 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:52.006717 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-74ab02fe-7c85d47c89-2fpz8" podStartSLOduration=2.006703266 podStartE2EDuration="2.006703266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:54:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:54:52.006205274 +0000 UTC m=+1442.783845413" watchObservedRunningTime="2026-11-01 20:54:52.006703266 +0000 UTC m=+1442.784343401" Nov 01 20:54:52.107940 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:52.107906 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7b8v8" Nov 01 20:54:52.108352 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:52.108325 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7b8v8" Nov 01 20:54:52.109265 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:52.109243 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7b8v8" Nov 01 20:54:52.164715 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:52.164578 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-74ab02fe-78888c8555-t9shv" podStartSLOduration=3.164561342 podStartE2EDuration="3.164561342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:54:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:54:52.150832023 +0000 UTC m=+1442.928472152" watchObservedRunningTime="2026-11-01 20:54:52.164561342 +0000 UTC m=+1442.942201476" Nov 01 20:54:53.538760 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:53.538729 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:55.939522 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:55.939470 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:57.335048 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:57.334970 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:57.500497 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:57.500463 4484 scope.go:122] "RemoveContainer" containerID="0fec2966bfe38d8d66c884c2c0769ddac9b3ab0e19f36d896ca179cfd4943305" Nov 01 20:54:58.618913 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:58.618827 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:58.730859 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:58.730814 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-8ddfb06d-54d7b96ff7-flgn4" Nov 01 20:54:58.737278 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:58.737249 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8ddfb06d-66db94bbb7-2fxht" Nov 01 20:54:58.845419 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:58.845385 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h"] Nov 01 20:54:58.852038 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:58.852014 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h" Nov 01 20:54:58.903460 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:58.903378 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h"] Nov 01 20:54:58.907272 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:58.907244 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-a5ab4417-openshift-default-dockercfg-4tcv5\"" Nov 01 20:54:58.929340 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:58.929258 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b91fbd7-5fe8-42de-b6be-fcbf67244747-workload-certs\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h\" (UID: \"3b91fbd7-5fe8-42de-b6be-fcbf67244747\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h" Nov 01 20:54:58.929540 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:58.929495 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b91fbd7-5fe8-42de-b6be-fcbf67244747-workload-socket\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h\" (UID: \"3b91fbd7-5fe8-42de-b6be-fcbf67244747\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h" Nov 01 20:54:58.929625 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:58.929566 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b91fbd7-5fe8-42de-b6be-fcbf67244747-istiod-ca-cert\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h\" (UID: \"3b91fbd7-5fe8-42de-b6be-fcbf67244747\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h" Nov 01 20:54:58.929717 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:58.929697 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b91fbd7-5fe8-42de-b6be-fcbf67244747-istio-token\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h\" (UID: \"3b91fbd7-5fe8-42de-b6be-fcbf67244747\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h" Nov 01 20:54:58.929942 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:58.929924 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b91fbd7-5fe8-42de-b6be-fcbf67244747-istio-envoy\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h\" (UID: \"3b91fbd7-5fe8-42de-b6be-fcbf67244747\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h" Nov 01 20:54:58.930123 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:58.930106 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b91fbd7-5fe8-42de-b6be-fcbf67244747-istio-data\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h\" (UID: \"3b91fbd7-5fe8-42de-b6be-fcbf67244747\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h" Nov 01 20:54:58.930324 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:58.930308 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b91fbd7-5fe8-42de-b6be-fcbf67244747-credential-socket\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h\" (UID: \"3b91fbd7-5fe8-42de-b6be-fcbf67244747\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h" Nov 01 20:54:58.930461 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:58.930447 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b91fbd7-5fe8-42de-b6be-fcbf67244747-istio-podinfo\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h\" (UID: \"3b91fbd7-5fe8-42de-b6be-fcbf67244747\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h" Nov 01 20:54:58.930606 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:58.930583 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn7m8\" (UniqueName: \"kubernetes.io/projected/3b91fbd7-5fe8-42de-b6be-fcbf67244747-kube-api-access-bn7m8\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h\" (UID: \"3b91fbd7-5fe8-42de-b6be-fcbf67244747\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h" Nov 01 20:54:58.949802 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:58.949762 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:59.175863 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:59.170702 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b91fbd7-5fe8-42de-b6be-fcbf67244747-istio-token\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h\" (UID: \"3b91fbd7-5fe8-42de-b6be-fcbf67244747\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h" Nov 01 20:54:59.175863 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:59.175614 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b91fbd7-5fe8-42de-b6be-fcbf67244747-istio-envoy\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h\" (UID: \"3b91fbd7-5fe8-42de-b6be-fcbf67244747\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h" Nov 01 20:54:59.175863 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:59.175724 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b91fbd7-5fe8-42de-b6be-fcbf67244747-istio-data\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h\" (UID: \"3b91fbd7-5fe8-42de-b6be-fcbf67244747\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h" Nov 01 20:54:59.175863 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:59.175860 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b91fbd7-5fe8-42de-b6be-fcbf67244747-credential-socket\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h\" (UID: \"3b91fbd7-5fe8-42de-b6be-fcbf67244747\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h" Nov 01 20:54:59.176327 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:59.175912 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b91fbd7-5fe8-42de-b6be-fcbf67244747-istio-podinfo\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h\" (UID: \"3b91fbd7-5fe8-42de-b6be-fcbf67244747\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h" Nov 01 20:54:59.177647 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:59.177305 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn7m8\" (UniqueName: \"kubernetes.io/projected/3b91fbd7-5fe8-42de-b6be-fcbf67244747-kube-api-access-bn7m8\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h\" (UID: \"3b91fbd7-5fe8-42de-b6be-fcbf67244747\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h" Nov 01 20:54:59.177647 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:59.177380 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b91fbd7-5fe8-42de-b6be-fcbf67244747-workload-certs\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h\" (UID: \"3b91fbd7-5fe8-42de-b6be-fcbf67244747\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h" Nov 01 20:54:59.177859 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:59.177677 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b91fbd7-5fe8-42de-b6be-fcbf67244747-workload-socket\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h\" (UID: \"3b91fbd7-5fe8-42de-b6be-fcbf67244747\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h" Nov 01 20:54:59.177859 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:59.177793 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b91fbd7-5fe8-42de-b6be-fcbf67244747-istiod-ca-cert\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h\" (UID: \"3b91fbd7-5fe8-42de-b6be-fcbf67244747\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h" Nov 01 20:54:59.184177 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:59.184136 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:59.184431 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:59.184207 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b91fbd7-5fe8-42de-b6be-fcbf67244747-istio-data\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h\" (UID: \"3b91fbd7-5fe8-42de-b6be-fcbf67244747\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h" Nov 01 20:54:59.184585 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:59.184437 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b91fbd7-5fe8-42de-b6be-fcbf67244747-credential-socket\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h\" (UID: \"3b91fbd7-5fe8-42de-b6be-fcbf67244747\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h" Nov 01 20:54:59.185207 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:59.185174 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b91fbd7-5fe8-42de-b6be-fcbf67244747-workload-socket\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h\" (UID: \"3b91fbd7-5fe8-42de-b6be-fcbf67244747\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h" Nov 01 20:54:59.185603 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:59.185573 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b91fbd7-5fe8-42de-b6be-fcbf67244747-workload-certs\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h\" (UID: \"3b91fbd7-5fe8-42de-b6be-fcbf67244747\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h" Nov 01 20:54:59.198967 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:59.198929 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b91fbd7-5fe8-42de-b6be-fcbf67244747-istiod-ca-cert\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h\" (UID: \"3b91fbd7-5fe8-42de-b6be-fcbf67244747\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h" Nov 01 20:54:59.200369 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:59.200312 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b91fbd7-5fe8-42de-b6be-fcbf67244747-istio-envoy\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h\" (UID: \"3b91fbd7-5fe8-42de-b6be-fcbf67244747\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h" Nov 01 20:54:59.246767 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:59.246311 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b91fbd7-5fe8-42de-b6be-fcbf67244747-istio-token\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h\" (UID: \"3b91fbd7-5fe8-42de-b6be-fcbf67244747\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h" Nov 01 20:54:59.287312 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:59.287224 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b91fbd7-5fe8-42de-b6be-fcbf67244747-istio-podinfo\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h\" (UID: \"3b91fbd7-5fe8-42de-b6be-fcbf67244747\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h" Nov 01 20:54:59.289095 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:59.289065 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn7m8\" (UniqueName: \"kubernetes.io/projected/3b91fbd7-5fe8-42de-b6be-fcbf67244747-kube-api-access-bn7m8\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h\" (UID: \"3b91fbd7-5fe8-42de-b6be-fcbf67244747\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h" Nov 01 20:54:59.364874 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:59.364834 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:59.486112 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:54:59.486022 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h" Nov 01 20:55:00.229771 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:55:00.229739 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b91fbd7_5fe8_42de_b6be_fcbf67244747.slice/crio-fa13d552b60e43d3b8686d009ed320c6b164acfbe20f502fb04ab9a929f9fc9f WatchSource:0}: Error finding container fa13d552b60e43d3b8686d009ed320c6b164acfbe20f502fb04ab9a929f9fc9f: Status 404 returned error can't find the container with id fa13d552b60e43d3b8686d009ed320c6b164acfbe20f502fb04ab9a929f9fc9f Nov 01 20:55:00.231659 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:00.231639 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h"] Nov 01 20:55:00.233563 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:00.233527 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:55:00.233644 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:00.233609 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:55:00.233692 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:00.233643 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:55:00.578121 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:00.578080 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:01.056125 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:01.056085 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-406eeae3-hzzpn"] Nov 01 20:55:01.127387 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:01.127004 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-406eeae3-hzzpn"] Nov 01 20:55:01.143029 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:01.136174 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:01.224740 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:01.224705 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h" event={"ID":"3b91fbd7-5fe8-42de-b6be-fcbf67244747","Type":"ContainerStarted","Data":"a03bdfbf3eeb002f5f1455a0bc0571dec18e0e0f9891e4e54033d8ef3a95366c"} Nov 01 20:55:01.224909 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:01.224767 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h" event={"ID":"3b91fbd7-5fe8-42de-b6be-fcbf67244747","Type":"ContainerStarted","Data":"fa13d552b60e43d3b8686d009ed320c6b164acfbe20f502fb04ab9a929f9fc9f"} Nov 01 20:55:01.353417 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:01.353310 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h" podStartSLOduration=3.353292911 podStartE2EDuration="3.353292911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:54:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:55:01.350045624 +0000 UTC m=+1452.127685761" watchObservedRunningTime="2026-11-01 20:55:01.353292911 +0000 UTC m=+1452.130933471" Nov 01 20:55:01.378642 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:01.378607 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d39c82f-ec44-4e41-a05b-649095064039" path="/var/lib/kubelet/pods/4d39c82f-ec44-4e41-a05b-649095064039/volumes" Nov 01 20:55:01.531017 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:01.494890 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h" Nov 01 20:55:01.531017 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:01.504385 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h" Nov 01 20:55:02.048605 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:02.048568 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gq6xw"] Nov 01 20:55:02.054760 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:02.054720 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gq6xw" Nov 01 20:55:02.100163 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:02.100119 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/17ef1a63-e718-4f21-958a-b353687cbff9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gq6xw\" (UID: \"17ef1a63-e718-4f21-958a-b353687cbff9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gq6xw" Nov 01 20:55:02.100348 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:02.100296 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17ef1a63-e718-4f21-958a-b353687cbff9-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gq6xw\" (UID: \"17ef1a63-e718-4f21-958a-b353687cbff9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gq6xw" Nov 01 20:55:02.100348 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:02.100333 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/17ef1a63-e718-4f21-958a-b353687cbff9-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gq6xw\" (UID: \"17ef1a63-e718-4f21-958a-b353687cbff9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gq6xw" Nov 01 20:55:02.100509 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:02.100480 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/17ef1a63-e718-4f21-958a-b353687cbff9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gq6xw\" (UID: \"17ef1a63-e718-4f21-958a-b353687cbff9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gq6xw" Nov 01 20:55:02.100547 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:02.100532 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86p56\" (UniqueName: \"kubernetes.io/projected/17ef1a63-e718-4f21-958a-b353687cbff9-kube-api-access-86p56\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gq6xw\" (UID: \"17ef1a63-e718-4f21-958a-b353687cbff9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gq6xw" Nov 01 20:55:02.100582 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:02.100571 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/17ef1a63-e718-4f21-958a-b353687cbff9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gq6xw\" (UID: \"17ef1a63-e718-4f21-958a-b353687cbff9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gq6xw" Nov 01 20:55:02.100751 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:02.100710 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/17ef1a63-e718-4f21-958a-b353687cbff9-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gq6xw\" (UID: \"17ef1a63-e718-4f21-958a-b353687cbff9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gq6xw" Nov 01 20:55:02.159706 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:02.159665 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gq6xw"] Nov 01 20:55:02.204418 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:02.204377 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/17ef1a63-e718-4f21-958a-b353687cbff9-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gq6xw\" (UID: \"17ef1a63-e718-4f21-958a-b353687cbff9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gq6xw" Nov 01 20:55:02.204836 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:02.204808 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/17ef1a63-e718-4f21-958a-b353687cbff9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gq6xw\" (UID: \"17ef1a63-e718-4f21-958a-b353687cbff9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gq6xw" Nov 01 20:55:02.205185 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:02.205162 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17ef1a63-e718-4f21-958a-b353687cbff9-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gq6xw\" (UID: \"17ef1a63-e718-4f21-958a-b353687cbff9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gq6xw" Nov 01 20:55:02.205375 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:02.205358 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/17ef1a63-e718-4f21-958a-b353687cbff9-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gq6xw\" (UID: \"17ef1a63-e718-4f21-958a-b353687cbff9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gq6xw" Nov 01 20:55:02.205566 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:02.205550 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/17ef1a63-e718-4f21-958a-b353687cbff9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gq6xw\" (UID: \"17ef1a63-e718-4f21-958a-b353687cbff9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gq6xw" Nov 01 20:55:02.205689 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:02.205673 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86p56\" (UniqueName: \"kubernetes.io/projected/17ef1a63-e718-4f21-958a-b353687cbff9-kube-api-access-86p56\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gq6xw\" (UID: \"17ef1a63-e718-4f21-958a-b353687cbff9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gq6xw" Nov 01 20:55:02.205848 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:02.205832 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/17ef1a63-e718-4f21-958a-b353687cbff9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gq6xw\" (UID: \"17ef1a63-e718-4f21-958a-b353687cbff9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gq6xw" Nov 01 20:55:02.212051 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:02.211650 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/17ef1a63-e718-4f21-958a-b353687cbff9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gq6xw\" (UID: \"17ef1a63-e718-4f21-958a-b353687cbff9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gq6xw" Nov 01 20:55:02.212650 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:02.212626 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/17ef1a63-e718-4f21-958a-b353687cbff9-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gq6xw\" (UID: \"17ef1a63-e718-4f21-958a-b353687cbff9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gq6xw" Nov 01 20:55:02.213532 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:02.213509 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/17ef1a63-e718-4f21-958a-b353687cbff9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gq6xw\" (UID: \"17ef1a63-e718-4f21-958a-b353687cbff9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gq6xw" Nov 01 20:55:02.216898 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:02.216694 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/17ef1a63-e718-4f21-958a-b353687cbff9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gq6xw\" (UID: \"17ef1a63-e718-4f21-958a-b353687cbff9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gq6xw" Nov 01 20:55:02.228700 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:02.228595 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/17ef1a63-e718-4f21-958a-b353687cbff9-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gq6xw\" (UID: \"17ef1a63-e718-4f21-958a-b353687cbff9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gq6xw" Nov 01 20:55:02.255897 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:02.250358 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h" Nov 01 20:55:02.255897 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:02.252162 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h" Nov 01 20:55:02.267571 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:02.267536 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17ef1a63-e718-4f21-958a-b353687cbff9-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gq6xw\" (UID: \"17ef1a63-e718-4f21-958a-b353687cbff9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gq6xw" Nov 01 20:55:02.276060 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:02.275594 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86p56\" (UniqueName: \"kubernetes.io/projected/17ef1a63-e718-4f21-958a-b353687cbff9-kube-api-access-86p56\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gq6xw\" (UID: \"17ef1a63-e718-4f21-958a-b353687cbff9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gq6xw" Nov 01 20:55:02.377328 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:02.377233 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gq6xw" Nov 01 20:55:02.906754 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:02.906721 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gq6xw"] Nov 01 20:55:02.907492 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:02.907463 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-74ab02fe-7c85d47c89-2fpz8" Nov 01 20:55:02.907612 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:02.907543 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-74ab02fe-78888c8555-t9shv" Nov 01 20:55:02.908731 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:55:02.908702 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17ef1a63_e718_4f21_958a_b353687cbff9.slice/crio-dc06938f2195edcd900dc508386d87faf44ceccdd6ebe3eccfb0903e025e4097 WatchSource:0}: Error finding container dc06938f2195edcd900dc508386d87faf44ceccdd6ebe3eccfb0903e025e4097: Status 404 returned error can't find the container with id dc06938f2195edcd900dc508386d87faf44ceccdd6ebe3eccfb0903e025e4097 Nov 01 20:55:02.911049 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:02.911016 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:55:02.911160 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:02.911144 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:55:03.174412 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:03.174199 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:03.283893 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:03.283851 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gq6xw" event={"ID":"17ef1a63-e718-4f21-958a-b353687cbff9","Type":"ContainerStarted","Data":"378cfaa22a0f1b84a9575991ac3b2a4e3af1e0f52b1b55c0fc388feb9d76c848"} Nov 01 20:55:03.284112 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:03.283919 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gq6xw" Nov 01 20:55:03.284112 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:03.283938 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gq6xw" event={"ID":"17ef1a63-e718-4f21-958a-b353687cbff9","Type":"ContainerStarted","Data":"dc06938f2195edcd900dc508386d87faf44ceccdd6ebe3eccfb0903e025e4097"} Nov 01 20:55:03.285819 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:03.285792 4484 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-gq6xw container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 20:55:03.285931 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:03.285843 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gq6xw" podUID="17ef1a63-e718-4f21-958a-b353687cbff9" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 20:55:03.391614 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:03.391577 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:03.407427 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:03.407373 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gq6xw" podStartSLOduration=2.407356594 podStartE2EDuration="2.407356594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:55:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:55:03.404766378 +0000 UTC m=+1454.182406510" watchObservedRunningTime="2026-11-01 20:55:03.407356594 +0000 UTC m=+1454.184996726" Nov 01 20:55:03.553626 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:03.553591 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:03.730552 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:03.730517 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:04.168012 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:04.167959 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:04.305017 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:04.304296 4484 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-gq6xw container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 20:55:04.305017 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:04.304353 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gq6xw" podUID="17ef1a63-e718-4f21-958a-b353687cbff9" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 20:55:04.378096 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:04.378044 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:04.580843 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:04.580808 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:05.164901 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:05.164867 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:05.323825 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:05.323787 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:05.484670 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:05.484589 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:07.311213 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:07.311178 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gq6xw" Nov 01 20:55:07.427950 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:07.427901 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:07.603266 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:07.603176 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:09.207525 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:09.207484 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:12.291638 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:12.291604 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:12.551984 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:12.551881 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:14.035598 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:14.035563 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:15.769214 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:55:15.769175 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 20:55:16.048977 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:16.048890 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a0cd91b-zvz2c"] Nov 01 20:55:16.099634 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:16.099594 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a0cd91b-zvz2c"] Nov 01 20:55:17.374702 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:17.374669 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d4f5a1e-364c-4282-ab9c-3b664313da65" path="/var/lib/kubelet/pods/8d4f5a1e-364c-4282-ab9c-3b664313da65/volumes" Nov 01 20:55:18.805410 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:18.805369 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:21.048550 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:21.048506 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1d4200f-4b5mk"] Nov 01 20:55:21.099732 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:21.099695 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1d4200f-4b5mk"] Nov 01 20:55:21.376383 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:21.376308 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9194558-fad7-4439-b531-278d6b4bea39" path="/var/lib/kubelet/pods/b9194558-fad7-4439-b531-278d6b4bea39/volumes" Nov 01 20:55:23.382259 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:23.382225 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:23.609302 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:23.609253 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:23.787031 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:23.784498 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:26.047918 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:26.047880 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bff043fa-lzr84"] Nov 01 20:55:26.100400 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:26.100366 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bff043fa-lzr84"] Nov 01 20:55:27.374336 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:27.374303 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fc560bd-a1ae-4889-8ca9-2c03d2ef39b4" path="/var/lib/kubelet/pods/3fc560bd-a1ae-4889-8ca9-2c03d2ef39b4/volumes" Nov 01 20:55:28.052874 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:28.052840 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:28.261022 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:28.253562 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:33.040915 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:33.040827 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:37.829357 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:37.829319 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:39.161768 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:39.161728 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:39.330729 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:39.330682 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:39.586319 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:39.586288 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:41.055249 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:41.055212 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9d537cc-9vsb4"] Nov 01 20:55:41.101207 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:41.101172 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9d537cc-9vsb4"] Nov 01 20:55:41.261725 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:41.261685 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:41.375382 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:41.375307 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="039b7fe4-4e78-41a5-aa66-61e5999c85bb" path="/var/lib/kubelet/pods/039b7fe4-4e78-41a5-aa66-61e5999c85bb/volumes" Nov 01 20:55:42.626657 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:42.626611 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:45.789124 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:55:45.789087 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 20:55:46.058669 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:46.058566 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78648dee-bptz9"] Nov 01 20:55:46.108789 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:46.108736 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:46.113136 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:46.113105 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78648dee-bptz9"] Nov 01 20:55:46.277327 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:46.277294 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:46.700580 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:46.700540 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:46.880398 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:46.880354 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:47.374830 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:47.374777 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="100bce4a-f707-4097-8902-50d8d064c4fd" path="/var/lib/kubelet/pods/100bce4a-f707-4097-8902-50d8d064c4fd/volumes" Nov 01 20:55:47.441896 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:47.441863 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:47.977065 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:47.976989 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:48.147923 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:48.147880 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:49.059691 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:49.059656 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c4eee07-2wkrt"] Nov 01 20:55:49.136244 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:49.136171 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c4eee07-2wkrt"] Nov 01 20:55:49.375287 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:49.375209 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77f7ce89-0b88-475b-b7b7-a9de5d77f34b" path="/var/lib/kubelet/pods/77f7ce89-0b88-475b-b7b7-a9de5d77f34b/volumes" Nov 01 20:55:51.604169 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:51.604075 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:52.195309 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:52.194974 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:52.503407 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:52.399317 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:52.846701 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:52.846576 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:54.247180 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:54.247142 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8aafd12-blzc2"] Nov 01 20:55:54.253243 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:54.253211 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8aafd12-blzc2"] Nov 01 20:55:55.171474 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:55.171441 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cced842-gp2px"] Nov 01 20:55:55.257276 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:55.257236 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cced842-gp2px"] Nov 01 20:55:55.257830 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:55.257395 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cced842-gp2px" Nov 01 20:55:55.288260 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:55.287379 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9d19cf1-edee-4256-b612-40b9e800c64a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cced842-gp2px\" (UID: \"a9d19cf1-edee-4256-b612-40b9e800c64a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cced842-gp2px" Nov 01 20:55:55.288260 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:55.287446 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhg89\" (UniqueName: \"kubernetes.io/projected/a9d19cf1-edee-4256-b612-40b9e800c64a-kube-api-access-xhg89\") pod \"maas-api-revoke-sub-0cced842-gp2px\" (UID: \"a9d19cf1-edee-4256-b612-40b9e800c64a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cced842-gp2px" Nov 01 20:55:55.288260 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:55.287546 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9d19cf1-edee-4256-b612-40b9e800c64a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cced842-gp2px\" (UID: \"a9d19cf1-edee-4256-b612-40b9e800c64a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cced842-gp2px" Nov 01 20:55:55.332097 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:55.327045 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:55:55.376513 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:55.367374 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:55:55.376513 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:55.367468 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 20:55:55.376513 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:55.367531 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:55:55.390094 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:55.389327 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9d19cf1-edee-4256-b612-40b9e800c64a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cced842-gp2px\" (UID: \"a9d19cf1-edee-4256-b612-40b9e800c64a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cced842-gp2px" Nov 01 20:55:55.390094 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:55.389404 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhg89\" (UniqueName: \"kubernetes.io/projected/a9d19cf1-edee-4256-b612-40b9e800c64a-kube-api-access-xhg89\") pod \"maas-api-revoke-sub-0cced842-gp2px\" (UID: \"a9d19cf1-edee-4256-b612-40b9e800c64a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cced842-gp2px" Nov 01 20:55:55.390094 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:55.389526 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9d19cf1-edee-4256-b612-40b9e800c64a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cced842-gp2px\" (UID: \"a9d19cf1-edee-4256-b612-40b9e800c64a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cced842-gp2px" Nov 01 20:55:55.391975 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:55.391940 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9d19cf1-edee-4256-b612-40b9e800c64a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cced842-gp2px\" (UID: \"a9d19cf1-edee-4256-b612-40b9e800c64a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cced842-gp2px" Nov 01 20:55:55.392172 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:55.392028 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a861ff5-a3a4-4a99-a555-cc253ca9dbe2" path="/var/lib/kubelet/pods/6a861ff5-a3a4-4a99-a555-cc253ca9dbe2/volumes" Nov 01 20:55:55.435753 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:55.435679 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9d19cf1-edee-4256-b612-40b9e800c64a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cced842-gp2px\" (UID: \"a9d19cf1-edee-4256-b612-40b9e800c64a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cced842-gp2px" Nov 01 20:55:55.449580 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:55.449544 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhg89\" (UniqueName: \"kubernetes.io/projected/a9d19cf1-edee-4256-b612-40b9e800c64a-kube-api-access-xhg89\") pod \"maas-api-revoke-sub-0cced842-gp2px\" (UID: \"a9d19cf1-edee-4256-b612-40b9e800c64a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cced842-gp2px" Nov 01 20:55:55.452833 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:55.452808 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cced842-gp2px" Nov 01 20:55:56.601162 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:56.601128 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:56.669173 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:56.669123 4484 patch_prober.go:29] interesting pod/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:55:56.669376 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:56.669193 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr" podUID="ce125f0b-43ab-40bf-ae39-5dc6e195ce39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:55:56.772254 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:56.772216 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:56.944149 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:56.944060 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:57.283021 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:57.282978 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:57.502312 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:57.501546 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:57.610636 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:57.610543 4484 scope.go:122] "RemoveContainer" containerID="2d07c40870b4d710599ff7fc3cd2a97fd130c0efca3c2f4a4e6298e7b99b9c53" Nov 01 20:55:57.674513 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:57.663471 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:57.695677 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:57.689404 4484 scope.go:122] "RemoveContainer" containerID="0ddc06b83759c2a680a9823fcaa63fdcdae437040acef41710457b2936e664d1" Nov 01 20:55:57.756067 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:57.756043 4484 scope.go:122] "RemoveContainer" containerID="0f53a479ccfa75fede0b3687eeb4f0dc68c8800a3c16030b1f40b4df0b6d48a1" Nov 01 20:55:57.792354 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:57.792332 4484 scope.go:122] "RemoveContainer" containerID="2f8a615f8064bdf305c7e3389389f5c5e0eb090a289042145801356573514750" Nov 01 20:55:58.364179 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:58.364154 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cced842-gp2px"] Nov 01 20:55:58.469020 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:55:58.468972 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9d19cf1_edee_4256_b612_40b9e800c64a.slice/crio-7987dda7b13c0b30fdcfae0a1ac4bab8b551870c278a2f5d75a5cf16e6bb29eb WatchSource:0}: Error finding container 7987dda7b13c0b30fdcfae0a1ac4bab8b551870c278a2f5d75a5cf16e6bb29eb: Status 404 returned error can't find the container with id 7987dda7b13c0b30fdcfae0a1ac4bab8b551870c278a2f5d75a5cf16e6bb29eb Nov 01 20:55:58.595893 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:58.595859 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cced842-gp2px" event={"ID":"a9d19cf1-edee-4256-b612-40b9e800c64a","Type":"ContainerStarted","Data":"7987dda7b13c0b30fdcfae0a1ac4bab8b551870c278a2f5d75a5cf16e6bb29eb"} Nov 01 20:55:59.702155 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:59.702113 4484 patch_prober.go:29] interesting pod/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:55:59.702611 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:55:59.702196 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg" podUID="8137e500-e107-49d5-81ba-948c7ae986f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:56:00.619975 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:00.618628 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cced842-gp2px" event={"ID":"a9d19cf1-edee-4256-b612-40b9e800c64a","Type":"ContainerStarted","Data":"13df01b9dcd89db86a44fe58a69b1abe0b74244a9339270a8a6443b2582db9e0"} Nov 01 20:56:00.736294 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:00.735845 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cced842-gp2px" podStartSLOduration=5.735826161 podStartE2EDuration="5.735826161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:55:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:56:00.733300037 +0000 UTC m=+1511.510940165" watchObservedRunningTime="2026-11-01 20:56:00.735826161 +0000 UTC m=+1511.513466297" Nov 01 20:56:01.048028 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:01.047971 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87582d8d-n8b29"] Nov 01 20:56:01.053118 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:01.053085 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87582d8d-n8b29"] Nov 01 20:56:01.411114 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:01.410990 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c1e21b4-946b-4351-92b3-948b442bdac3" path="/var/lib/kubelet/pods/9c1e21b4-946b-4351-92b3-948b442bdac3/volumes" Nov 01 20:56:02.052409 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:02.052366 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a49a9a09-p2vd8"] Nov 01 20:56:02.103412 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:02.103369 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a49a9a09-p2vd8"] Nov 01 20:56:02.408660 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:02.408571 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:56:02.660405 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:02.660311 4484 generic.go:356] "Generic (PLEG): container finished" podID="a9d19cf1-edee-4256-b612-40b9e800c64a" containerID="13df01b9dcd89db86a44fe58a69b1abe0b74244a9339270a8a6443b2582db9e0" exitCode=0 Nov 01 20:56:02.660638 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:02.660619 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cced842-gp2px" event={"ID":"a9d19cf1-edee-4256-b612-40b9e800c64a","Type":"ContainerDied","Data":"13df01b9dcd89db86a44fe58a69b1abe0b74244a9339270a8a6443b2582db9e0"} Nov 01 20:56:02.808881 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:02.808846 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:56:03.020019 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:03.019571 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:56:03.217107 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:03.216956 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dce296-65f5556975-m2d7v"] Nov 01 20:56:03.259304 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:03.252682 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dce296-65f5556975-m2d7v" Nov 01 20:56:03.288787 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:03.288657 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dce296-65f5556975-m2d7v"] Nov 01 20:56:03.308475 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:03.307755 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-dce296\"" Nov 01 20:56:03.453018 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:03.426285 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtzfw\" (UniqueName: \"kubernetes.io/projected/67b581c3-6009-4d4d-ba06-28df3b32142d-kube-api-access-vtzfw\") pod \"payload-pre-processing-e2e-shared-a-w1-dce296-65f5556975-m2d7v\" (UID: \"67b581c3-6009-4d4d-ba06-28df3b32142d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dce296-65f5556975-m2d7v" Nov 01 20:56:03.578381 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:03.570613 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67b581c3-6009-4d4d-ba06-28df3b32142d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-dce296-65f5556975-m2d7v\" (UID: \"67b581c3-6009-4d4d-ba06-28df3b32142d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dce296-65f5556975-m2d7v" Nov 01 20:56:03.608065 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:03.512863 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-dce296-dockercfg-7fchn\"" Nov 01 20:56:03.697519 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:03.697448 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fbaa918-a4bc-4157-8c63-5181f2e0ac86" path="/var/lib/kubelet/pods/2fbaa918-a4bc-4157-8c63-5181f2e0ac86/volumes" Nov 01 20:56:03.710305 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:03.710262 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67b581c3-6009-4d4d-ba06-28df3b32142d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-dce296-65f5556975-m2d7v\" (UID: \"67b581c3-6009-4d4d-ba06-28df3b32142d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dce296-65f5556975-m2d7v" Nov 01 20:56:03.795137 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:03.795091 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtzfw\" (UniqueName: \"kubernetes.io/projected/67b581c3-6009-4d4d-ba06-28df3b32142d-kube-api-access-vtzfw\") pod \"payload-pre-processing-e2e-shared-a-w1-dce296-65f5556975-m2d7v\" (UID: \"67b581c3-6009-4d4d-ba06-28df3b32142d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dce296-65f5556975-m2d7v" Nov 01 20:56:03.797554 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:03.733235 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-dce296-85d75dbb6f-rq556"] Nov 01 20:56:03.814209 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:03.714965 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67b581c3-6009-4d4d-ba06-28df3b32142d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-dce296-65f5556975-m2d7v\" (UID: \"67b581c3-6009-4d4d-ba06-28df3b32142d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dce296-65f5556975-m2d7v" Nov 01 20:56:03.908775 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:03.840572 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-dce296-85d75dbb6f-rq556" Nov 01 20:56:03.934161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:03.934043 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-dce296-85d75dbb6f-rq556"] Nov 01 20:56:04.009853 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:04.009813 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtzfw\" (UniqueName: \"kubernetes.io/projected/67b581c3-6009-4d4d-ba06-28df3b32142d-kube-api-access-vtzfw\") pod \"payload-pre-processing-e2e-shared-a-w1-dce296-65f5556975-m2d7v\" (UID: \"67b581c3-6009-4d4d-ba06-28df3b32142d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dce296-65f5556975-m2d7v" Nov 01 20:56:04.013062 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:04.012676 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/310baaa5-6b66-4af9-b08b-fe96fb2343bc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-dce296-85d75dbb6f-rq556\" (UID: \"310baaa5-6b66-4af9-b08b-fe96fb2343bc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-dce296-85d75dbb6f-rq556" Nov 01 20:56:04.013062 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:04.012727 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmwrc\" (UniqueName: \"kubernetes.io/projected/310baaa5-6b66-4af9-b08b-fe96fb2343bc-kube-api-access-jmwrc\") pod \"payload-processing-e2e-shared-a-w1-dce296-85d75dbb6f-rq556\" (UID: \"310baaa5-6b66-4af9-b08b-fe96fb2343bc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-dce296-85d75dbb6f-rq556" Nov 01 20:56:04.114470 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:04.114421 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/310baaa5-6b66-4af9-b08b-fe96fb2343bc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-dce296-85d75dbb6f-rq556\" (UID: \"310baaa5-6b66-4af9-b08b-fe96fb2343bc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-dce296-85d75dbb6f-rq556" Nov 01 20:56:04.114663 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:04.114501 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmwrc\" (UniqueName: \"kubernetes.io/projected/310baaa5-6b66-4af9-b08b-fe96fb2343bc-kube-api-access-jmwrc\") pod \"payload-processing-e2e-shared-a-w1-dce296-85d75dbb6f-rq556\" (UID: \"310baaa5-6b66-4af9-b08b-fe96fb2343bc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-dce296-85d75dbb6f-rq556" Nov 01 20:56:04.115391 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:04.115367 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/310baaa5-6b66-4af9-b08b-fe96fb2343bc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-dce296-85d75dbb6f-rq556\" (UID: \"310baaa5-6b66-4af9-b08b-fe96fb2343bc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-dce296-85d75dbb6f-rq556" Nov 01 20:56:04.172121 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:04.172043 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmwrc\" (UniqueName: \"kubernetes.io/projected/310baaa5-6b66-4af9-b08b-fe96fb2343bc-kube-api-access-jmwrc\") pod \"payload-processing-e2e-shared-a-w1-dce296-85d75dbb6f-rq556\" (UID: \"310baaa5-6b66-4af9-b08b-fe96fb2343bc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-dce296-85d75dbb6f-rq556" Nov 01 20:56:04.258055 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:04.258015 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-dce296-85d75dbb6f-rq556" Nov 01 20:56:04.278916 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:04.278872 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dce296-65f5556975-m2d7v" Nov 01 20:56:04.691208 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:04.691164 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cced842-gp2px" Nov 01 20:56:04.748138 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:04.748105 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-dce296-85d75dbb6f-rq556"] Nov 01 20:56:04.786015 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:56:04.785972 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod310baaa5_6b66_4af9_b08b_fe96fb2343bc.slice/crio-328e939a7b85691f6a88c408c59ab35732c2160183ee66fce0e97c266b18730b WatchSource:0}: Error finding container 328e939a7b85691f6a88c408c59ab35732c2160183ee66fce0e97c266b18730b: Status 404 returned error can't find the container with id 328e939a7b85691f6a88c408c59ab35732c2160183ee66fce0e97c266b18730b Nov 01 20:56:04.796566 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:56:04.796531 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67b581c3_6009_4d4d_ba06_28df3b32142d.slice/crio-6ec73e695ae378551b349ca8eee20c0bd2aa4db87127f98a1f37ac4fb9cfd1a9 WatchSource:0}: Error finding container 6ec73e695ae378551b349ca8eee20c0bd2aa4db87127f98a1f37ac4fb9cfd1a9: Status 404 returned error can't find the container with id 6ec73e695ae378551b349ca8eee20c0bd2aa4db87127f98a1f37ac4fb9cfd1a9 Nov 01 20:56:04.799071 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:04.799048 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dce296-65f5556975-m2d7v"] Nov 01 20:56:04.827899 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:04.826311 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cced842-gp2px" Nov 01 20:56:04.827899 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:04.826484 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cced842-gp2px" event={"ID":"a9d19cf1-edee-4256-b612-40b9e800c64a","Type":"ContainerDied","Data":"7987dda7b13c0b30fdcfae0a1ac4bab8b551870c278a2f5d75a5cf16e6bb29eb"} Nov 01 20:56:04.827899 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:04.826522 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7987dda7b13c0b30fdcfae0a1ac4bab8b551870c278a2f5d75a5cf16e6bb29eb" Nov 01 20:56:04.827899 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:04.827461 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9d19cf1-edee-4256-b612-40b9e800c64a-kube-api-access-xhg89\" (UniqueName: \"kubernetes.io/projected/a9d19cf1-edee-4256-b612-40b9e800c64a-kube-api-access-xhg89\") pod \"a9d19cf1-edee-4256-b612-40b9e800c64a\" (UID: \"a9d19cf1-edee-4256-b612-40b9e800c64a\") " Nov 01 20:56:04.827899 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:04.827792 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9d19cf1-edee-4256-b612-40b9e800c64a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9d19cf1-edee-4256-b612-40b9e800c64a-maas-api-xks-ca\") pod \"a9d19cf1-edee-4256-b612-40b9e800c64a\" (UID: \"a9d19cf1-edee-4256-b612-40b9e800c64a\") " Nov 01 20:56:04.828299 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:04.828072 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9d19cf1-edee-4256-b612-40b9e800c64a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9d19cf1-edee-4256-b612-40b9e800c64a-maas-api-openshift-ca\") pod \"a9d19cf1-edee-4256-b612-40b9e800c64a\" (UID: \"a9d19cf1-edee-4256-b612-40b9e800c64a\") " Nov 01 20:56:04.829290 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:04.829262 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9d19cf1-edee-4256-b612-40b9e800c64a-maas-api-openshift-ca" pod "a9d19cf1-edee-4256-b612-40b9e800c64a" (UID: "a9d19cf1-edee-4256-b612-40b9e800c64a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:56:04.831966 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:04.829638 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-dce296-85d75dbb6f-rq556" event={"ID":"310baaa5-6b66-4af9-b08b-fe96fb2343bc","Type":"ContainerStarted","Data":"328e939a7b85691f6a88c408c59ab35732c2160183ee66fce0e97c266b18730b"} Nov 01 20:56:04.833796 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:04.833763 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dce296-65f5556975-m2d7v" event={"ID":"67b581c3-6009-4d4d-ba06-28df3b32142d","Type":"ContainerStarted","Data":"6ec73e695ae378551b349ca8eee20c0bd2aa4db87127f98a1f37ac4fb9cfd1a9"} Nov 01 20:56:04.840432 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:04.840384 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9d19cf1-edee-4256-b612-40b9e800c64a-maas-api-xks-ca" pod "a9d19cf1-edee-4256-b612-40b9e800c64a" (UID: "a9d19cf1-edee-4256-b612-40b9e800c64a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:56:04.840714 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:04.840687 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9d19cf1-edee-4256-b612-40b9e800c64a-kube-api-access-xhg89" pod "a9d19cf1-edee-4256-b612-40b9e800c64a" (UID: "a9d19cf1-edee-4256-b612-40b9e800c64a"). InnerVolumeSpecName "kube-api-access-xhg89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:56:04.931237 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:04.931076 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9d19cf1-edee-4256-b612-40b9e800c64a-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:56:04.931237 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:04.931105 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9d19cf1-edee-4256-b612-40b9e800c64a-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:56:04.931237 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:04.931115 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhg89\" (UniqueName: \"kubernetes.io/projected/a9d19cf1-edee-4256-b612-40b9e800c64a-kube-api-access-xhg89\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:56:05.845866 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:05.845826 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-dce296-85d75dbb6f-rq556" event={"ID":"310baaa5-6b66-4af9-b08b-fe96fb2343bc","Type":"ContainerStarted","Data":"694caacbd8b4a69773fc61e773f0b0ba4c3e268d45f78f66a0d6fa45558b50c7"} Nov 01 20:56:05.846551 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:05.845897 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-dce296-85d75dbb6f-rq556" Nov 01 20:56:05.847867 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:05.847839 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dce296-65f5556975-m2d7v" event={"ID":"67b581c3-6009-4d4d-ba06-28df3b32142d","Type":"ContainerStarted","Data":"37e444356439ca69ee7f3503bfbb2a9b769e319e877fc9078b0ee7fc9c4120d0"} Nov 01 20:56:05.847990 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:05.847894 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dce296-65f5556975-m2d7v" Nov 01 20:56:05.964201 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:05.964147 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-dce296-85d75dbb6f-rq556" podStartSLOduration=2.96413157 podStartE2EDuration="2.96413157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:56:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:56:05.960574675 +0000 UTC m=+1516.738214811" watchObservedRunningTime="2026-11-01 20:56:05.96413157 +0000 UTC m=+1516.741771705" Nov 01 20:56:06.051105 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:06.051068 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad59ce44-2g6wh"] Nov 01 20:56:06.056259 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:06.056227 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad59ce44-2g6wh"] Nov 01 20:56:06.105090 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:06.102367 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dce296-65f5556975-m2d7v" podStartSLOduration=3.10234576 podStartE2EDuration="3.10234576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:56:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:56:06.099348761 +0000 UTC m=+1516.876988897" watchObservedRunningTime="2026-11-01 20:56:06.10234576 +0000 UTC m=+1516.879985896" Nov 01 20:56:07.229019 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:07.223009 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:56:07.378508 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:07.378475 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbb6aaa5-b66d-421d-91ea-1f65e593b8f5" path="/var/lib/kubelet/pods/dbb6aaa5-b66d-421d-91ea-1f65e593b8f5/volumes" Nov 01 20:56:09.889220 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:09.889185 4484 generic.go:356] "Generic (PLEG): container finished" podID="8137e500-e107-49d5-81ba-948c7ae986f1" containerID="c76773bf6f98025744a313005c3c15dc5dc2e8f3e252a02daf73199676275fb0" exitCode=0 Nov 01 20:56:09.889731 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:09.889255 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg" event={"ID":"8137e500-e107-49d5-81ba-948c7ae986f1","Type":"ContainerDied","Data":"c76773bf6f98025744a313005c3c15dc5dc2e8f3e252a02daf73199676275fb0"} Nov 01 20:56:09.889786 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:09.889775 4484 scope.go:122] "RemoveContainer" containerID="c76773bf6f98025744a313005c3c15dc5dc2e8f3e252a02daf73199676275fb0" Nov 01 20:56:09.891288 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:09.891266 4484 generic.go:356] "Generic (PLEG): container finished" podID="ce125f0b-43ab-40bf-ae39-5dc6e195ce39" containerID="4207d5f3773e1840c07c8df01b337d98418cd7d3b1813cc705a5e1673dfa7760" exitCode=0 Nov 01 20:56:09.891381 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:09.891292 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr" event={"ID":"ce125f0b-43ab-40bf-ae39-5dc6e195ce39","Type":"ContainerDied","Data":"4207d5f3773e1840c07c8df01b337d98418cd7d3b1813cc705a5e1673dfa7760"} Nov 01 20:56:09.891701 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:09.891688 4484 scope.go:122] "RemoveContainer" containerID="4207d5f3773e1840c07c8df01b337d98418cd7d3b1813cc705a5e1673dfa7760" Nov 01 20:56:09.891739 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:09.891707 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:56:09.891791 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:09.891777 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:56:09.891831 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:09.891813 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:56:09.893251 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:09.893221 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:56:09.893365 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:09.893301 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:56:09.893365 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:09.893354 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:56:10.052476 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:10.048958 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa38f47b-45shn"] Nov 01 20:56:10.055754 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:10.055715 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa38f47b-45shn"] Nov 01 20:56:10.626915 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:10.626879 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr" Nov 01 20:56:10.903725 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:10.903631 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg" event={"ID":"8137e500-e107-49d5-81ba-948c7ae986f1","Type":"ContainerStarted","Data":"a2e6c67edb3b9a3cc7729e33e2f7e1f38315d93b3ee4df918886d9cb59ca5650"} Nov 01 20:56:10.908188 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:10.908155 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr" event={"ID":"ce125f0b-43ab-40bf-ae39-5dc6e195ce39","Type":"ContainerStarted","Data":"f3adf3c2d14d3aac70cf5721d09c367dfad1b21a4e8b37014fbbf98c8b10e37f"} Nov 01 20:56:11.378259 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:11.378223 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e3f93a5-1229-4a1f-994a-e0698add77e9" path="/var/lib/kubelet/pods/6e3f93a5-1229-4a1f-994a-e0698add77e9/volumes" Nov 01 20:56:11.847802 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:11.847763 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg" Nov 01 20:56:11.848969 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:11.848929 4484 patch_prober.go:29] interesting pod/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 01 20:56:11.849077 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:11.849018 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg" podUID="8137e500-e107-49d5-81ba-948c7ae986f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 01 20:56:11.883369 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:11.883337 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr" Nov 01 20:56:11.884204 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:11.884177 4484 patch_prober.go:29] interesting pod/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Nov 01 20:56:11.884335 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:11.884234 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr" podUID="ce125f0b-43ab-40bf-ae39-5dc6e195ce39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Nov 01 20:56:11.935968 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:11.935935 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:56:12.485382 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:12.485348 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw"] Nov 01 20:56:12.485574 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:12.485551 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw" podUID="f4b1073f-60a3-444a-8746-a3bbba04ee86" containerName="istio-proxy" containerID="cri-o://a2b5fbfbd2122488f2433f61d5e30cc2e6450652a7b82943134887eacc4b4e61" gracePeriod=30 Nov 01 20:56:12.863027 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:12.862954 4484 patch_prober.go:29] interesting pod/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 01 20:56:12.863289 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:12.863264 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg" podUID="8137e500-e107-49d5-81ba-948c7ae986f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 01 20:56:12.883279 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:12.883250 4484 patch_prober.go:29] interesting pod/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Nov 01 20:56:12.883444 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:12.883335 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr" podUID="ce125f0b-43ab-40bf-ae39-5dc6e195ce39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Nov 01 20:56:13.663807 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:13.663774 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg" Nov 01 20:56:13.852631 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:13.852592 4484 patch_prober.go:29] interesting pod/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 01 20:56:13.852911 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:13.852880 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg" podUID="8137e500-e107-49d5-81ba-948c7ae986f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 01 20:56:13.882531 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:13.882501 4484 patch_prober.go:29] interesting pod/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Nov 01 20:56:13.882755 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:13.882735 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr" podUID="ce125f0b-43ab-40bf-ae39-5dc6e195ce39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Nov 01 20:56:14.852539 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:14.852500 4484 patch_prober.go:29] interesting pod/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 01 20:56:14.853193 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:14.853157 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg" podUID="8137e500-e107-49d5-81ba-948c7ae986f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 01 20:56:14.882674 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:14.882644 4484 patch_prober.go:29] interesting pod/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Nov 01 20:56:14.882844 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:14.882694 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr" podUID="ce125f0b-43ab-40bf-ae39-5dc6e195ce39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Nov 01 20:56:15.849501 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:15.849458 4484 patch_prober.go:29] interesting pod/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 01 20:56:15.849807 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:15.849766 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg" podUID="8137e500-e107-49d5-81ba-948c7ae986f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 01 20:56:15.858531 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:56:15.857638 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 20:56:15.883422 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:15.883393 4484 patch_prober.go:29] interesting pod/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Nov 01 20:56:15.883586 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:15.883463 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr" podUID="ce125f0b-43ab-40bf-ae39-5dc6e195ce39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Nov 01 20:56:15.941103 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:15.941063 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:56:16.744451 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:16.744421 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:56:16.897026 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:16.885329 4484 patch_prober.go:29] interesting pod/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Nov 01 20:56:16.897026 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:16.885407 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr" podUID="ce125f0b-43ab-40bf-ae39-5dc6e195ce39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Nov 01 20:56:16.897026 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:16.887168 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dce296-65f5556975-m2d7v" Nov 01 20:56:16.898485 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:16.898461 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-dce296-85d75dbb6f-rq556" Nov 01 20:56:17.383395 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:17.383362 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg" Nov 01 20:56:17.807085 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:17.804689 4484 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" podUID="e175b298-2c23-45d4-8f9c-0c9b9f6aab92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:56:17.840074 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:17.813748 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg" Nov 01 20:56:17.890320 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:17.883904 4484 patch_prober.go:29] interesting pod/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Nov 01 20:56:17.890320 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:17.883964 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr" podUID="ce125f0b-43ab-40bf-ae39-5dc6e195ce39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Nov 01 20:56:18.059965 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:18.059893 4484 generic.go:356] "Generic (PLEG): container finished" podID="f4b1073f-60a3-444a-8746-a3bbba04ee86" containerID="a2b5fbfbd2122488f2433f61d5e30cc2e6450652a7b82943134887eacc4b4e61" exitCode=0 Nov 01 20:56:18.061269 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:18.061048 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw" event={"ID":"f4b1073f-60a3-444a-8746-a3bbba04ee86","Type":"ContainerDied","Data":"a2b5fbfbd2122488f2433f61d5e30cc2e6450652a7b82943134887eacc4b4e61"} Nov 01 20:56:18.403022 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:18.401355 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw" Nov 01 20:56:18.518026 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:18.490222 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4b1073f-60a3-444a-8746-a3bbba04ee86-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4b1073f-60a3-444a-8746-a3bbba04ee86-workload-socket\") pod \"f4b1073f-60a3-444a-8746-a3bbba04ee86\" (UID: \"f4b1073f-60a3-444a-8746-a3bbba04ee86\") " Nov 01 20:56:18.518357 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:18.518326 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4b1073f-60a3-444a-8746-a3bbba04ee86-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4b1073f-60a3-444a-8746-a3bbba04ee86-istiod-ca-cert\") pod \"f4b1073f-60a3-444a-8746-a3bbba04ee86\" (UID: \"f4b1073f-60a3-444a-8746-a3bbba04ee86\") " Nov 01 20:56:18.518638 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:18.518610 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f4b1073f-60a3-444a-8746-a3bbba04ee86-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4b1073f-60a3-444a-8746-a3bbba04ee86-istio-podinfo\") pod \"f4b1073f-60a3-444a-8746-a3bbba04ee86\" (UID: \"f4b1073f-60a3-444a-8746-a3bbba04ee86\") " Nov 01 20:56:18.518835 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:18.518817 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4b1073f-60a3-444a-8746-a3bbba04ee86-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4b1073f-60a3-444a-8746-a3bbba04ee86-istio-data\") pod \"f4b1073f-60a3-444a-8746-a3bbba04ee86\" (UID: \"f4b1073f-60a3-444a-8746-a3bbba04ee86\") " Nov 01 20:56:18.519212 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:18.519191 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4b1073f-60a3-444a-8746-a3bbba04ee86-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4b1073f-60a3-444a-8746-a3bbba04ee86-istio-envoy\") pod \"f4b1073f-60a3-444a-8746-a3bbba04ee86\" (UID: \"f4b1073f-60a3-444a-8746-a3bbba04ee86\") " Nov 01 20:56:18.519392 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:18.519374 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4b1073f-60a3-444a-8746-a3bbba04ee86-kube-api-access-92n2s\" (UniqueName: \"kubernetes.io/projected/f4b1073f-60a3-444a-8746-a3bbba04ee86-kube-api-access-92n2s\") pod \"f4b1073f-60a3-444a-8746-a3bbba04ee86\" (UID: \"f4b1073f-60a3-444a-8746-a3bbba04ee86\") " Nov 01 20:56:18.519594 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:18.519577 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4b1073f-60a3-444a-8746-a3bbba04ee86-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4b1073f-60a3-444a-8746-a3bbba04ee86-credential-socket\") pod \"f4b1073f-60a3-444a-8746-a3bbba04ee86\" (UID: \"f4b1073f-60a3-444a-8746-a3bbba04ee86\") " Nov 01 20:56:18.519877 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:18.519858 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4b1073f-60a3-444a-8746-a3bbba04ee86-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4b1073f-60a3-444a-8746-a3bbba04ee86-workload-certs\") pod \"f4b1073f-60a3-444a-8746-a3bbba04ee86\" (UID: \"f4b1073f-60a3-444a-8746-a3bbba04ee86\") " Nov 01 20:56:18.520081 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:18.520062 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4b1073f-60a3-444a-8746-a3bbba04ee86-istio-token\" (UniqueName: \"kubernetes.io/projected/f4b1073f-60a3-444a-8746-a3bbba04ee86-istio-token\") pod \"f4b1073f-60a3-444a-8746-a3bbba04ee86\" (UID: \"f4b1073f-60a3-444a-8746-a3bbba04ee86\") " Nov 01 20:56:18.551035 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:18.519574 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4b1073f-60a3-444a-8746-a3bbba04ee86-istio-data" pod "f4b1073f-60a3-444a-8746-a3bbba04ee86" (UID: "f4b1073f-60a3-444a-8746-a3bbba04ee86"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:56:18.551035 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:18.519909 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4b1073f-60a3-444a-8746-a3bbba04ee86-workload-socket" pod "f4b1073f-60a3-444a-8746-a3bbba04ee86" (UID: "f4b1073f-60a3-444a-8746-a3bbba04ee86"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:56:18.551035 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:18.525838 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4b1073f-60a3-444a-8746-a3bbba04ee86-istiod-ca-cert" pod "f4b1073f-60a3-444a-8746-a3bbba04ee86" (UID: "f4b1073f-60a3-444a-8746-a3bbba04ee86"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:56:18.551035 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:18.537068 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4b1073f-60a3-444a-8746-a3bbba04ee86-credential-socket" pod "f4b1073f-60a3-444a-8746-a3bbba04ee86" (UID: "f4b1073f-60a3-444a-8746-a3bbba04ee86"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:56:18.551035 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:18.539850 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4b1073f-60a3-444a-8746-a3bbba04ee86-workload-certs" pod "f4b1073f-60a3-444a-8746-a3bbba04ee86" (UID: "f4b1073f-60a3-444a-8746-a3bbba04ee86"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:56:18.551035 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:18.547111 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4b1073f-60a3-444a-8746-a3bbba04ee86-istio-envoy" pod "f4b1073f-60a3-444a-8746-a3bbba04ee86" (UID: "f4b1073f-60a3-444a-8746-a3bbba04ee86"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:56:18.606206 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:18.606104 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4b1073f-60a3-444a-8746-a3bbba04ee86-istio-podinfo" pod "f4b1073f-60a3-444a-8746-a3bbba04ee86" (UID: "f4b1073f-60a3-444a-8746-a3bbba04ee86"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:56:18.655388 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:18.632109 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4b1073f-60a3-444a-8746-a3bbba04ee86-kube-api-access-92n2s" pod "f4b1073f-60a3-444a-8746-a3bbba04ee86" (UID: "f4b1073f-60a3-444a-8746-a3bbba04ee86"). InnerVolumeSpecName "kube-api-access-92n2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:56:18.655388 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:18.632594 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4b1073f-60a3-444a-8746-a3bbba04ee86-istio-token" pod "f4b1073f-60a3-444a-8746-a3bbba04ee86" (UID: "f4b1073f-60a3-444a-8746-a3bbba04ee86"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:56:18.724077 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:18.696979 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4b1073f-60a3-444a-8746-a3bbba04ee86-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:56:18.724077 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:18.697028 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4b1073f-60a3-444a-8746-a3bbba04ee86-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:56:18.724077 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:18.697042 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4b1073f-60a3-444a-8746-a3bbba04ee86-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:56:18.724077 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:18.697055 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4b1073f-60a3-444a-8746-a3bbba04ee86-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:56:18.724077 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:18.697071 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4b1073f-60a3-444a-8746-a3bbba04ee86-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:56:18.724077 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:18.697085 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4b1073f-60a3-444a-8746-a3bbba04ee86-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:56:18.724077 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:18.697098 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4b1073f-60a3-444a-8746-a3bbba04ee86-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:56:18.724077 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:18.697111 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4b1073f-60a3-444a-8746-a3bbba04ee86-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:56:18.724077 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:18.697126 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92n2s\" (UniqueName: \"kubernetes.io/projected/f4b1073f-60a3-444a-8746-a3bbba04ee86-kube-api-access-92n2s\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:56:18.967070 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:18.966118 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr" Nov 01 20:56:18.967538 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:18.967510 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr" Nov 01 20:56:18.970612 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:18.970588 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr" Nov 01 20:56:19.075438 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:19.075412 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw" Nov 01 20:56:19.075979 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:19.075400 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw" event={"ID":"f4b1073f-60a3-444a-8746-a3bbba04ee86","Type":"ContainerDied","Data":"7c99a32adf6a596e6da8019c850c8fd35926991890e09e41ff3c0bef84a00162"} Nov 01 20:56:19.075979 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:19.075625 4484 scope.go:122] "RemoveContainer" containerID="a2b5fbfbd2122488f2433f61d5e30cc2e6450652a7b82943134887eacc4b4e61" Nov 01 20:56:19.493791 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:19.493045 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw"] Nov 01 20:56:19.493791 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:19.493128 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-xzjrw"] Nov 01 20:56:21.088685 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:21.088651 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c325b215-sf5cd"] Nov 01 20:56:21.119928 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:21.119876 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c325b215-sf5cd"] Nov 01 20:56:21.413613 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:21.413519 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" podUID="e175b298-2c23-45d4-8f9c-0c9b9f6aab92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:56:21.463016 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:21.462969 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4507914c-adb9-40c8-8b1e-7494f6cfde91" path="/var/lib/kubelet/pods/4507914c-adb9-40c8-8b1e-7494f6cfde91/volumes" Nov 01 20:56:21.464625 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:21.464590 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4b1073f-60a3-444a-8746-a3bbba04ee86" path="/var/lib/kubelet/pods/f4b1073f-60a3-444a-8746-a3bbba04ee86/volumes" Nov 01 20:56:21.670203 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:21.670097 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" podUID="70b4dacf-b3ac-46ec-99e8-a44f05587e7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:56:21.689590 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:21.689546 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" podUID="9f19be29-2c84-4ae4-9674-acca4905f4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:56:21.713030 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:21.710179 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:56:22.007802 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:22.007654 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:56:22.148034 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:22.138101 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" podUID="17a9ac24-19cb-48c4-99b5-d74ce8aac0a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:56:23.048714 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:23.048675 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49fe0f19-rpvxc"] Nov 01 20:56:23.100512 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:23.100476 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49fe0f19-rpvxc"] Nov 01 20:56:23.374519 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:23.374436 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c3df29e-158c-4dc1-aef8-849271bdf172" path="/var/lib/kubelet/pods/0c3df29e-158c-4dc1-aef8-849271bdf172/volumes" Nov 01 20:56:26.303049 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:26.299561 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:56:26.490372 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:26.490316 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:56:27.105027 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:27.102622 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:56:27.935219 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:27.935135 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:56:29.052879 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:29.052837 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-422d19ef-wqltd"] Nov 01 20:56:29.100748 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:29.100708 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-422d19ef-wqltd"] Nov 01 20:56:29.382939 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:29.382847 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5b0b65e-dce0-4cf8-a6d1-10d3836e6b83" path="/var/lib/kubelet/pods/d5b0b65e-dce0-4cf8-a6d1-10d3836e6b83/volumes" Nov 01 20:56:29.485774 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:29.485736 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ddfb06d-kgkv5"] Nov 01 20:56:29.486812 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:29.486790 4484 cpu_manager.go:414] "containerMap: removing container" podUID="a9d19cf1-edee-4256-b612-40b9e800c64a" containerName="revoke-keys" Nov 01 20:56:29.486988 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:29.486973 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9d19cf1-edee-4256-b612-40b9e800c64a" containerName="revoke-keys" Nov 01 20:56:29.487195 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:29.487178 4484 cpu_manager.go:414] "containerMap: removing container" podUID="f4b1073f-60a3-444a-8746-a3bbba04ee86" containerName="istio-proxy" Nov 01 20:56:29.487297 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:29.487285 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4b1073f-60a3-444a-8746-a3bbba04ee86" containerName="istio-proxy" Nov 01 20:56:29.487640 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:29.487622 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4b1073f-60a3-444a-8746-a3bbba04ee86" containerName="istio-proxy" Nov 01 20:56:29.487959 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:29.487940 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9d19cf1-edee-4256-b612-40b9e800c64a" containerName="revoke-keys" Nov 01 20:56:29.523130 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:29.523098 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ddfb06d-kgkv5" Nov 01 20:56:29.545138 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:29.545106 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ddfb06d-kgkv5"] Nov 01 20:56:29.589683 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:29.589649 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:56:29.589864 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:29.589743 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:56:29.589864 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:29.589784 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 20:56:29.589864 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:29.589820 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:56:29.656947 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:29.656856 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2"] Nov 01 20:56:29.673538 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:29.673198 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2" Nov 01 20:56:29.688237 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:29.688178 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcd1f232-a94e-46f8-9f55-87153161bb61-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8ddfb06d-kgkv5\" (UID: \"fcd1f232-a94e-46f8-9f55-87153161bb61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ddfb06d-kgkv5" Nov 01 20:56:29.688387 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:29.688288 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcd1f232-a94e-46f8-9f55-87153161bb61-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8ddfb06d-kgkv5\" (UID: \"fcd1f232-a94e-46f8-9f55-87153161bb61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ddfb06d-kgkv5" Nov 01 20:56:29.688441 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:29.688403 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n8r7\" (UniqueName: \"kubernetes.io/projected/fcd1f232-a94e-46f8-9f55-87153161bb61-kube-api-access-6n8r7\") pod \"maas-api-revoke-keys-e2e-migrate-8ddfb06d-kgkv5\" (UID: \"fcd1f232-a94e-46f8-9f55-87153161bb61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ddfb06d-kgkv5" Nov 01 20:56:29.757197 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:29.757046 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2"] Nov 01 20:56:29.790269 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:29.790156 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71cb71c6-14d7-4e6d-a269-22b3b66fe170-istio-envoy\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2\" (UID: \"71cb71c6-14d7-4e6d-a269-22b3b66fe170\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2" Nov 01 20:56:29.790553 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:29.790321 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71cb71c6-14d7-4e6d-a269-22b3b66fe170-istio-podinfo\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2\" (UID: \"71cb71c6-14d7-4e6d-a269-22b3b66fe170\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2" Nov 01 20:56:29.790553 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:29.790360 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n8r7\" (UniqueName: \"kubernetes.io/projected/fcd1f232-a94e-46f8-9f55-87153161bb61-kube-api-access-6n8r7\") pod \"maas-api-revoke-keys-e2e-migrate-8ddfb06d-kgkv5\" (UID: \"fcd1f232-a94e-46f8-9f55-87153161bb61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ddfb06d-kgkv5" Nov 01 20:56:29.790553 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:29.790519 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71cb71c6-14d7-4e6d-a269-22b3b66fe170-workload-socket\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2\" (UID: \"71cb71c6-14d7-4e6d-a269-22b3b66fe170\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2" Nov 01 20:56:29.790838 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:29.790650 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71cb71c6-14d7-4e6d-a269-22b3b66fe170-istiod-ca-cert\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2\" (UID: \"71cb71c6-14d7-4e6d-a269-22b3b66fe170\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2" Nov 01 20:56:29.790838 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:29.790731 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71cb71c6-14d7-4e6d-a269-22b3b66fe170-workload-certs\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2\" (UID: \"71cb71c6-14d7-4e6d-a269-22b3b66fe170\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2" Nov 01 20:56:29.790838 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:29.790799 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98jc5\" (UniqueName: \"kubernetes.io/projected/71cb71c6-14d7-4e6d-a269-22b3b66fe170-kube-api-access-98jc5\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2\" (UID: \"71cb71c6-14d7-4e6d-a269-22b3b66fe170\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2" Nov 01 20:56:29.791845 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:29.791816 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71cb71c6-14d7-4e6d-a269-22b3b66fe170-credential-socket\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2\" (UID: \"71cb71c6-14d7-4e6d-a269-22b3b66fe170\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2" Nov 01 20:56:29.791973 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:29.791954 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcd1f232-a94e-46f8-9f55-87153161bb61-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8ddfb06d-kgkv5\" (UID: \"fcd1f232-a94e-46f8-9f55-87153161bb61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ddfb06d-kgkv5" Nov 01 20:56:29.792107 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:29.792070 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcd1f232-a94e-46f8-9f55-87153161bb61-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8ddfb06d-kgkv5\" (UID: \"fcd1f232-a94e-46f8-9f55-87153161bb61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ddfb06d-kgkv5" Nov 01 20:56:29.792183 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:29.792148 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71cb71c6-14d7-4e6d-a269-22b3b66fe170-istio-token\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2\" (UID: \"71cb71c6-14d7-4e6d-a269-22b3b66fe170\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2" Nov 01 20:56:29.792276 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:29.792259 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71cb71c6-14d7-4e6d-a269-22b3b66fe170-istio-data\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2\" (UID: \"71cb71c6-14d7-4e6d-a269-22b3b66fe170\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2" Nov 01 20:56:29.793562 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:29.793534 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcd1f232-a94e-46f8-9f55-87153161bb61-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8ddfb06d-kgkv5\" (UID: \"fcd1f232-a94e-46f8-9f55-87153161bb61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ddfb06d-kgkv5" Nov 01 20:56:29.796751 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:29.796719 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcd1f232-a94e-46f8-9f55-87153161bb61-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8ddfb06d-kgkv5\" (UID: \"fcd1f232-a94e-46f8-9f55-87153161bb61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ddfb06d-kgkv5" Nov 01 20:56:29.850434 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:29.850394 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n8r7\" (UniqueName: \"kubernetes.io/projected/fcd1f232-a94e-46f8-9f55-87153161bb61-kube-api-access-6n8r7\") pod \"maas-api-revoke-keys-e2e-migrate-8ddfb06d-kgkv5\" (UID: \"fcd1f232-a94e-46f8-9f55-87153161bb61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ddfb06d-kgkv5" Nov 01 20:56:29.855441 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:29.855409 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ddfb06d-kgkv5" Nov 01 20:56:29.894299 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:29.894261 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71cb71c6-14d7-4e6d-a269-22b3b66fe170-credential-socket\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2\" (UID: \"71cb71c6-14d7-4e6d-a269-22b3b66fe170\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2" Nov 01 20:56:29.894498 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:29.894401 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71cb71c6-14d7-4e6d-a269-22b3b66fe170-istio-token\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2\" (UID: \"71cb71c6-14d7-4e6d-a269-22b3b66fe170\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2" Nov 01 20:56:29.894498 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:29.894446 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71cb71c6-14d7-4e6d-a269-22b3b66fe170-istio-data\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2\" (UID: \"71cb71c6-14d7-4e6d-a269-22b3b66fe170\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2" Nov 01 20:56:29.894498 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:29.894467 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71cb71c6-14d7-4e6d-a269-22b3b66fe170-istio-envoy\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2\" (UID: \"71cb71c6-14d7-4e6d-a269-22b3b66fe170\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2" Nov 01 20:56:29.894714 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:29.894502 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71cb71c6-14d7-4e6d-a269-22b3b66fe170-istio-podinfo\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2\" (UID: \"71cb71c6-14d7-4e6d-a269-22b3b66fe170\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2" Nov 01 20:56:29.894714 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:29.894529 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71cb71c6-14d7-4e6d-a269-22b3b66fe170-workload-socket\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2\" (UID: \"71cb71c6-14d7-4e6d-a269-22b3b66fe170\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2" Nov 01 20:56:29.894714 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:29.894572 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71cb71c6-14d7-4e6d-a269-22b3b66fe170-istiod-ca-cert\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2\" (UID: \"71cb71c6-14d7-4e6d-a269-22b3b66fe170\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2" Nov 01 20:56:29.894714 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:29.894610 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71cb71c6-14d7-4e6d-a269-22b3b66fe170-workload-certs\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2\" (UID: \"71cb71c6-14d7-4e6d-a269-22b3b66fe170\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2" Nov 01 20:56:29.894714 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:29.894654 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98jc5\" (UniqueName: \"kubernetes.io/projected/71cb71c6-14d7-4e6d-a269-22b3b66fe170-kube-api-access-98jc5\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2\" (UID: \"71cb71c6-14d7-4e6d-a269-22b3b66fe170\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2" Nov 01 20:56:29.896676 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:29.895582 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71cb71c6-14d7-4e6d-a269-22b3b66fe170-credential-socket\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2\" (UID: \"71cb71c6-14d7-4e6d-a269-22b3b66fe170\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2" Nov 01 20:56:29.896676 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:29.896109 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71cb71c6-14d7-4e6d-a269-22b3b66fe170-workload-socket\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2\" (UID: \"71cb71c6-14d7-4e6d-a269-22b3b66fe170\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2" Nov 01 20:56:29.896676 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:29.896398 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71cb71c6-14d7-4e6d-a269-22b3b66fe170-istio-data\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2\" (UID: \"71cb71c6-14d7-4e6d-a269-22b3b66fe170\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2" Nov 01 20:56:29.897098 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:29.896808 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71cb71c6-14d7-4e6d-a269-22b3b66fe170-istiod-ca-cert\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2\" (UID: \"71cb71c6-14d7-4e6d-a269-22b3b66fe170\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2" Nov 01 20:56:29.897523 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:29.897497 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71cb71c6-14d7-4e6d-a269-22b3b66fe170-workload-certs\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2\" (UID: \"71cb71c6-14d7-4e6d-a269-22b3b66fe170\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2" Nov 01 20:56:29.900080 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:29.899215 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71cb71c6-14d7-4e6d-a269-22b3b66fe170-istio-podinfo\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2\" (UID: \"71cb71c6-14d7-4e6d-a269-22b3b66fe170\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2" Nov 01 20:56:29.900080 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:29.900033 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71cb71c6-14d7-4e6d-a269-22b3b66fe170-istio-envoy\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2\" (UID: \"71cb71c6-14d7-4e6d-a269-22b3b66fe170\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2" Nov 01 20:56:29.963691 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:29.963650 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71cb71c6-14d7-4e6d-a269-22b3b66fe170-istio-token\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2\" (UID: \"71cb71c6-14d7-4e6d-a269-22b3b66fe170\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2" Nov 01 20:56:29.965106 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:29.965076 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98jc5\" (UniqueName: \"kubernetes.io/projected/71cb71c6-14d7-4e6d-a269-22b3b66fe170-kube-api-access-98jc5\") pod \"e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2\" (UID: \"71cb71c6-14d7-4e6d-a269-22b3b66fe170\") " pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2" Nov 01 20:56:30.003271 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:30.002983 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2" Nov 01 20:56:30.309619 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:30.308981 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ddfb06d-kgkv5"] Nov 01 20:56:30.450019 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:30.446092 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2"] Nov 01 20:56:30.467078 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:56:30.460141 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71cb71c6_14d7_4e6d_a269_22b3b66fe170.slice/crio-fca6534bb45ef07689ef179134de8549695b7cee94014ae2dd0b98ae85e54e8b WatchSource:0}: Error finding container fca6534bb45ef07689ef179134de8549695b7cee94014ae2dd0b98ae85e54e8b: Status 404 returned error can't find the container with id fca6534bb45ef07689ef179134de8549695b7cee94014ae2dd0b98ae85e54e8b Nov 01 20:56:30.467078 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:30.464449 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:56:30.467078 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:30.464532 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:56:30.467078 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:30.464577 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:56:30.598617 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:30.598521 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2" event={"ID":"71cb71c6-14d7-4e6d-a269-22b3b66fe170","Type":"ContainerStarted","Data":"fca6534bb45ef07689ef179134de8549695b7cee94014ae2dd0b98ae85e54e8b"} Nov 01 20:56:30.608014 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:30.607959 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ddfb06d-kgkv5" event={"ID":"fcd1f232-a94e-46f8-9f55-87153161bb61","Type":"ContainerStarted","Data":"d1775aa8a2c7b2d454835c0b1cdeb928dc4203787a4a7070cefe32afa825f694"} Nov 01 20:56:31.105225 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:31.105085 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-feab1a53-8qzv9"] Nov 01 20:56:31.177075 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:31.174903 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-feab1a53-8qzv9"] Nov 01 20:56:31.385212 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:31.385178 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18d805d2-dbd0-4576-92ec-204a0c44c078" path="/var/lib/kubelet/pods/18d805d2-dbd0-4576-92ec-204a0c44c078/volumes" Nov 01 20:56:31.542323 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:31.542289 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:56:31.684671 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:31.683773 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ddfb06d-kgkv5" event={"ID":"fcd1f232-a94e-46f8-9f55-87153161bb61","Type":"ContainerStarted","Data":"1dab405e5b2cc18eda77a2e06a5d8fe6fe8c65b70a3e67b3a04d5b123b535de6"} Nov 01 20:56:31.701562 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:31.700269 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h" event={"ID":"3b91fbd7-5fe8-42de-b6be-fcbf67244747","Type":"ContainerDied","Data":"a03bdfbf3eeb002f5f1455a0bc0571dec18e0e0f9891e4e54033d8ef3a95366c"} Nov 01 20:56:31.701562 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:31.700795 4484 scope.go:122] "RemoveContainer" containerID="a03bdfbf3eeb002f5f1455a0bc0571dec18e0e0f9891e4e54033d8ef3a95366c" Nov 01 20:56:31.704720 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:31.703795 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:56:31.704720 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:31.703892 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:56:31.704720 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:31.703942 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:56:31.729217 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:31.729163 4484 generic.go:356] "Generic (PLEG): container finished" podID="3b91fbd7-5fe8-42de-b6be-fcbf67244747" containerID="a03bdfbf3eeb002f5f1455a0bc0571dec18e0e0f9891e4e54033d8ef3a95366c" exitCode=0 Nov 01 20:56:31.783017 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:31.767183 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2" event={"ID":"71cb71c6-14d7-4e6d-a269-22b3b66fe170","Type":"ContainerStarted","Data":"cd11d06566365a70138161b43321e9b9c4c45deb5dc7f2cf42208dc04e355338"} Nov 01 20:56:31.938557 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:31.938445 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2" podStartSLOduration=2.938425817 podStartE2EDuration="2.938425817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:56:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:56:31.925507375 +0000 UTC m=+1542.703147510" watchObservedRunningTime="2026-11-01 20:56:31.938425817 +0000 UTC m=+1542.716065956" Nov 01 20:56:32.006020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:32.005727 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2" Nov 01 20:56:32.006020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:32.005855 4484 patch_prober.go:29] interesting pod/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 01 20:56:32.006020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:32.005899 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2" podUID="71cb71c6-14d7-4e6d-a269-22b3b66fe170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 01 20:56:32.053070 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:32.051603 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d2f4c30-gskrl"] Nov 01 20:56:32.067088 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:32.066515 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d2f4c30-gskrl"] Nov 01 20:56:32.082252 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:32.081857 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:56:32.250959 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:32.250920 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h" Nov 01 20:56:32.274259 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:32.274223 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:56:32.467513 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:32.467478 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:56:32.787050 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:32.786987 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h" event={"ID":"3b91fbd7-5fe8-42de-b6be-fcbf67244747","Type":"ContainerStarted","Data":"e83d80fb7c64778872459faedd92c0daf10744f5a1c69d26dbd0648d54b98295"} Nov 01 20:56:32.789022 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:32.788983 4484 generic.go:356] "Generic (PLEG): container finished" podID="fcd1f232-a94e-46f8-9f55-87153161bb61" containerID="1dab405e5b2cc18eda77a2e06a5d8fe6fe8c65b70a3e67b3a04d5b123b535de6" exitCode=0 Nov 01 20:56:32.789247 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:32.789046 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ddfb06d-kgkv5" event={"ID":"fcd1f232-a94e-46f8-9f55-87153161bb61","Type":"ContainerDied","Data":"1dab405e5b2cc18eda77a2e06a5d8fe6fe8c65b70a3e67b3a04d5b123b535de6"} Nov 01 20:56:33.067516 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:33.067359 4484 patch_prober.go:29] interesting pod/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 01 20:56:33.067516 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:33.067427 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2" podUID="71cb71c6-14d7-4e6d-a269-22b3b66fe170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 01 20:56:33.171762 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:33.171736 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ddfb06d-kgkv5" Nov 01 20:56:33.271814 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:33.271779 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcd1f232-a94e-46f8-9f55-87153161bb61-kube-api-access-6n8r7\" (UniqueName: \"kubernetes.io/projected/fcd1f232-a94e-46f8-9f55-87153161bb61-kube-api-access-6n8r7\") pod \"fcd1f232-a94e-46f8-9f55-87153161bb61\" (UID: \"fcd1f232-a94e-46f8-9f55-87153161bb61\") " Nov 01 20:56:33.272016 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:33.271930 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcd1f232-a94e-46f8-9f55-87153161bb61-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcd1f232-a94e-46f8-9f55-87153161bb61-maas-api-openshift-ca\") pod \"fcd1f232-a94e-46f8-9f55-87153161bb61\" (UID: \"fcd1f232-a94e-46f8-9f55-87153161bb61\") " Nov 01 20:56:33.272081 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:33.272026 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcd1f232-a94e-46f8-9f55-87153161bb61-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcd1f232-a94e-46f8-9f55-87153161bb61-maas-api-xks-ca\") pod \"fcd1f232-a94e-46f8-9f55-87153161bb61\" (UID: \"fcd1f232-a94e-46f8-9f55-87153161bb61\") " Nov 01 20:56:33.272476 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:33.272440 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcd1f232-a94e-46f8-9f55-87153161bb61-maas-api-openshift-ca" pod "fcd1f232-a94e-46f8-9f55-87153161bb61" (UID: "fcd1f232-a94e-46f8-9f55-87153161bb61"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:56:33.272701 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:33.272683 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcd1f232-a94e-46f8-9f55-87153161bb61-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:56:33.274365 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:33.274341 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcd1f232-a94e-46f8-9f55-87153161bb61-maas-api-xks-ca" pod "fcd1f232-a94e-46f8-9f55-87153161bb61" (UID: "fcd1f232-a94e-46f8-9f55-87153161bb61"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:56:33.276371 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:33.276352 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcd1f232-a94e-46f8-9f55-87153161bb61-kube-api-access-6n8r7" pod "fcd1f232-a94e-46f8-9f55-87153161bb61" (UID: "fcd1f232-a94e-46f8-9f55-87153161bb61"). InnerVolumeSpecName "kube-api-access-6n8r7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:56:33.374758 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:33.374590 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n8r7\" (UniqueName: \"kubernetes.io/projected/fcd1f232-a94e-46f8-9f55-87153161bb61-kube-api-access-6n8r7\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:56:33.374758 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:33.374615 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcd1f232-a94e-46f8-9f55-87153161bb61-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:56:33.380065 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:33.380040 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a086ec6d-bd79-41b5-b29d-64994f49a0a7" path="/var/lib/kubelet/pods/a086ec6d-bd79-41b5-b29d-64994f49a0a7/volumes" Nov 01 20:56:33.487079 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:33.487042 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h" Nov 01 20:56:33.487954 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:33.487931 4484 patch_prober.go:29] interesting pod/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 01 20:56:33.488024 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:33.487988 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h" podUID="3b91fbd7-5fe8-42de-b6be-fcbf67244747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 01 20:56:33.800480 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:33.800447 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ddfb06d-kgkv5" Nov 01 20:56:33.800692 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:33.800447 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ddfb06d-kgkv5" event={"ID":"fcd1f232-a94e-46f8-9f55-87153161bb61","Type":"ContainerDied","Data":"d1775aa8a2c7b2d454835c0b1cdeb928dc4203787a4a7070cefe32afa825f694"} Nov 01 20:56:33.800692 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:33.800575 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1775aa8a2c7b2d454835c0b1cdeb928dc4203787a4a7070cefe32afa825f694" Nov 01 20:56:34.007021 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:34.006472 4484 patch_prober.go:29] interesting pod/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 01 20:56:34.007021 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:34.006543 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2" podUID="71cb71c6-14d7-4e6d-a269-22b3b66fe170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 01 20:56:34.488178 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:34.487547 4484 patch_prober.go:29] interesting pod/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 01 20:56:34.488178 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:34.487598 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h" podUID="3b91fbd7-5fe8-42de-b6be-fcbf67244747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 01 20:56:35.031627 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:35.031140 4484 patch_prober.go:29] interesting pod/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 01 20:56:35.031627 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:35.031209 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2" podUID="71cb71c6-14d7-4e6d-a269-22b3b66fe170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 01 20:56:35.486636 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:35.486550 4484 patch_prober.go:29] interesting pod/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 01 20:56:35.486636 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:35.486607 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h" podUID="3b91fbd7-5fe8-42de-b6be-fcbf67244747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 01 20:56:36.022021 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:36.011558 4484 patch_prober.go:29] interesting pod/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 01 20:56:36.022021 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:36.011626 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2" podUID="71cb71c6-14d7-4e6d-a269-22b3b66fe170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 01 20:56:36.487147 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:36.487059 4484 patch_prober.go:29] interesting pod/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 01 20:56:36.487147 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:36.487109 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h" podUID="3b91fbd7-5fe8-42de-b6be-fcbf67244747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 01 20:56:36.858637 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:36.858340 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:56:37.044225 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:37.044171 4484 patch_prober.go:29] interesting pod/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 01 20:56:37.044878 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:37.044830 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2" podUID="71cb71c6-14d7-4e6d-a269-22b3b66fe170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 01 20:56:37.047129 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:37.047105 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:56:37.279020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:37.278396 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-24f923-869bcbd967-w25fl"] Nov 01 20:56:37.285047 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:37.284629 4484 cpu_manager.go:414] "containerMap: removing container" podUID="fcd1f232-a94e-46f8-9f55-87153161bb61" containerName="revoke-keys" Nov 01 20:56:37.285047 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:37.284660 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcd1f232-a94e-46f8-9f55-87153161bb61" containerName="revoke-keys" Nov 01 20:56:37.285047 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:37.284909 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcd1f232-a94e-46f8-9f55-87153161bb61" containerName="revoke-keys" Nov 01 20:56:37.312018 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:37.296021 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-24f923-869bcbd967-w25fl" Nov 01 20:56:37.417369 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:37.388404 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-24f923-dockercfg-fs5mz\"" Nov 01 20:56:37.417369 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:37.389207 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-24f923\"" Nov 01 20:56:37.522041 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:37.442540 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdl8q\" (UniqueName: \"kubernetes.io/projected/009125b3-fc6d-4cbe-aa45-5cf27cc39a2e-kube-api-access-vdl8q\") pod \"payload-pre-processing-e2e-shared-a-w3-24f923-869bcbd967-w25fl\" (UID: \"009125b3-fc6d-4cbe-aa45-5cf27cc39a2e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-24f923-869bcbd967-w25fl" Nov 01 20:56:37.522041 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:37.442832 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/009125b3-fc6d-4cbe-aa45-5cf27cc39a2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-24f923-869bcbd967-w25fl\" (UID: \"009125b3-fc6d-4cbe-aa45-5cf27cc39a2e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-24f923-869bcbd967-w25fl" Nov 01 20:56:37.543252 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:37.543164 4484 patch_prober.go:29] interesting pod/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 01 20:56:37.543518 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:37.543489 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h" podUID="3b91fbd7-5fe8-42de-b6be-fcbf67244747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 01 20:56:37.559141 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:37.553367 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdl8q\" (UniqueName: \"kubernetes.io/projected/009125b3-fc6d-4cbe-aa45-5cf27cc39a2e-kube-api-access-vdl8q\") pod \"payload-pre-processing-e2e-shared-a-w3-24f923-869bcbd967-w25fl\" (UID: \"009125b3-fc6d-4cbe-aa45-5cf27cc39a2e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-24f923-869bcbd967-w25fl" Nov 01 20:56:37.559141 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:37.553654 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/009125b3-fc6d-4cbe-aa45-5cf27cc39a2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-24f923-869bcbd967-w25fl\" (UID: \"009125b3-fc6d-4cbe-aa45-5cf27cc39a2e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-24f923-869bcbd967-w25fl" Nov 01 20:56:37.559141 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:37.554756 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/009125b3-fc6d-4cbe-aa45-5cf27cc39a2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-24f923-869bcbd967-w25fl\" (UID: \"009125b3-fc6d-4cbe-aa45-5cf27cc39a2e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-24f923-869bcbd967-w25fl" Nov 01 20:56:37.619964 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:37.619930 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-24f923-869bcbd967-w25fl"] Nov 01 20:56:37.753853 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:37.753818 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdl8q\" (UniqueName: \"kubernetes.io/projected/009125b3-fc6d-4cbe-aa45-5cf27cc39a2e-kube-api-access-vdl8q\") pod \"payload-pre-processing-e2e-shared-a-w3-24f923-869bcbd967-w25fl\" (UID: \"009125b3-fc6d-4cbe-aa45-5cf27cc39a2e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-24f923-869bcbd967-w25fl" Nov 01 20:56:37.779363 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:37.779325 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74ab02fe-c6dpl"] Nov 01 20:56:37.860320 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:37.860211 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74ab02fe-c6dpl" Nov 01 20:56:37.863388 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:37.863360 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-24f923-869bcbd967-w25fl" Nov 01 20:56:37.942794 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:37.917255 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 20:56:37.942794 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:37.917350 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:56:37.942794 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:37.917444 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:56:37.942794 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:37.917497 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:56:37.971041 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:37.970440 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-24f923-8474d84f5-kfvdm"] Nov 01 20:56:38.005619 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:38.005136 4484 patch_prober.go:29] interesting pod/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 01 20:56:38.005619 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:38.005206 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2" podUID="71cb71c6-14d7-4e6d-a269-22b3b66fe170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 01 20:56:38.012670 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:38.011572 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74ab02fe-c6dpl"] Nov 01 20:56:38.012670 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:38.011630 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-24f923-8474d84f5-kfvdm"] Nov 01 20:56:38.012670 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:38.011731 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-24f923-8474d84f5-kfvdm" Nov 01 20:56:38.157247 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:38.157142 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" podUID="17a9ac24-19cb-48c4-99b5-d74ce8aac0a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:56:38.163078 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:38.163014 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b594f63-48a1-4a3c-bda5-e52c600c9a41-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-74ab02fe-c6dpl\" (UID: \"7b594f63-48a1-4a3c-bda5-e52c600c9a41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74ab02fe-c6dpl" Nov 01 20:56:38.163300 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:38.163277 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgxrx\" (UniqueName: \"kubernetes.io/projected/7b594f63-48a1-4a3c-bda5-e52c600c9a41-kube-api-access-wgxrx\") pod \"maas-api-revoke-keys-e2e-ait-74ab02fe-c6dpl\" (UID: \"7b594f63-48a1-4a3c-bda5-e52c600c9a41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74ab02fe-c6dpl" Nov 01 20:56:38.163514 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:38.163498 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b594f63-48a1-4a3c-bda5-e52c600c9a41-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-74ab02fe-c6dpl\" (UID: \"7b594f63-48a1-4a3c-bda5-e52c600c9a41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74ab02fe-c6dpl" Nov 01 20:56:38.167803 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:38.165584 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g779\" (UniqueName: \"kubernetes.io/projected/f9a691db-7b44-40aa-8924-1f2b716da33a-kube-api-access-7g779\") pod \"payload-processing-e2e-shared-a-w3-24f923-8474d84f5-kfvdm\" (UID: \"f9a691db-7b44-40aa-8924-1f2b716da33a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-24f923-8474d84f5-kfvdm" Nov 01 20:56:38.167803 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:38.165788 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9a691db-7b44-40aa-8924-1f2b716da33a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-24f923-8474d84f5-kfvdm\" (UID: \"f9a691db-7b44-40aa-8924-1f2b716da33a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-24f923-8474d84f5-kfvdm" Nov 01 20:56:38.267594 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:38.267562 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g779\" (UniqueName: \"kubernetes.io/projected/f9a691db-7b44-40aa-8924-1f2b716da33a-kube-api-access-7g779\") pod \"payload-processing-e2e-shared-a-w3-24f923-8474d84f5-kfvdm\" (UID: \"f9a691db-7b44-40aa-8924-1f2b716da33a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-24f923-8474d84f5-kfvdm" Nov 01 20:56:38.267764 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:38.267662 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9a691db-7b44-40aa-8924-1f2b716da33a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-24f923-8474d84f5-kfvdm\" (UID: \"f9a691db-7b44-40aa-8924-1f2b716da33a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-24f923-8474d84f5-kfvdm" Nov 01 20:56:38.267922 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:38.267903 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b594f63-48a1-4a3c-bda5-e52c600c9a41-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-74ab02fe-c6dpl\" (UID: \"7b594f63-48a1-4a3c-bda5-e52c600c9a41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74ab02fe-c6dpl" Nov 01 20:56:38.268022 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:38.267935 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgxrx\" (UniqueName: \"kubernetes.io/projected/7b594f63-48a1-4a3c-bda5-e52c600c9a41-kube-api-access-wgxrx\") pod \"maas-api-revoke-keys-e2e-ait-74ab02fe-c6dpl\" (UID: \"7b594f63-48a1-4a3c-bda5-e52c600c9a41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74ab02fe-c6dpl" Nov 01 20:56:38.268089 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:38.268072 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b594f63-48a1-4a3c-bda5-e52c600c9a41-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-74ab02fe-c6dpl\" (UID: \"7b594f63-48a1-4a3c-bda5-e52c600c9a41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74ab02fe-c6dpl" Nov 01 20:56:38.268838 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:38.268811 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9a691db-7b44-40aa-8924-1f2b716da33a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-24f923-8474d84f5-kfvdm\" (UID: \"f9a691db-7b44-40aa-8924-1f2b716da33a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-24f923-8474d84f5-kfvdm" Nov 01 20:56:38.268933 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:38.268823 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b594f63-48a1-4a3c-bda5-e52c600c9a41-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-74ab02fe-c6dpl\" (UID: \"7b594f63-48a1-4a3c-bda5-e52c600c9a41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74ab02fe-c6dpl" Nov 01 20:56:38.270483 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:38.270459 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b594f63-48a1-4a3c-bda5-e52c600c9a41-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-74ab02fe-c6dpl\" (UID: \"7b594f63-48a1-4a3c-bda5-e52c600c9a41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74ab02fe-c6dpl" Nov 01 20:56:38.335719 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:38.335681 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgxrx\" (UniqueName: \"kubernetes.io/projected/7b594f63-48a1-4a3c-bda5-e52c600c9a41-kube-api-access-wgxrx\") pod \"maas-api-revoke-keys-e2e-ait-74ab02fe-c6dpl\" (UID: \"7b594f63-48a1-4a3c-bda5-e52c600c9a41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74ab02fe-c6dpl" Nov 01 20:56:38.336338 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:38.336315 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g779\" (UniqueName: \"kubernetes.io/projected/f9a691db-7b44-40aa-8924-1f2b716da33a-kube-api-access-7g779\") pod \"payload-processing-e2e-shared-a-w3-24f923-8474d84f5-kfvdm\" (UID: \"f9a691db-7b44-40aa-8924-1f2b716da33a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-24f923-8474d84f5-kfvdm" Nov 01 20:56:38.480250 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:38.480162 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-24f923-8474d84f5-kfvdm" Nov 01 20:56:38.489651 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:38.489234 4484 patch_prober.go:29] interesting pod/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 01 20:56:38.489651 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:38.489294 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h" podUID="3b91fbd7-5fe8-42de-b6be-fcbf67244747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 01 20:56:38.533795 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:56:38.533764 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod009125b3_fc6d_4cbe_aa45_5cf27cc39a2e.slice/crio-72ef89d70a98b50e49304debe5b476e6227c669559fd81d52f2488c28ed4431c WatchSource:0}: Error finding container 72ef89d70a98b50e49304debe5b476e6227c669559fd81d52f2488c28ed4431c: Status 404 returned error can't find the container with id 72ef89d70a98b50e49304debe5b476e6227c669559fd81d52f2488c28ed4431c Nov 01 20:56:38.536913 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:38.536809 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-24f923-869bcbd967-w25fl"] Nov 01 20:56:38.615048 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:38.614297 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74ab02fe-c6dpl" Nov 01 20:56:38.896441 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:56:38.896412 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9a691db_7b44_40aa_8924_1f2b716da33a.slice/crio-f6d06be7602d1f86dea743402725550b7c07375cc430f1abaf828c859de67a4e WatchSource:0}: Error finding container f6d06be7602d1f86dea743402725550b7c07375cc430f1abaf828c859de67a4e: Status 404 returned error can't find the container with id f6d06be7602d1f86dea743402725550b7c07375cc430f1abaf828c859de67a4e Nov 01 20:56:38.897930 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:38.897902 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-24f923-8474d84f5-kfvdm"] Nov 01 20:56:39.009711 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:39.009684 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2" Nov 01 20:56:39.010529 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:39.010156 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2" Nov 01 20:56:39.010921 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:39.010902 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2" Nov 01 20:56:39.012161 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:39.012137 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-24f923-8474d84f5-kfvdm" event={"ID":"f9a691db-7b44-40aa-8924-1f2b716da33a","Type":"ContainerStarted","Data":"f6d06be7602d1f86dea743402725550b7c07375cc430f1abaf828c859de67a4e"} Nov 01 20:56:39.012255 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:39.012223 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-24f923-8474d84f5-kfvdm" Nov 01 20:56:39.014046 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:39.014026 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-24f923-869bcbd967-w25fl" event={"ID":"009125b3-fc6d-4cbe-aa45-5cf27cc39a2e","Type":"ContainerStarted","Data":"2d28d03c26fd16a454b42acc07c5e6a8621b22a7cc1389d8f64e20483b732afd"} Nov 01 20:56:39.014134 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:39.014068 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-24f923-869bcbd967-w25fl" event={"ID":"009125b3-fc6d-4cbe-aa45-5cf27cc39a2e","Type":"ContainerStarted","Data":"72ef89d70a98b50e49304debe5b476e6227c669559fd81d52f2488c28ed4431c"} Nov 01 20:56:39.014134 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:39.014120 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-24f923-869bcbd967-w25fl" Nov 01 20:56:39.037033 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:56:39.033222 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b594f63_48a1_4a3c_bda5_e52c600c9a41.slice/crio-88bca8f9e2326f30f47b4c23f6435729ba367130140173beb7a25345da30d3cc WatchSource:0}: Error finding container 88bca8f9e2326f30f47b4c23f6435729ba367130140173beb7a25345da30d3cc: Status 404 returned error can't find the container with id 88bca8f9e2326f30f47b4c23f6435729ba367130140173beb7a25345da30d3cc Nov 01 20:56:39.037033 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:39.035961 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74ab02fe-c6dpl"] Nov 01 20:56:39.053739 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:39.053715 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fcc929e-fgr7c"] Nov 01 20:56:39.063082 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:39.061595 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fcc929e-fgr7c"] Nov 01 20:56:39.377535 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:39.377394 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-24f923-8474d84f5-kfvdm" podStartSLOduration=2.377371047 podStartE2EDuration="2.377371047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:56:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:56:39.375827287 +0000 UTC m=+1550.153467423" watchObservedRunningTime="2026-11-01 20:56:39.377371047 +0000 UTC m=+1550.155011184" Nov 01 20:56:39.387279 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:39.387252 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="146ef2d2-52b4-43ea-8d0b-c9eac81f4ff5" path="/var/lib/kubelet/pods/146ef2d2-52b4-43ea-8d0b-c9eac81f4ff5/volumes" Nov 01 20:56:39.537518 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:39.537420 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-24f923-869bcbd967-w25fl" podStartSLOduration=2.537397968 podStartE2EDuration="2.537397968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:56:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:56:39.490552176 +0000 UTC m=+1550.268192311" watchObservedRunningTime="2026-11-01 20:56:39.537397968 +0000 UTC m=+1550.315038104" Nov 01 20:56:39.550696 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:39.550652 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h" Nov 01 20:56:39.552215 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:39.552188 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h" Nov 01 20:56:39.555534 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:39.555507 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h" Nov 01 20:56:40.024850 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:40.024814 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74ab02fe-c6dpl" event={"ID":"7b594f63-48a1-4a3c-bda5-e52c600c9a41","Type":"ContainerStarted","Data":"639cf15de3f98e5c2a3688d9c3baf1b0cfeb3ae0f5fc6bb4e427b90279d80954"} Nov 01 20:56:40.025045 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:40.024878 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74ab02fe-c6dpl" event={"ID":"7b594f63-48a1-4a3c-bda5-e52c600c9a41","Type":"ContainerStarted","Data":"88bca8f9e2326f30f47b4c23f6435729ba367130140173beb7a25345da30d3cc"} Nov 01 20:56:40.026729 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:40.026700 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-24f923-8474d84f5-kfvdm" event={"ID":"f9a691db-7b44-40aa-8924-1f2b716da33a","Type":"ContainerStarted","Data":"00b3d54099fc3fe310e330d47243bf6cac95017600cf9dc8f2c19695cd428dd4"} Nov 01 20:56:40.051100 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:40.051070 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e407a6a4-55qlv"] Nov 01 20:56:40.055700 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:40.055673 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e407a6a4-55qlv"] Nov 01 20:56:40.140893 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:40.140836 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74ab02fe-c6dpl" podStartSLOduration=3.140817767 podStartE2EDuration="3.140817767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:56:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:56:40.139531859 +0000 UTC m=+1550.917171993" watchObservedRunningTime="2026-11-01 20:56:40.140817767 +0000 UTC m=+1550.918457906" Nov 01 20:56:40.264752 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:40.264716 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8ddfb06d-54d7b96ff7-flgn4"] Nov 01 20:56:40.397097 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:40.397015 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8ddfb06d-66db94bbb7-2fxht"] Nov 01 20:56:40.657821 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:40.657737 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:56:40.826021 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:40.825758 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:56:41.059127 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:41.059087 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70bcb6f1-fbph2"] Nov 01 20:56:41.064571 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:41.064536 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70bcb6f1-fbph2"] Nov 01 20:56:41.076438 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:41.066084 4484 generic.go:356] "Generic (PLEG): container finished" podID="7b594f63-48a1-4a3c-bda5-e52c600c9a41" containerID="639cf15de3f98e5c2a3688d9c3baf1b0cfeb3ae0f5fc6bb4e427b90279d80954" exitCode=0 Nov 01 20:56:41.076438 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:41.066248 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74ab02fe-c6dpl" event={"ID":"7b594f63-48a1-4a3c-bda5-e52c600c9a41","Type":"ContainerDied","Data":"639cf15de3f98e5c2a3688d9c3baf1b0cfeb3ae0f5fc6bb4e427b90279d80954"} Nov 01 20:56:41.374504 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:41.374425 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95b4299f-f0db-4901-b902-9acc154edc3f" path="/var/lib/kubelet/pods/95b4299f-f0db-4901-b902-9acc154edc3f/volumes" Nov 01 20:56:41.374971 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:41.374957 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2e9e863-ca7f-4f6f-9943-557293100f33" path="/var/lib/kubelet/pods/c2e9e863-ca7f-4f6f-9943-557293100f33/volumes" Nov 01 20:56:41.598311 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:41.598278 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:56:42.700111 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:42.700081 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74ab02fe-c6dpl" Nov 01 20:56:42.758370 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:42.758339 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b594f63-48a1-4a3c-bda5-e52c600c9a41-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b594f63-48a1-4a3c-bda5-e52c600c9a41-maas-api-xks-ca\") pod \"7b594f63-48a1-4a3c-bda5-e52c600c9a41\" (UID: \"7b594f63-48a1-4a3c-bda5-e52c600c9a41\") " Nov 01 20:56:42.758540 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:42.758389 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b594f63-48a1-4a3c-bda5-e52c600c9a41-kube-api-access-wgxrx\" (UniqueName: \"kubernetes.io/projected/7b594f63-48a1-4a3c-bda5-e52c600c9a41-kube-api-access-wgxrx\") pod \"7b594f63-48a1-4a3c-bda5-e52c600c9a41\" (UID: \"7b594f63-48a1-4a3c-bda5-e52c600c9a41\") " Nov 01 20:56:42.758540 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:42.758508 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b594f63-48a1-4a3c-bda5-e52c600c9a41-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b594f63-48a1-4a3c-bda5-e52c600c9a41-maas-api-openshift-ca\") pod \"7b594f63-48a1-4a3c-bda5-e52c600c9a41\" (UID: \"7b594f63-48a1-4a3c-bda5-e52c600c9a41\") " Nov 01 20:56:42.758904 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:42.758879 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b594f63-48a1-4a3c-bda5-e52c600c9a41-maas-api-openshift-ca" pod "7b594f63-48a1-4a3c-bda5-e52c600c9a41" (UID: "7b594f63-48a1-4a3c-bda5-e52c600c9a41"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:56:42.759447 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:42.759426 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b594f63-48a1-4a3c-bda5-e52c600c9a41-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:56:42.760542 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:42.760519 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b594f63-48a1-4a3c-bda5-e52c600c9a41-maas-api-xks-ca" pod "7b594f63-48a1-4a3c-bda5-e52c600c9a41" (UID: "7b594f63-48a1-4a3c-bda5-e52c600c9a41"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:56:42.760620 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:42.760543 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b594f63-48a1-4a3c-bda5-e52c600c9a41-kube-api-access-wgxrx" pod "7b594f63-48a1-4a3c-bda5-e52c600c9a41" (UID: "7b594f63-48a1-4a3c-bda5-e52c600c9a41"). InnerVolumeSpecName "kube-api-access-wgxrx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:56:42.860916 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:42.860880 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b594f63-48a1-4a3c-bda5-e52c600c9a41-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:56:42.860916 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:42.860911 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgxrx\" (UniqueName: \"kubernetes.io/projected/7b594f63-48a1-4a3c-bda5-e52c600c9a41-kube-api-access-wgxrx\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:56:43.163192 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:43.163161 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74ab02fe-c6dpl" event={"ID":"7b594f63-48a1-4a3c-bda5-e52c600c9a41","Type":"ContainerDied","Data":"88bca8f9e2326f30f47b4c23f6435729ba367130140173beb7a25345da30d3cc"} Nov 01 20:56:43.163192 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:43.163195 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74ab02fe-c6dpl" Nov 01 20:56:43.163412 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:43.163215 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88bca8f9e2326f30f47b4c23f6435729ba367130140173beb7a25345da30d3cc" Nov 01 20:56:43.345357 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:43.345319 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:56:44.049008 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:44.048972 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53208b70-czwcx"] Nov 01 20:56:44.099476 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:44.099444 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53208b70-czwcx"] Nov 01 20:56:45.265036 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:45.264982 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-8ddfb06d-54d7b96ff7-flgn4" podUID="497bee83-44f6-4eee-88c9-e62f051c8a28" containerName="payload-processing" containerID="cri-o://18fa80bcf8ab16d987952008ece3f248a6c76c04874936346b60c65504950ac2" gracePeriod=25 Nov 01 20:56:45.375663 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:45.375631 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ca4d667-38b7-4e87-9f63-cf66f78ca2bd" path="/var/lib/kubelet/pods/2ca4d667-38b7-4e87-9f63-cf66f78ca2bd/volumes" Nov 01 20:56:45.398204 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:45.398170 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8ddfb06d-66db94bbb7-2fxht" podUID="15651aae-6296-488c-b5b3-fb525dbb5a68" containerName="payload-pre-processing" containerID="cri-o://2c760fa0c73e46ca90c0b02f93b85562a900185aa05a3f8d46979205413c30c0" gracePeriod=25 Nov 01 20:56:45.581142 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:45.581117 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-8ddfb06d-54d7b96ff7-flgn4" Nov 01 20:56:45.702690 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:45.702658 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8ddfb06d-66db94bbb7-2fxht" Nov 01 20:56:45.729184 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:45.729145 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/497bee83-44f6-4eee-88c9-e62f051c8a28-kube-api-access-t5vss\" (UniqueName: \"kubernetes.io/projected/497bee83-44f6-4eee-88c9-e62f051c8a28-kube-api-access-t5vss\") pod \"497bee83-44f6-4eee-88c9-e62f051c8a28\" (UID: \"497bee83-44f6-4eee-88c9-e62f051c8a28\") " Nov 01 20:56:45.729384 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:45.729213 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/497bee83-44f6-4eee-88c9-e62f051c8a28-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/497bee83-44f6-4eee-88c9-e62f051c8a28-plugins-config-volume\") pod \"497bee83-44f6-4eee-88c9-e62f051c8a28\" (UID: \"497bee83-44f6-4eee-88c9-e62f051c8a28\") " Nov 01 20:56:45.729658 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:45.729624 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/497bee83-44f6-4eee-88c9-e62f051c8a28-plugins-config-volume" pod "497bee83-44f6-4eee-88c9-e62f051c8a28" (UID: "497bee83-44f6-4eee-88c9-e62f051c8a28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:56:45.730352 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:45.730323 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/497bee83-44f6-4eee-88c9-e62f051c8a28-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:56:45.731706 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:45.731677 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/497bee83-44f6-4eee-88c9-e62f051c8a28-kube-api-access-t5vss" pod "497bee83-44f6-4eee-88c9-e62f051c8a28" (UID: "497bee83-44f6-4eee-88c9-e62f051c8a28"). InnerVolumeSpecName "kube-api-access-t5vss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:56:45.768920 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:56:45.768885 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 20:56:45.831851 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:45.831779 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15651aae-6296-488c-b5b3-fb525dbb5a68-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15651aae-6296-488c-b5b3-fb525dbb5a68-plugins-config-volume\") pod \"15651aae-6296-488c-b5b3-fb525dbb5a68\" (UID: \"15651aae-6296-488c-b5b3-fb525dbb5a68\") " Nov 01 20:56:45.831851 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:45.831823 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15651aae-6296-488c-b5b3-fb525dbb5a68-kube-api-access-rwzxg\" (UniqueName: \"kubernetes.io/projected/15651aae-6296-488c-b5b3-fb525dbb5a68-kube-api-access-rwzxg\") pod \"15651aae-6296-488c-b5b3-fb525dbb5a68\" (UID: \"15651aae-6296-488c-b5b3-fb525dbb5a68\") " Nov 01 20:56:45.832274 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:45.832246 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15651aae-6296-488c-b5b3-fb525dbb5a68-plugins-config-volume" pod "15651aae-6296-488c-b5b3-fb525dbb5a68" (UID: "15651aae-6296-488c-b5b3-fb525dbb5a68"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:56:45.832927 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:45.832899 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5vss\" (UniqueName: \"kubernetes.io/projected/497bee83-44f6-4eee-88c9-e62f051c8a28-kube-api-access-t5vss\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:56:45.833096 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:45.832930 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15651aae-6296-488c-b5b3-fb525dbb5a68-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:56:45.834334 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:45.834291 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15651aae-6296-488c-b5b3-fb525dbb5a68-kube-api-access-rwzxg" pod "15651aae-6296-488c-b5b3-fb525dbb5a68" (UID: "15651aae-6296-488c-b5b3-fb525dbb5a68"). InnerVolumeSpecName "kube-api-access-rwzxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:56:45.934517 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:45.934477 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwzxg\" (UniqueName: \"kubernetes.io/projected/15651aae-6296-488c-b5b3-fb525dbb5a68-kube-api-access-rwzxg\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:56:46.192333 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:46.192249 4484 generic.go:356] "Generic (PLEG): container finished" podID="497bee83-44f6-4eee-88c9-e62f051c8a28" containerID="18fa80bcf8ab16d987952008ece3f248a6c76c04874936346b60c65504950ac2" exitCode=0 Nov 01 20:56:46.192333 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:46.192301 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-8ddfb06d-54d7b96ff7-flgn4" Nov 01 20:56:46.192572 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:46.192336 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-8ddfb06d-54d7b96ff7-flgn4" event={"ID":"497bee83-44f6-4eee-88c9-e62f051c8a28","Type":"ContainerDied","Data":"18fa80bcf8ab16d987952008ece3f248a6c76c04874936346b60c65504950ac2"} Nov 01 20:56:46.192572 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:46.192386 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-8ddfb06d-54d7b96ff7-flgn4" event={"ID":"497bee83-44f6-4eee-88c9-e62f051c8a28","Type":"ContainerDied","Data":"21edefba1902b8d8bbfcc655ef403635ee9e3a7ab1ef41ddfe27ae4e06f7be1f"} Nov 01 20:56:46.192572 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:46.192413 4484 scope.go:122] "RemoveContainer" containerID="18fa80bcf8ab16d987952008ece3f248a6c76c04874936346b60c65504950ac2" Nov 01 20:56:46.193956 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:46.193934 4484 generic.go:356] "Generic (PLEG): container finished" podID="15651aae-6296-488c-b5b3-fb525dbb5a68" containerID="2c760fa0c73e46ca90c0b02f93b85562a900185aa05a3f8d46979205413c30c0" exitCode=0 Nov 01 20:56:46.194079 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:46.193986 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8ddfb06d-66db94bbb7-2fxht" Nov 01 20:56:46.194079 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:46.194036 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8ddfb06d-66db94bbb7-2fxht" event={"ID":"15651aae-6296-488c-b5b3-fb525dbb5a68","Type":"ContainerDied","Data":"2c760fa0c73e46ca90c0b02f93b85562a900185aa05a3f8d46979205413c30c0"} Nov 01 20:56:46.194079 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:46.194077 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8ddfb06d-66db94bbb7-2fxht" event={"ID":"15651aae-6296-488c-b5b3-fb525dbb5a68","Type":"ContainerDied","Data":"0306a97af5a97990311c95f0dc3ff2047c296fea076e0614884093c9a62cd515"} Nov 01 20:56:46.206476 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:46.206458 4484 scope.go:122] "RemoveContainer" containerID="18fa80bcf8ab16d987952008ece3f248a6c76c04874936346b60c65504950ac2" Nov 01 20:56:46.206802 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:56:46.206781 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18fa80bcf8ab16d987952008ece3f248a6c76c04874936346b60c65504950ac2\": container with ID starting with 18fa80bcf8ab16d987952008ece3f248a6c76c04874936346b60c65504950ac2 not found: ID does not exist" containerID="18fa80bcf8ab16d987952008ece3f248a6c76c04874936346b60c65504950ac2" Nov 01 20:56:46.206876 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:46.206810 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18fa80bcf8ab16d987952008ece3f248a6c76c04874936346b60c65504950ac2"} err="failed to get container status \"18fa80bcf8ab16d987952008ece3f248a6c76c04874936346b60c65504950ac2\": rpc error: code = NotFound desc = could not find container \"18fa80bcf8ab16d987952008ece3f248a6c76c04874936346b60c65504950ac2\": container with ID starting with 18fa80bcf8ab16d987952008ece3f248a6c76c04874936346b60c65504950ac2 not found: ID does not exist" Nov 01 20:56:46.206876 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:46.206829 4484 scope.go:122] "RemoveContainer" containerID="2c760fa0c73e46ca90c0b02f93b85562a900185aa05a3f8d46979205413c30c0" Nov 01 20:56:46.219086 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:46.219068 4484 scope.go:122] "RemoveContainer" containerID="2c760fa0c73e46ca90c0b02f93b85562a900185aa05a3f8d46979205413c30c0" Nov 01 20:56:46.219347 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:56:46.219329 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c760fa0c73e46ca90c0b02f93b85562a900185aa05a3f8d46979205413c30c0\": container with ID starting with 2c760fa0c73e46ca90c0b02f93b85562a900185aa05a3f8d46979205413c30c0 not found: ID does not exist" containerID="2c760fa0c73e46ca90c0b02f93b85562a900185aa05a3f8d46979205413c30c0" Nov 01 20:56:46.219401 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:46.219355 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c760fa0c73e46ca90c0b02f93b85562a900185aa05a3f8d46979205413c30c0"} err="failed to get container status \"2c760fa0c73e46ca90c0b02f93b85562a900185aa05a3f8d46979205413c30c0\": rpc error: code = NotFound desc = could not find container \"2c760fa0c73e46ca90c0b02f93b85562a900185aa05a3f8d46979205413c30c0\": container with ID starting with 2c760fa0c73e46ca90c0b02f93b85562a900185aa05a3f8d46979205413c30c0 not found: ID does not exist" Nov 01 20:56:46.361194 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:46.361160 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8ddfb06d-66db94bbb7-2fxht"] Nov 01 20:56:46.411894 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:46.411862 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8ddfb06d-66db94bbb7-2fxht"] Nov 01 20:56:46.474893 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:46.474809 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8ddfb06d-54d7b96ff7-flgn4"] Nov 01 20:56:46.479602 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:46.479578 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8ddfb06d-54d7b96ff7-flgn4"] Nov 01 20:56:47.047986 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:47.047951 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b58f354-bczft"] Nov 01 20:56:47.052661 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:47.052631 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b58f354-bczft"] Nov 01 20:56:47.376354 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:47.376266 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15651aae-6296-488c-b5b3-fb525dbb5a68" path="/var/lib/kubelet/pods/15651aae-6296-488c-b5b3-fb525dbb5a68/volumes" Nov 01 20:56:47.376865 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:47.376846 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="497bee83-44f6-4eee-88c9-e62f051c8a28" path="/var/lib/kubelet/pods/497bee83-44f6-4eee-88c9-e62f051c8a28/volumes" Nov 01 20:56:47.377395 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:47.377373 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76bbc184-657e-4bf3-a072-92d47164ce86" path="/var/lib/kubelet/pods/76bbc184-657e-4bf3-a072-92d47164ce86/volumes" Nov 01 20:56:48.584582 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:48.584548 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:56:48.667308 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:48.651537 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-74ab02fe-7c85d47c89-2fpz8"] Nov 01 20:56:48.746016 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:48.745814 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-74ab02fe-78888c8555-t9shv"] Nov 01 20:56:49.115022 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:49.093721 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:56:49.352017 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:49.343583 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:56:50.077020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:50.076455 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-24f923-869bcbd967-w25fl" Nov 01 20:56:51.198481 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:51.138689 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" podUID="17a9ac24-19cb-48c4-99b5-d74ce8aac0a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:56:51.203229 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:51.202181 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-24f923-8474d84f5-kfvdm" Nov 01 20:56:53.652605 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:53.652565 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-74ab02fe-7c85d47c89-2fpz8" podUID="7bcd38da-1bd5-4293-9d1b-531907031b36" containerName="payload-processing" containerID="cri-o://ebc20707f8f6c44e9c12ddc51bcf2891caf32d30e455b426093537b537b386a0" gracePeriod=25 Nov 01 20:56:53.747190 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:53.747130 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-74ab02fe-78888c8555-t9shv" podUID="f51e345c-11ff-428a-a715-8e0ec3ff98f8" containerName="payload-pre-processing" containerID="cri-o://183229c1ab5dd344b6f4a1c9ceb59d9e924858f8915be2d47257d289e68b4d8b" gracePeriod=25 Nov 01 20:56:54.047521 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:54.047485 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2607e260-dgmfz"] Nov 01 20:56:54.050205 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:54.050181 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-74ab02fe-7c85d47c89-2fpz8" Nov 01 20:56:54.092781 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:54.092754 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-74ab02fe-78888c8555-t9shv" Nov 01 20:56:54.097571 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:54.097541 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2607e260-dgmfz"] Nov 01 20:56:54.135365 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:54.135324 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:56:54.150121 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:54.150092 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f51e345c-11ff-428a-a715-8e0ec3ff98f8-kube-api-access-g5ld9\" (UniqueName: \"kubernetes.io/projected/f51e345c-11ff-428a-a715-8e0ec3ff98f8-kube-api-access-g5ld9\") pod \"f51e345c-11ff-428a-a715-8e0ec3ff98f8\" (UID: \"f51e345c-11ff-428a-a715-8e0ec3ff98f8\") " Nov 01 20:56:54.150270 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:54.150255 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bcd38da-1bd5-4293-9d1b-531907031b36-kube-api-access-2sp89\" (UniqueName: \"kubernetes.io/projected/7bcd38da-1bd5-4293-9d1b-531907031b36-kube-api-access-2sp89\") pod \"7bcd38da-1bd5-4293-9d1b-531907031b36\" (UID: \"7bcd38da-1bd5-4293-9d1b-531907031b36\") " Nov 01 20:56:54.150373 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:54.150360 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f51e345c-11ff-428a-a715-8e0ec3ff98f8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f51e345c-11ff-428a-a715-8e0ec3ff98f8-plugins-config-volume\") pod \"f51e345c-11ff-428a-a715-8e0ec3ff98f8\" (UID: \"f51e345c-11ff-428a-a715-8e0ec3ff98f8\") " Nov 01 20:56:54.150434 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:54.150423 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bcd38da-1bd5-4293-9d1b-531907031b36-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bcd38da-1bd5-4293-9d1b-531907031b36-plugins-config-volume\") pod \"7bcd38da-1bd5-4293-9d1b-531907031b36\" (UID: \"7bcd38da-1bd5-4293-9d1b-531907031b36\") " Nov 01 20:56:54.150833 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:54.150797 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f51e345c-11ff-428a-a715-8e0ec3ff98f8-plugins-config-volume" pod "f51e345c-11ff-428a-a715-8e0ec3ff98f8" (UID: "f51e345c-11ff-428a-a715-8e0ec3ff98f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:56:54.150960 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:54.150838 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bcd38da-1bd5-4293-9d1b-531907031b36-plugins-config-volume" pod "7bcd38da-1bd5-4293-9d1b-531907031b36" (UID: "7bcd38da-1bd5-4293-9d1b-531907031b36"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:56:54.151309 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:54.151290 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f51e345c-11ff-428a-a715-8e0ec3ff98f8-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:56:54.151309 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:54.151309 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bcd38da-1bd5-4293-9d1b-531907031b36-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:56:54.152276 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:54.152253 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bcd38da-1bd5-4293-9d1b-531907031b36-kube-api-access-2sp89" pod "7bcd38da-1bd5-4293-9d1b-531907031b36" (UID: "7bcd38da-1bd5-4293-9d1b-531907031b36"). InnerVolumeSpecName "kube-api-access-2sp89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:56:54.152371 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:54.152346 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f51e345c-11ff-428a-a715-8e0ec3ff98f8-kube-api-access-g5ld9" pod "f51e345c-11ff-428a-a715-8e0ec3ff98f8" (UID: "f51e345c-11ff-428a-a715-8e0ec3ff98f8"). InnerVolumeSpecName "kube-api-access-g5ld9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:56:54.309073 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:54.301549 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5ld9\" (UniqueName: \"kubernetes.io/projected/f51e345c-11ff-428a-a715-8e0ec3ff98f8-kube-api-access-g5ld9\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:56:54.309073 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:54.301584 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sp89\" (UniqueName: \"kubernetes.io/projected/7bcd38da-1bd5-4293-9d1b-531907031b36-kube-api-access-2sp89\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:56:54.357813 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:54.357779 4484 generic.go:356] "Generic (PLEG): container finished" podID="f51e345c-11ff-428a-a715-8e0ec3ff98f8" containerID="183229c1ab5dd344b6f4a1c9ceb59d9e924858f8915be2d47257d289e68b4d8b" exitCode=0 Nov 01 20:56:54.358143 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:54.358121 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-74ab02fe-78888c8555-t9shv" event={"ID":"f51e345c-11ff-428a-a715-8e0ec3ff98f8","Type":"ContainerDied","Data":"183229c1ab5dd344b6f4a1c9ceb59d9e924858f8915be2d47257d289e68b4d8b"} Nov 01 20:56:54.358202 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:54.358178 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-74ab02fe-78888c8555-t9shv" event={"ID":"f51e345c-11ff-428a-a715-8e0ec3ff98f8","Type":"ContainerDied","Data":"08d5366797c9568e444effa964ef81d6988bf003fe59c83e3e93b1c6095715da"} Nov 01 20:56:54.358240 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:54.358206 4484 scope.go:122] "RemoveContainer" containerID="183229c1ab5dd344b6f4a1c9ceb59d9e924858f8915be2d47257d289e68b4d8b" Nov 01 20:56:54.358382 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:54.358369 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-74ab02fe-78888c8555-t9shv" Nov 01 20:56:54.365784 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:54.365757 4484 generic.go:356] "Generic (PLEG): container finished" podID="7bcd38da-1bd5-4293-9d1b-531907031b36" containerID="ebc20707f8f6c44e9c12ddc51bcf2891caf32d30e455b426093537b537b386a0" exitCode=0 Nov 01 20:56:54.365929 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:54.365808 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-74ab02fe-7c85d47c89-2fpz8" Nov 01 20:56:54.365929 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:54.365828 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-74ab02fe-7c85d47c89-2fpz8" event={"ID":"7bcd38da-1bd5-4293-9d1b-531907031b36","Type":"ContainerDied","Data":"ebc20707f8f6c44e9c12ddc51bcf2891caf32d30e455b426093537b537b386a0"} Nov 01 20:56:54.365929 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:54.365868 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-74ab02fe-7c85d47c89-2fpz8" event={"ID":"7bcd38da-1bd5-4293-9d1b-531907031b36","Type":"ContainerDied","Data":"51c9e0ef7c0f15622f78fb21edc61b033e0a7e102327aeca94ae5f70dab8cc0b"} Nov 01 20:56:54.382103 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:54.382081 4484 scope.go:122] "RemoveContainer" containerID="183229c1ab5dd344b6f4a1c9ceb59d9e924858f8915be2d47257d289e68b4d8b" Nov 01 20:56:54.382461 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:56:54.382432 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"183229c1ab5dd344b6f4a1c9ceb59d9e924858f8915be2d47257d289e68b4d8b\": container with ID starting with 183229c1ab5dd344b6f4a1c9ceb59d9e924858f8915be2d47257d289e68b4d8b not found: ID does not exist" containerID="183229c1ab5dd344b6f4a1c9ceb59d9e924858f8915be2d47257d289e68b4d8b" Nov 01 20:56:54.382582 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:54.382473 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"183229c1ab5dd344b6f4a1c9ceb59d9e924858f8915be2d47257d289e68b4d8b"} err="failed to get container status \"183229c1ab5dd344b6f4a1c9ceb59d9e924858f8915be2d47257d289e68b4d8b\": rpc error: code = NotFound desc = could not find container \"183229c1ab5dd344b6f4a1c9ceb59d9e924858f8915be2d47257d289e68b4d8b\": container with ID starting with 183229c1ab5dd344b6f4a1c9ceb59d9e924858f8915be2d47257d289e68b4d8b not found: ID does not exist" Nov 01 20:56:54.382582 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:54.382499 4484 scope.go:122] "RemoveContainer" containerID="ebc20707f8f6c44e9c12ddc51bcf2891caf32d30e455b426093537b537b386a0" Nov 01 20:56:54.395349 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:54.395331 4484 scope.go:122] "RemoveContainer" containerID="ebc20707f8f6c44e9c12ddc51bcf2891caf32d30e455b426093537b537b386a0" Nov 01 20:56:54.395721 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:56:54.395685 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebc20707f8f6c44e9c12ddc51bcf2891caf32d30e455b426093537b537b386a0\": container with ID starting with ebc20707f8f6c44e9c12ddc51bcf2891caf32d30e455b426093537b537b386a0 not found: ID does not exist" containerID="ebc20707f8f6c44e9c12ddc51bcf2891caf32d30e455b426093537b537b386a0" Nov 01 20:56:54.395802 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:54.395735 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebc20707f8f6c44e9c12ddc51bcf2891caf32d30e455b426093537b537b386a0"} err="failed to get container status \"ebc20707f8f6c44e9c12ddc51bcf2891caf32d30e455b426093537b537b386a0\": rpc error: code = NotFound desc = could not find container \"ebc20707f8f6c44e9c12ddc51bcf2891caf32d30e455b426093537b537b386a0\": container with ID starting with ebc20707f8f6c44e9c12ddc51bcf2891caf32d30e455b426093537b537b386a0 not found: ID does not exist" Nov 01 20:56:54.537849 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:54.537805 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-74ab02fe-7c85d47c89-2fpz8"] Nov 01 20:56:54.542858 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:54.542813 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-74ab02fe-7c85d47c89-2fpz8"] Nov 01 20:56:54.645030 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:54.644926 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-74ab02fe-78888c8555-t9shv"] Nov 01 20:56:54.649373 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:54.649345 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-74ab02fe-78888c8555-t9shv"] Nov 01 20:56:55.374736 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:55.374704 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="004f0b18-2fdc-443f-9b43-91b998037058" path="/var/lib/kubelet/pods/004f0b18-2fdc-443f-9b43-91b998037058/volumes" Nov 01 20:56:55.375304 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:55.375285 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bcd38da-1bd5-4293-9d1b-531907031b36" path="/var/lib/kubelet/pods/7bcd38da-1bd5-4293-9d1b-531907031b36/volumes" Nov 01 20:56:55.375873 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:55.375856 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f51e345c-11ff-428a-a715-8e0ec3ff98f8" path="/var/lib/kubelet/pods/f51e345c-11ff-428a-a715-8e0ec3ff98f8/volumes" Nov 01 20:56:58.774851 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:58.774813 4484 scope.go:122] "RemoveContainer" containerID="9d4acc7b681b2f003986d233869841a14bafd33d8707a5a533c9401b0b9a6a1e" Nov 01 20:56:58.790237 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:58.790208 4484 scope.go:122] "RemoveContainer" containerID="67843f2db6ee32ab4120445cbaa8956485ffd8066102947766c63a9cc268dc20" Nov 01 20:56:58.801190 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:58.801165 4484 scope.go:122] "RemoveContainer" containerID="cafe933880332ee2ab7d9ff0d1687c4171d93ef8d92e4920fb2f7a62cedaaf75" Nov 01 20:56:58.812200 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:58.812179 4484 scope.go:122] "RemoveContainer" containerID="153f0ce574bb8ab8efda1f57031712ec0329d1c04da9284dc17e55a179500ca1" Nov 01 20:56:58.823232 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:58.823208 4484 scope.go:122] "RemoveContainer" containerID="03b251dbb3d8691f8b72475b7c30d1f2c8851c87a1835878d0a0a93e27434956" Nov 01 20:56:58.832171 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:58.832140 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:56:58.834283 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:58.834260 4484 scope.go:122] "RemoveContainer" containerID="243056b64df75f2a68af1fbcd9216366c182dad3c09fa6f3eb8426326c34239a" Nov 01 20:56:58.845140 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:58.845120 4484 scope.go:122] "RemoveContainer" containerID="c45e275f9e2c27873a964593824d5ebb931a73f5ab5ccfc134329194767b79e1" Nov 01 20:56:58.899286 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:58.888942 4484 scope.go:122] "RemoveContainer" containerID="cc82af2585ec718e50f15c8a0dede339daaa04577714d7ab0e3896c054fc2cfc" Nov 01 20:56:58.952019 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:58.944337 4484 scope.go:122] "RemoveContainer" containerID="14e3daeb011e99bda8862b4b26336c48301a0ab44e8266039e90cb6f34da8705" Nov 01 20:56:58.986244 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:58.986169 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:56:59.156073 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:56:59.154806 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:01.390288 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:01.390210 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:02.068045 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:02.066067 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e42be329-mvc5r"] Nov 01 20:57:02.119016 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:02.116231 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e42be329-mvc5r"] Nov 01 20:57:02.208717 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:02.208633 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:02.389426 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:02.389320 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:02.815288 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:02.815249 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:03.377958 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:03.377921 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae5dd0f4-7d3f-47c7-9510-c12d528db8bc" path="/var/lib/kubelet/pods/ae5dd0f4-7d3f-47c7-9510-c12d528db8bc/volumes" Nov 01 20:57:04.082571 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:04.082531 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:08.756648 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:08.756607 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:08.921824 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:08.921789 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:09.056358 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:09.056267 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d6e880b-vjrfs"] Nov 01 20:57:09.086692 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:09.086653 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:09.119052 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:09.116571 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d6e880b-vjrfs"] Nov 01 20:57:09.384792 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:09.384709 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de777fdf-8903-433f-b82e-46ed1771bc1a" path="/var/lib/kubelet/pods/de777fdf-8903-433f-b82e-46ed1771bc1a/volumes" Nov 01 20:57:09.576471 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:09.576431 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a5ab4417-jx9w5"] Nov 01 20:57:09.577246 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:09.577229 4484 cpu_manager.go:414] "containerMap: removing container" podUID="7b594f63-48a1-4a3c-bda5-e52c600c9a41" containerName="revoke-keys" Nov 01 20:57:09.577323 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:09.577247 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b594f63-48a1-4a3c-bda5-e52c600c9a41" containerName="revoke-keys" Nov 01 20:57:09.577323 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:09.577287 4484 cpu_manager.go:414] "containerMap: removing container" podUID="f51e345c-11ff-428a-a715-8e0ec3ff98f8" containerName="payload-pre-processing" Nov 01 20:57:09.577323 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:09.577292 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f51e345c-11ff-428a-a715-8e0ec3ff98f8" containerName="payload-pre-processing" Nov 01 20:57:09.577323 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:09.577311 4484 cpu_manager.go:414] "containerMap: removing container" podUID="497bee83-44f6-4eee-88c9-e62f051c8a28" containerName="payload-processing" Nov 01 20:57:09.577323 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:09.577316 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="497bee83-44f6-4eee-88c9-e62f051c8a28" containerName="payload-processing" Nov 01 20:57:09.577480 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:09.577328 4484 cpu_manager.go:414] "containerMap: removing container" podUID="7bcd38da-1bd5-4293-9d1b-531907031b36" containerName="payload-processing" Nov 01 20:57:09.577480 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:09.577333 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bcd38da-1bd5-4293-9d1b-531907031b36" containerName="payload-processing" Nov 01 20:57:09.577480 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:09.577343 4484 cpu_manager.go:414] "containerMap: removing container" podUID="15651aae-6296-488c-b5b3-fb525dbb5a68" containerName="payload-pre-processing" Nov 01 20:57:09.577480 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:09.577348 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15651aae-6296-488c-b5b3-fb525dbb5a68" containerName="payload-pre-processing" Nov 01 20:57:09.577480 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:09.577452 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="f51e345c-11ff-428a-a715-8e0ec3ff98f8" containerName="payload-pre-processing" Nov 01 20:57:09.577480 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:09.577462 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="497bee83-44f6-4eee-88c9-e62f051c8a28" containerName="payload-processing" Nov 01 20:57:09.577480 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:09.577470 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bcd38da-1bd5-4293-9d1b-531907031b36" containerName="payload-processing" Nov 01 20:57:09.577480 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:09.577477 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="15651aae-6296-488c-b5b3-fb525dbb5a68" containerName="payload-pre-processing" Nov 01 20:57:09.577723 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:09.577485 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b594f63-48a1-4a3c-bda5-e52c600c9a41" containerName="revoke-keys" Nov 01 20:57:09.581413 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:09.581395 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a5ab4417-jx9w5" Nov 01 20:57:09.638768 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:09.638675 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a5ab4417-jx9w5"] Nov 01 20:57:09.683211 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:09.683170 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:57:09.683434 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:09.683218 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:57:09.683434 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:09.683240 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:57:09.683434 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:09.683348 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 20:57:09.693599 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:09.693564 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqk6d\" (UniqueName: \"kubernetes.io/projected/2b4a05cb-3c59-420e-9aaf-9d57535fd45b-kube-api-access-gqk6d\") pod \"maas-api-revoke-keys-e2e-mt-a5ab4417-jx9w5\" (UID: \"2b4a05cb-3c59-420e-9aaf-9d57535fd45b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a5ab4417-jx9w5" Nov 01 20:57:09.693882 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:09.693859 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b4a05cb-3c59-420e-9aaf-9d57535fd45b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-a5ab4417-jx9w5\" (UID: \"2b4a05cb-3c59-420e-9aaf-9d57535fd45b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a5ab4417-jx9w5" Nov 01 20:57:09.694128 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:09.694108 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b4a05cb-3c59-420e-9aaf-9d57535fd45b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-a5ab4417-jx9w5\" (UID: \"2b4a05cb-3c59-420e-9aaf-9d57535fd45b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a5ab4417-jx9w5" Nov 01 20:57:09.795886 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:09.795845 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b4a05cb-3c59-420e-9aaf-9d57535fd45b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-a5ab4417-jx9w5\" (UID: \"2b4a05cb-3c59-420e-9aaf-9d57535fd45b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a5ab4417-jx9w5" Nov 01 20:57:09.796311 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:09.795985 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b4a05cb-3c59-420e-9aaf-9d57535fd45b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-a5ab4417-jx9w5\" (UID: \"2b4a05cb-3c59-420e-9aaf-9d57535fd45b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a5ab4417-jx9w5" Nov 01 20:57:09.796311 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:09.796054 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqk6d\" (UniqueName: \"kubernetes.io/projected/2b4a05cb-3c59-420e-9aaf-9d57535fd45b-kube-api-access-gqk6d\") pod \"maas-api-revoke-keys-e2e-mt-a5ab4417-jx9w5\" (UID: \"2b4a05cb-3c59-420e-9aaf-9d57535fd45b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a5ab4417-jx9w5" Nov 01 20:57:09.796668 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:09.796633 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b4a05cb-3c59-420e-9aaf-9d57535fd45b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-a5ab4417-jx9w5\" (UID: \"2b4a05cb-3c59-420e-9aaf-9d57535fd45b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a5ab4417-jx9w5" Nov 01 20:57:09.798405 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:09.798381 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b4a05cb-3c59-420e-9aaf-9d57535fd45b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-a5ab4417-jx9w5\" (UID: \"2b4a05cb-3c59-420e-9aaf-9d57535fd45b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a5ab4417-jx9w5" Nov 01 20:57:09.853700 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:09.853666 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqk6d\" (UniqueName: \"kubernetes.io/projected/2b4a05cb-3c59-420e-9aaf-9d57535fd45b-kube-api-access-gqk6d\") pod \"maas-api-revoke-keys-e2e-mt-a5ab4417-jx9w5\" (UID: \"2b4a05cb-3c59-420e-9aaf-9d57535fd45b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a5ab4417-jx9w5" Nov 01 20:57:09.894623 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:09.894548 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a5ab4417-jx9w5" Nov 01 20:57:10.277303 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:57:10.277259 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b4a05cb_3c59_420e_9aaf_9d57535fd45b.slice/crio-9ce6f9117a19830f3f59f46a637f72b666535598e3eb3b33e24c9b32938456b2 WatchSource:0}: Error finding container 9ce6f9117a19830f3f59f46a637f72b666535598e3eb3b33e24c9b32938456b2: Status 404 returned error can't find the container with id 9ce6f9117a19830f3f59f46a637f72b666535598e3eb3b33e24c9b32938456b2 Nov 01 20:57:10.278443 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:10.278398 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a5ab4417-jx9w5"] Nov 01 20:57:10.576451 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:10.576414 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a5ab4417-jx9w5" event={"ID":"2b4a05cb-3c59-420e-9aaf-9d57535fd45b","Type":"ContainerStarted","Data":"9ce6f9117a19830f3f59f46a637f72b666535598e3eb3b33e24c9b32938456b2"} Nov 01 20:57:11.589107 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:11.589073 4484 generic.go:356] "Generic (PLEG): container finished" podID="2b4a05cb-3c59-420e-9aaf-9d57535fd45b" containerID="5e892c2131faf44397dc1b3c79c661d6c943078e5f5ba4ad20800b32c12f48f4" exitCode=0 Nov 01 20:57:11.589460 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:11.589159 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a5ab4417-jx9w5" event={"ID":"2b4a05cb-3c59-420e-9aaf-9d57535fd45b","Type":"ContainerDied","Data":"5e892c2131faf44397dc1b3c79c661d6c943078e5f5ba4ad20800b32c12f48f4"} Nov 01 20:57:12.744813 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:12.744772 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:12.853748 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:12.852250 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a5ab4417-jx9w5" Nov 01 20:57:12.949985 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:12.949946 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b4a05cb-3c59-420e-9aaf-9d57535fd45b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b4a05cb-3c59-420e-9aaf-9d57535fd45b-maas-api-xks-ca\") pod \"2b4a05cb-3c59-420e-9aaf-9d57535fd45b\" (UID: \"2b4a05cb-3c59-420e-9aaf-9d57535fd45b\") " Nov 01 20:57:12.950213 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:12.950092 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b4a05cb-3c59-420e-9aaf-9d57535fd45b-kube-api-access-gqk6d\" (UniqueName: \"kubernetes.io/projected/2b4a05cb-3c59-420e-9aaf-9d57535fd45b-kube-api-access-gqk6d\") pod \"2b4a05cb-3c59-420e-9aaf-9d57535fd45b\" (UID: \"2b4a05cb-3c59-420e-9aaf-9d57535fd45b\") " Nov 01 20:57:12.950420 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:12.950404 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b4a05cb-3c59-420e-9aaf-9d57535fd45b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b4a05cb-3c59-420e-9aaf-9d57535fd45b-maas-api-openshift-ca\") pod \"2b4a05cb-3c59-420e-9aaf-9d57535fd45b\" (UID: \"2b4a05cb-3c59-420e-9aaf-9d57535fd45b\") " Nov 01 20:57:12.950756 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:12.950728 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b4a05cb-3c59-420e-9aaf-9d57535fd45b-maas-api-openshift-ca" pod "2b4a05cb-3c59-420e-9aaf-9d57535fd45b" (UID: "2b4a05cb-3c59-420e-9aaf-9d57535fd45b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:57:12.951096 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:12.951078 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b4a05cb-3c59-420e-9aaf-9d57535fd45b-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:12.952126 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:12.952093 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b4a05cb-3c59-420e-9aaf-9d57535fd45b-maas-api-xks-ca" pod "2b4a05cb-3c59-420e-9aaf-9d57535fd45b" (UID: "2b4a05cb-3c59-420e-9aaf-9d57535fd45b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:57:12.952289 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:12.952269 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b4a05cb-3c59-420e-9aaf-9d57535fd45b-kube-api-access-gqk6d" pod "2b4a05cb-3c59-420e-9aaf-9d57535fd45b" (UID: "2b4a05cb-3c59-420e-9aaf-9d57535fd45b"). InnerVolumeSpecName "kube-api-access-gqk6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:57:13.053069 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:13.053027 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b4a05cb-3c59-420e-9aaf-9d57535fd45b-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:13.053069 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:13.053063 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqk6d\" (UniqueName: \"kubernetes.io/projected/2b4a05cb-3c59-420e-9aaf-9d57535fd45b-kube-api-access-gqk6d\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:13.608679 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:13.608650 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a5ab4417-jx9w5" Nov 01 20:57:13.608679 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:13.608665 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a5ab4417-jx9w5" event={"ID":"2b4a05cb-3c59-420e-9aaf-9d57535fd45b","Type":"ContainerDied","Data":"9ce6f9117a19830f3f59f46a637f72b666535598e3eb3b33e24c9b32938456b2"} Nov 01 20:57:13.608924 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:13.608707 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ce6f9117a19830f3f59f46a637f72b666535598e3eb3b33e24c9b32938456b2" Nov 01 20:57:13.955065 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:13.955034 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:15.768528 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:57:15.768493 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 20:57:17.285026 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:17.283030 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:19.232095 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:19.232061 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:20.047218 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:20.047183 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d6e1806-9fddr"] Nov 01 20:57:20.097979 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:20.097948 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d6e1806-9fddr"] Nov 01 20:57:21.052189 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:21.052149 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67e82e0c-8zqjz"] Nov 01 20:57:21.098429 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:21.098392 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67e82e0c-8zqjz"] Nov 01 20:57:21.375464 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:21.375385 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f896c2a-12e2-44b2-9dbe-92af1c8b324e" path="/var/lib/kubelet/pods/2f896c2a-12e2-44b2-9dbe-92af1c8b324e/volumes" Nov 01 20:57:21.375864 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:21.375848 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5c0df19-a16b-4b18-98dd-c900627a705e" path="/var/lib/kubelet/pods/e5c0df19-a16b-4b18-98dd-c900627a705e/volumes" Nov 01 20:57:22.148714 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:22.148674 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:22.339071 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:22.334944 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:23.271063 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:23.270589 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:24.225249 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:24.225200 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:27.234568 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:27.234530 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9"] Nov 01 20:57:27.234989 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:27.234751 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9" podUID="3d20fc4c-5175-4154-8669-2494376150ce" containerName="istio-proxy" containerID="cri-o://c3cc07d5d8d25d52eabbaed8d6102a7c6ad4962435a83d0a0a2bed1d88d14207" gracePeriod=30 Nov 01 20:57:27.285915 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:27.285880 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx"] Nov 01 20:57:27.286191 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:27.286145 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx" podUID="95617e0d-4bf3-435d-86ec-b6c87a0996b9" containerName="istio-proxy" containerID="cri-o://f1fca49a911b5a444a2588075e0dde0e95de2ea943a5e48f78b6946afefbce7f" gracePeriod=30 Nov 01 20:57:27.291332 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:27.291305 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh"] Nov 01 20:57:27.291526 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:27.291506 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh" podUID="388311ca-26b2-4bd9-b355-a5b4c7b459cb" containerName="istio-proxy" containerID="cri-o://9f1f72313446b073366578987cc4a9fef5c87d44bb84830c48a12af23e622def" gracePeriod=30 Nov 01 20:57:28.527705 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:28.527667 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:29.705485 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:29.705452 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:30.095017 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:30.091644 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:30.407156 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:30.407065 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:30.683073 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:30.682699 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ddfb06d-kgkv5"] Nov 01 20:57:30.707107 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:30.707069 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8ddfb06d-kgkv5"] Nov 01 20:57:30.770039 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:30.734335 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr"] Nov 01 20:57:30.770039 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:30.734588 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr" podUID="ce125f0b-43ab-40bf-ae39-5dc6e195ce39" containerName="istio-proxy" containerID="cri-o://f3adf3c2d14d3aac70cf5721d09c367dfad1b21a4e8b37014fbbf98c8b10e37f" gracePeriod=30 Nov 01 20:57:30.770039 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:30.751549 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:31.311982 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:31.310603 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:31.413689 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:31.413651 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcd1f232-a94e-46f8-9f55-87153161bb61" path="/var/lib/kubelet/pods/fcd1f232-a94e-46f8-9f55-87153161bb61/volumes" Nov 01 20:57:31.480611 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:31.480578 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:31.654886 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:31.654780 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:31.823622 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:31.823583 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:31.874733 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:31.874701 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74ab02fe-c6dpl"] Nov 01 20:57:31.879164 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:31.879138 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74ab02fe-c6dpl"] Nov 01 20:57:31.980195 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:31.980118 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:32.016191 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:32.016151 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6"] Nov 01 20:57:32.018256 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:32.018227 4484 cpu_manager.go:414] "containerMap: removing container" podUID="2b4a05cb-3c59-420e-9aaf-9d57535fd45b" containerName="revoke-keys" Nov 01 20:57:32.018497 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:32.018475 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b4a05cb-3c59-420e-9aaf-9d57535fd45b" containerName="revoke-keys" Nov 01 20:57:32.018797 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:32.018779 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b4a05cb-3c59-420e-9aaf-9d57535fd45b" containerName="revoke-keys" Nov 01 20:57:32.025461 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:32.025438 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6" Nov 01 20:57:32.120021 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:32.119652 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-de8145-openshift-default-dockercfg-d8vwq\"" Nov 01 20:57:32.167017 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:32.165564 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5caf509e-0540-4ff6-a544-5b9e83bda042-credential-socket\") pod \"e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6\" (UID: \"5caf509e-0540-4ff6-a544-5b9e83bda042\") " pod="openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6" Nov 01 20:57:32.167017 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:32.165649 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5caf509e-0540-4ff6-a544-5b9e83bda042-istiod-ca-cert\") pod \"e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6\" (UID: \"5caf509e-0540-4ff6-a544-5b9e83bda042\") " pod="openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6" Nov 01 20:57:32.167017 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:32.165699 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5caf509e-0540-4ff6-a544-5b9e83bda042-workload-certs\") pod \"e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6\" (UID: \"5caf509e-0540-4ff6-a544-5b9e83bda042\") " pod="openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6" Nov 01 20:57:32.167017 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:32.165770 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5caf509e-0540-4ff6-a544-5b9e83bda042-istio-token\") pod \"e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6\" (UID: \"5caf509e-0540-4ff6-a544-5b9e83bda042\") " pod="openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6" Nov 01 20:57:32.167017 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:32.165918 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5caf509e-0540-4ff6-a544-5b9e83bda042-workload-socket\") pod \"e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6\" (UID: \"5caf509e-0540-4ff6-a544-5b9e83bda042\") " pod="openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6" Nov 01 20:57:32.167017 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:32.165952 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkwp7\" (UniqueName: \"kubernetes.io/projected/5caf509e-0540-4ff6-a544-5b9e83bda042-kube-api-access-lkwp7\") pod \"e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6\" (UID: \"5caf509e-0540-4ff6-a544-5b9e83bda042\") " pod="openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6" Nov 01 20:57:32.167017 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:32.166077 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5caf509e-0540-4ff6-a544-5b9e83bda042-istio-podinfo\") pod \"e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6\" (UID: \"5caf509e-0540-4ff6-a544-5b9e83bda042\") " pod="openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6" Nov 01 20:57:32.167017 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:32.166159 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5caf509e-0540-4ff6-a544-5b9e83bda042-istio-envoy\") pod \"e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6\" (UID: \"5caf509e-0540-4ff6-a544-5b9e83bda042\") " pod="openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6" Nov 01 20:57:32.167017 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:32.166223 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5caf509e-0540-4ff6-a544-5b9e83bda042-istio-data\") pod \"e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6\" (UID: \"5caf509e-0540-4ff6-a544-5b9e83bda042\") " pod="openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6" Nov 01 20:57:32.268019 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:32.258257 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6"] Nov 01 20:57:32.340086 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:32.335076 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5caf509e-0540-4ff6-a544-5b9e83bda042-workload-socket\") pod \"e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6\" (UID: \"5caf509e-0540-4ff6-a544-5b9e83bda042\") " pod="openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6" Nov 01 20:57:32.340086 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:32.335149 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkwp7\" (UniqueName: \"kubernetes.io/projected/5caf509e-0540-4ff6-a544-5b9e83bda042-kube-api-access-lkwp7\") pod \"e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6\" (UID: \"5caf509e-0540-4ff6-a544-5b9e83bda042\") " pod="openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6" Nov 01 20:57:32.340086 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:32.335289 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5caf509e-0540-4ff6-a544-5b9e83bda042-istio-podinfo\") pod \"e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6\" (UID: \"5caf509e-0540-4ff6-a544-5b9e83bda042\") " pod="openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6" Nov 01 20:57:32.340086 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:32.335379 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5caf509e-0540-4ff6-a544-5b9e83bda042-istio-envoy\") pod \"e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6\" (UID: \"5caf509e-0540-4ff6-a544-5b9e83bda042\") " pod="openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6" Nov 01 20:57:32.340086 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:32.335445 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5caf509e-0540-4ff6-a544-5b9e83bda042-istio-data\") pod \"e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6\" (UID: \"5caf509e-0540-4ff6-a544-5b9e83bda042\") " pod="openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6" Nov 01 20:57:32.340086 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:32.335543 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5caf509e-0540-4ff6-a544-5b9e83bda042-credential-socket\") pod \"e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6\" (UID: \"5caf509e-0540-4ff6-a544-5b9e83bda042\") " pod="openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6" Nov 01 20:57:32.340086 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:32.335607 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5caf509e-0540-4ff6-a544-5b9e83bda042-istiod-ca-cert\") pod \"e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6\" (UID: \"5caf509e-0540-4ff6-a544-5b9e83bda042\") " pod="openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6" Nov 01 20:57:32.340086 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:32.335659 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5caf509e-0540-4ff6-a544-5b9e83bda042-workload-certs\") pod \"e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6\" (UID: \"5caf509e-0540-4ff6-a544-5b9e83bda042\") " pod="openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6" Nov 01 20:57:32.340086 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:32.335730 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5caf509e-0540-4ff6-a544-5b9e83bda042-istio-token\") pod \"e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6\" (UID: \"5caf509e-0540-4ff6-a544-5b9e83bda042\") " pod="openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6" Nov 01 20:57:32.340086 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:32.336572 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5caf509e-0540-4ff6-a544-5b9e83bda042-workload-socket\") pod \"e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6\" (UID: \"5caf509e-0540-4ff6-a544-5b9e83bda042\") " pod="openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6" Nov 01 20:57:32.406669 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:32.405064 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5caf509e-0540-4ff6-a544-5b9e83bda042-istio-podinfo\") pod \"e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6\" (UID: \"5caf509e-0540-4ff6-a544-5b9e83bda042\") " pod="openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6" Nov 01 20:57:32.448398 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:32.419582 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5caf509e-0540-4ff6-a544-5b9e83bda042-credential-socket\") pod \"e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6\" (UID: \"5caf509e-0540-4ff6-a544-5b9e83bda042\") " pod="openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6" Nov 01 20:57:32.452876 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:32.452751 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:32.453675 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:32.453650 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5caf509e-0540-4ff6-a544-5b9e83bda042-istiod-ca-cert\") pod \"e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6\" (UID: \"5caf509e-0540-4ff6-a544-5b9e83bda042\") " pod="openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6" Nov 01 20:57:32.553059 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:32.550281 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5caf509e-0540-4ff6-a544-5b9e83bda042-istio-data\") pod \"e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6\" (UID: \"5caf509e-0540-4ff6-a544-5b9e83bda042\") " pod="openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6" Nov 01 20:57:32.805559 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:32.793865 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5caf509e-0540-4ff6-a544-5b9e83bda042-istio-envoy\") pod \"e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6\" (UID: \"5caf509e-0540-4ff6-a544-5b9e83bda042\") " pod="openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6" Nov 01 20:57:32.825043 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:32.819555 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5caf509e-0540-4ff6-a544-5b9e83bda042-workload-certs\") pod \"e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6\" (UID: \"5caf509e-0540-4ff6-a544-5b9e83bda042\") " pod="openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6" Nov 01 20:57:32.868018 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:32.863606 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5caf509e-0540-4ff6-a544-5b9e83bda042-istio-token\") pod \"e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6\" (UID: \"5caf509e-0540-4ff6-a544-5b9e83bda042\") " pod="openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6" Nov 01 20:57:32.938034 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:32.937296 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:33.053018 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:33.027155 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg"] Nov 01 20:57:33.053618 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:33.053581 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg" podUID="8137e500-e107-49d5-81ba-948c7ae986f1" containerName="istio-proxy" containerID="cri-o://a2e6c67edb3b9a3cc7729e33e2f7e1f38315d93b3ee4df918886d9cb59ca5650" gracePeriod=30 Nov 01 20:57:33.068481 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:33.031185 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkwp7\" (UniqueName: \"kubernetes.io/projected/5caf509e-0540-4ff6-a544-5b9e83bda042-kube-api-access-lkwp7\") pod \"e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6\" (UID: \"5caf509e-0540-4ff6-a544-5b9e83bda042\") " pod="openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6" Nov 01 20:57:33.142173 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:33.142137 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:33.167184 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:33.162588 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6" Nov 01 20:57:33.368812 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:33.368719 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:33.390313 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:33.389982 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b594f63-48a1-4a3c-bda5-e52c600c9a41" path="/var/lib/kubelet/pods/7b594f63-48a1-4a3c-bda5-e52c600c9a41/volumes" Nov 01 20:57:33.789496 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:33.779860 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx" Nov 01 20:57:33.789496 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:33.781590 4484 generic.go:356] "Generic (PLEG): container finished" podID="3d20fc4c-5175-4154-8669-2494376150ce" containerID="c3cc07d5d8d25d52eabbaed8d6102a7c6ad4962435a83d0a0a2bed1d88d14207" exitCode=0 Nov 01 20:57:33.789496 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:33.781767 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9" event={"ID":"3d20fc4c-5175-4154-8669-2494376150ce","Type":"ContainerDied","Data":"c3cc07d5d8d25d52eabbaed8d6102a7c6ad4962435a83d0a0a2bed1d88d14207"} Nov 01 20:57:33.789496 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:33.781845 4484 scope.go:122] "RemoveContainer" containerID="e6568ce5159b31428ae7351e04d78dadeaf28e26e6db11e199f4feb8af86d852" Nov 01 20:57:33.789496 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:33.782763 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh" Nov 01 20:57:33.789496 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:33.786754 4484 generic.go:356] "Generic (PLEG): container finished" podID="95617e0d-4bf3-435d-86ec-b6c87a0996b9" containerID="f1fca49a911b5a444a2588075e0dde0e95de2ea943a5e48f78b6946afefbce7f" exitCode=0 Nov 01 20:57:33.813021 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:33.803092 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx" event={"ID":"95617e0d-4bf3-435d-86ec-b6c87a0996b9","Type":"ContainerDied","Data":"f1fca49a911b5a444a2588075e0dde0e95de2ea943a5e48f78b6946afefbce7f"} Nov 01 20:57:33.813021 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:33.803176 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx" event={"ID":"95617e0d-4bf3-435d-86ec-b6c87a0996b9","Type":"ContainerDied","Data":"2ebf960f365afc7cdf88dbd0b5043bdb1e26345d68121e79de614afd83b852be"} Nov 01 20:57:33.813021 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:33.803289 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx" Nov 01 20:57:33.863072 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:33.861444 4484 generic.go:356] "Generic (PLEG): container finished" podID="388311ca-26b2-4bd9-b355-a5b4c7b459cb" containerID="9f1f72313446b073366578987cc4a9fef5c87d44bb84830c48a12af23e622def" exitCode=0 Nov 01 20:57:33.863072 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:33.861731 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh" event={"ID":"388311ca-26b2-4bd9-b355-a5b4c7b459cb","Type":"ContainerDied","Data":"9f1f72313446b073366578987cc4a9fef5c87d44bb84830c48a12af23e622def"} Nov 01 20:57:33.863072 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:33.861790 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh" event={"ID":"388311ca-26b2-4bd9-b355-a5b4c7b459cb","Type":"ContainerDied","Data":"bdb8eaa00f57d4ad6c31bbd9bb82dfa60fa474ccc92e516b0c060b6e63f22e62"} Nov 01 20:57:33.863072 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:33.861915 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh" Nov 01 20:57:33.878041 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:33.868348 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6"] Nov 01 20:57:33.878041 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:33.868985 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95617e0d-4bf3-435d-86ec-b6c87a0996b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95617e0d-4bf3-435d-86ec-b6c87a0996b9-istio-envoy\") pod \"95617e0d-4bf3-435d-86ec-b6c87a0996b9\" (UID: \"95617e0d-4bf3-435d-86ec-b6c87a0996b9\") " Nov 01 20:57:33.878041 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:33.869072 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/388311ca-26b2-4bd9-b355-a5b4c7b459cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/388311ca-26b2-4bd9-b355-a5b4c7b459cb-istiod-ca-cert\") pod \"388311ca-26b2-4bd9-b355-a5b4c7b459cb\" (UID: \"388311ca-26b2-4bd9-b355-a5b4c7b459cb\") " Nov 01 20:57:33.878041 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:33.869135 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/388311ca-26b2-4bd9-b355-a5b4c7b459cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/388311ca-26b2-4bd9-b355-a5b4c7b459cb-istio-podinfo\") pod \"388311ca-26b2-4bd9-b355-a5b4c7b459cb\" (UID: \"388311ca-26b2-4bd9-b355-a5b4c7b459cb\") " Nov 01 20:57:33.878041 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:33.869203 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/388311ca-26b2-4bd9-b355-a5b4c7b459cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/388311ca-26b2-4bd9-b355-a5b4c7b459cb-credential-socket\") pod \"388311ca-26b2-4bd9-b355-a5b4c7b459cb\" (UID: \"388311ca-26b2-4bd9-b355-a5b4c7b459cb\") " Nov 01 20:57:33.878041 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:33.869268 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95617e0d-4bf3-435d-86ec-b6c87a0996b9-istio-token\" (UniqueName: \"kubernetes.io/projected/95617e0d-4bf3-435d-86ec-b6c87a0996b9-istio-token\") pod \"95617e0d-4bf3-435d-86ec-b6c87a0996b9\" (UID: \"95617e0d-4bf3-435d-86ec-b6c87a0996b9\") " Nov 01 20:57:33.878041 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:33.869297 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/388311ca-26b2-4bd9-b355-a5b4c7b459cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/388311ca-26b2-4bd9-b355-a5b4c7b459cb-workload-certs\") pod \"388311ca-26b2-4bd9-b355-a5b4c7b459cb\" (UID: \"388311ca-26b2-4bd9-b355-a5b4c7b459cb\") " Nov 01 20:57:33.878041 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:33.869442 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/388311ca-26b2-4bd9-b355-a5b4c7b459cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/388311ca-26b2-4bd9-b355-a5b4c7b459cb-istio-data\") pod \"388311ca-26b2-4bd9-b355-a5b4c7b459cb\" (UID: \"388311ca-26b2-4bd9-b355-a5b4c7b459cb\") " Nov 01 20:57:33.878041 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:33.869477 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/388311ca-26b2-4bd9-b355-a5b4c7b459cb-kube-api-access-zf5q9\" (UniqueName: \"kubernetes.io/projected/388311ca-26b2-4bd9-b355-a5b4c7b459cb-kube-api-access-zf5q9\") pod \"388311ca-26b2-4bd9-b355-a5b4c7b459cb\" (UID: \"388311ca-26b2-4bd9-b355-a5b4c7b459cb\") " Nov 01 20:57:33.878041 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:33.869507 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95617e0d-4bf3-435d-86ec-b6c87a0996b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95617e0d-4bf3-435d-86ec-b6c87a0996b9-workload-socket\") pod \"95617e0d-4bf3-435d-86ec-b6c87a0996b9\" (UID: \"95617e0d-4bf3-435d-86ec-b6c87a0996b9\") " Nov 01 20:57:33.878041 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:33.869559 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/388311ca-26b2-4bd9-b355-a5b4c7b459cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/388311ca-26b2-4bd9-b355-a5b4c7b459cb-workload-socket\") pod \"388311ca-26b2-4bd9-b355-a5b4c7b459cb\" (UID: \"388311ca-26b2-4bd9-b355-a5b4c7b459cb\") " Nov 01 20:57:33.878041 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:33.869639 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/95617e0d-4bf3-435d-86ec-b6c87a0996b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95617e0d-4bf3-435d-86ec-b6c87a0996b9-istio-podinfo\") pod \"95617e0d-4bf3-435d-86ec-b6c87a0996b9\" (UID: \"95617e0d-4bf3-435d-86ec-b6c87a0996b9\") " Nov 01 20:57:33.878041 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:33.870239 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95617e0d-4bf3-435d-86ec-b6c87a0996b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95617e0d-4bf3-435d-86ec-b6c87a0996b9-credential-socket\") pod \"95617e0d-4bf3-435d-86ec-b6c87a0996b9\" (UID: \"95617e0d-4bf3-435d-86ec-b6c87a0996b9\") " Nov 01 20:57:33.878041 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:33.870312 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/388311ca-26b2-4bd9-b355-a5b4c7b459cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/388311ca-26b2-4bd9-b355-a5b4c7b459cb-istio-envoy\") pod \"388311ca-26b2-4bd9-b355-a5b4c7b459cb\" (UID: \"388311ca-26b2-4bd9-b355-a5b4c7b459cb\") " Nov 01 20:57:33.878041 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:33.870380 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/388311ca-26b2-4bd9-b355-a5b4c7b459cb-istio-token\" (UniqueName: \"kubernetes.io/projected/388311ca-26b2-4bd9-b355-a5b4c7b459cb-istio-token\") pod \"388311ca-26b2-4bd9-b355-a5b4c7b459cb\" (UID: \"388311ca-26b2-4bd9-b355-a5b4c7b459cb\") " Nov 01 20:57:33.878041 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:33.870475 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95617e0d-4bf3-435d-86ec-b6c87a0996b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95617e0d-4bf3-435d-86ec-b6c87a0996b9-istio-data\") pod \"95617e0d-4bf3-435d-86ec-b6c87a0996b9\" (UID: \"95617e0d-4bf3-435d-86ec-b6c87a0996b9\") " Nov 01 20:57:33.878041 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:33.870532 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95617e0d-4bf3-435d-86ec-b6c87a0996b9-kube-api-access-c757m\" (UniqueName: \"kubernetes.io/projected/95617e0d-4bf3-435d-86ec-b6c87a0996b9-kube-api-access-c757m\") pod \"95617e0d-4bf3-435d-86ec-b6c87a0996b9\" (UID: \"95617e0d-4bf3-435d-86ec-b6c87a0996b9\") " Nov 01 20:57:33.878041 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:33.870560 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95617e0d-4bf3-435d-86ec-b6c87a0996b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95617e0d-4bf3-435d-86ec-b6c87a0996b9-workload-certs\") pod \"95617e0d-4bf3-435d-86ec-b6c87a0996b9\" (UID: \"95617e0d-4bf3-435d-86ec-b6c87a0996b9\") " Nov 01 20:57:33.878041 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:33.870615 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95617e0d-4bf3-435d-86ec-b6c87a0996b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95617e0d-4bf3-435d-86ec-b6c87a0996b9-istiod-ca-cert\") pod \"95617e0d-4bf3-435d-86ec-b6c87a0996b9\" (UID: \"95617e0d-4bf3-435d-86ec-b6c87a0996b9\") " Nov 01 20:57:33.879213 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:33.878462 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/388311ca-26b2-4bd9-b355-a5b4c7b459cb-istiod-ca-cert" pod "388311ca-26b2-4bd9-b355-a5b4c7b459cb" (UID: "388311ca-26b2-4bd9-b355-a5b4c7b459cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:57:33.879213 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:33.878841 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95617e0d-4bf3-435d-86ec-b6c87a0996b9-credential-socket" pod "95617e0d-4bf3-435d-86ec-b6c87a0996b9" (UID: "95617e0d-4bf3-435d-86ec-b6c87a0996b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:57:33.889076 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:33.879905 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95617e0d-4bf3-435d-86ec-b6c87a0996b9-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:33.889076 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:33.879938 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/388311ca-26b2-4bd9-b355-a5b4c7b459cb-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:33.889076 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:33.880345 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95617e0d-4bf3-435d-86ec-b6c87a0996b9-workload-certs" pod "95617e0d-4bf3-435d-86ec-b6c87a0996b9" (UID: "95617e0d-4bf3-435d-86ec-b6c87a0996b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:57:33.889076 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:33.880679 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95617e0d-4bf3-435d-86ec-b6c87a0996b9-istiod-ca-cert" pod "95617e0d-4bf3-435d-86ec-b6c87a0996b9" (UID: "95617e0d-4bf3-435d-86ec-b6c87a0996b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:57:33.889506 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:33.889305 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95617e0d-4bf3-435d-86ec-b6c87a0996b9-workload-socket" pod "95617e0d-4bf3-435d-86ec-b6c87a0996b9" (UID: "95617e0d-4bf3-435d-86ec-b6c87a0996b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:57:33.950021 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:33.948474 4484 scope.go:122] "RemoveContainer" containerID="f1fca49a911b5a444a2588075e0dde0e95de2ea943a5e48f78b6946afefbce7f" Nov 01 20:57:34.031024 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:33.983438 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95617e0d-4bf3-435d-86ec-b6c87a0996b9-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:34.031024 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:33.983477 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95617e0d-4bf3-435d-86ec-b6c87a0996b9-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:34.031024 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:33.983502 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95617e0d-4bf3-435d-86ec-b6c87a0996b9-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:34.031024 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:33.983957 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m"] Nov 01 20:57:34.031024 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:33.985452 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/388311ca-26b2-4bd9-b355-a5b4c7b459cb-workload-socket" pod "388311ca-26b2-4bd9-b355-a5b4c7b459cb" (UID: "388311ca-26b2-4bd9-b355-a5b4c7b459cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:57:34.031024 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:33.990326 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/388311ca-26b2-4bd9-b355-a5b4c7b459cb-istio-data" pod "388311ca-26b2-4bd9-b355-a5b4c7b459cb" (UID: "388311ca-26b2-4bd9-b355-a5b4c7b459cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:57:34.031024 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:33.990567 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/388311ca-26b2-4bd9-b355-a5b4c7b459cb-workload-certs" pod "388311ca-26b2-4bd9-b355-a5b4c7b459cb" (UID: "388311ca-26b2-4bd9-b355-a5b4c7b459cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:57:34.031024 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:33.994203 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95617e0d-4bf3-435d-86ec-b6c87a0996b9-istio-data" pod "95617e0d-4bf3-435d-86ec-b6c87a0996b9" (UID: "95617e0d-4bf3-435d-86ec-b6c87a0996b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:57:34.031024 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:33.995873 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/388311ca-26b2-4bd9-b355-a5b4c7b459cb-credential-socket" pod "388311ca-26b2-4bd9-b355-a5b4c7b459cb" (UID: "388311ca-26b2-4bd9-b355-a5b4c7b459cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:57:34.031024 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.010197 4484 patch_prober.go:29] interesting pod/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 20:57:34.031024 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.010293 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr" podUID="ce125f0b-43ab-40bf-ae39-5dc6e195ce39" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 20:57:34.077019 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.046501 4484 cpu_manager.go:414] "containerMap: removing container" podUID="388311ca-26b2-4bd9-b355-a5b4c7b459cb" containerName="istio-proxy" Nov 01 20:57:34.077019 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.046534 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="388311ca-26b2-4bd9-b355-a5b4c7b459cb" containerName="istio-proxy" Nov 01 20:57:34.077019 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.046555 4484 cpu_manager.go:414] "containerMap: removing container" podUID="95617e0d-4bf3-435d-86ec-b6c87a0996b9" containerName="istio-proxy" Nov 01 20:57:34.077019 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.046564 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95617e0d-4bf3-435d-86ec-b6c87a0996b9" containerName="istio-proxy" Nov 01 20:57:34.077019 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.046858 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="388311ca-26b2-4bd9-b355-a5b4c7b459cb" containerName="istio-proxy" Nov 01 20:57:34.077019 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.046872 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="95617e0d-4bf3-435d-86ec-b6c87a0996b9" containerName="istio-proxy" Nov 01 20:57:34.077019 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.062986 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:34.077019 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.073563 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:57:34.077019 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.073653 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:57:34.077019 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.073695 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:57:34.091612 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.086917 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/388311ca-26b2-4bd9-b355-a5b4c7b459cb-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:34.091612 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.086949 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/388311ca-26b2-4bd9-b355-a5b4c7b459cb-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:34.091612 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.086965 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/388311ca-26b2-4bd9-b355-a5b4c7b459cb-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:34.091612 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.086981 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/388311ca-26b2-4bd9-b355-a5b4c7b459cb-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:34.091612 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.087010 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95617e0d-4bf3-435d-86ec-b6c87a0996b9-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:34.091612 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.090724 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m"] Nov 01 20:57:34.091612 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.090884 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m" Nov 01 20:57:34.119158 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.119101 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/388311ca-26b2-4bd9-b355-a5b4c7b459cb-istio-envoy" pod "388311ca-26b2-4bd9-b355-a5b4c7b459cb" (UID: "388311ca-26b2-4bd9-b355-a5b4c7b459cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:57:34.119298 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.119186 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95617e0d-4bf3-435d-86ec-b6c87a0996b9-istio-envoy" pod "95617e0d-4bf3-435d-86ec-b6c87a0996b9" (UID: "95617e0d-4bf3-435d-86ec-b6c87a0996b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:57:34.120298 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.120175 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95617e0d-4bf3-435d-86ec-b6c87a0996b9-kube-api-access-c757m" pod "95617e0d-4bf3-435d-86ec-b6c87a0996b9" (UID: "95617e0d-4bf3-435d-86ec-b6c87a0996b9"). InnerVolumeSpecName "kube-api-access-c757m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:57:34.123196 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.123155 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95617e0d-4bf3-435d-86ec-b6c87a0996b9-istio-podinfo" pod "95617e0d-4bf3-435d-86ec-b6c87a0996b9" (UID: "95617e0d-4bf3-435d-86ec-b6c87a0996b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:57:34.123499 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.123459 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95617e0d-4bf3-435d-86ec-b6c87a0996b9-istio-token" pod "95617e0d-4bf3-435d-86ec-b6c87a0996b9" (UID: "95617e0d-4bf3-435d-86ec-b6c87a0996b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:57:34.151150 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.151069 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-a9b439-openshift-default-dockercfg-4sfv9\"" Nov 01 20:57:34.229077 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.198166 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/388311ca-26b2-4bd9-b355-a5b4c7b459cb-istio-podinfo" pod "388311ca-26b2-4bd9-b355-a5b4c7b459cb" (UID: "388311ca-26b2-4bd9-b355-a5b4c7b459cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:57:34.229077 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.199044 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/388311ca-26b2-4bd9-b355-a5b4c7b459cb-kube-api-access-zf5q9" pod "388311ca-26b2-4bd9-b355-a5b4c7b459cb" (UID: "388311ca-26b2-4bd9-b355-a5b4c7b459cb"). InnerVolumeSpecName "kube-api-access-zf5q9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:57:34.229077 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.199369 4484 scope.go:122] "RemoveContainer" containerID="f1fca49a911b5a444a2588075e0dde0e95de2ea943a5e48f78b6946afefbce7f" Nov 01 20:57:34.229077 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.200343 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/388311ca-26b2-4bd9-b355-a5b4c7b459cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/388311ca-26b2-4bd9-b355-a5b4c7b459cb-istio-podinfo\") pod \"388311ca-26b2-4bd9-b355-a5b4c7b459cb\" (UID: \"388311ca-26b2-4bd9-b355-a5b4c7b459cb\") " Nov 01 20:57:34.229077 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:57:34.200428 4484 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/388311ca-26b2-4bd9-b355-a5b4c7b459cb/volumes/kubernetes.io~downward-api/istio-podinfo Nov 01 20:57:34.229077 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.200441 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/388311ca-26b2-4bd9-b355-a5b4c7b459cb-istio-podinfo" pod "388311ca-26b2-4bd9-b355-a5b4c7b459cb" (UID: "388311ca-26b2-4bd9-b355-a5b4c7b459cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:57:34.229077 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.201387 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/388311ca-26b2-4bd9-b355-a5b4c7b459cb-kube-api-access-zf5q9\" (UniqueName: \"kubernetes.io/projected/388311ca-26b2-4bd9-b355-a5b4c7b459cb-kube-api-access-zf5q9\") pod \"388311ca-26b2-4bd9-b355-a5b4c7b459cb\" (UID: \"388311ca-26b2-4bd9-b355-a5b4c7b459cb\") " Nov 01 20:57:34.229077 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:57:34.201511 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1fca49a911b5a444a2588075e0dde0e95de2ea943a5e48f78b6946afefbce7f\": container with ID starting with f1fca49a911b5a444a2588075e0dde0e95de2ea943a5e48f78b6946afefbce7f not found: ID does not exist" containerID="f1fca49a911b5a444a2588075e0dde0e95de2ea943a5e48f78b6946afefbce7f" Nov 01 20:57:34.229077 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.201591 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1fca49a911b5a444a2588075e0dde0e95de2ea943a5e48f78b6946afefbce7f"} err="failed to get container status \"f1fca49a911b5a444a2588075e0dde0e95de2ea943a5e48f78b6946afefbce7f\": rpc error: code = NotFound desc = could not find container \"f1fca49a911b5a444a2588075e0dde0e95de2ea943a5e48f78b6946afefbce7f\": container with ID starting with f1fca49a911b5a444a2588075e0dde0e95de2ea943a5e48f78b6946afefbce7f not found: ID does not exist" Nov 01 20:57:34.229077 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.201615 4484 scope.go:122] "RemoveContainer" containerID="9f1f72313446b073366578987cc4a9fef5c87d44bb84830c48a12af23e622def" Nov 01 20:57:34.229077 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.201989 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ecec0b0-7e22-4803-891c-0a75173445ad-istiod-ca-cert\") pod \"e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m\" (UID: \"7ecec0b0-7e22-4803-891c-0a75173445ad\") " pod="openshift-ingress/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m" Nov 01 20:57:34.229077 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.202167 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ecec0b0-7e22-4803-891c-0a75173445ad-istio-envoy\") pod \"e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m\" (UID: \"7ecec0b0-7e22-4803-891c-0a75173445ad\") " pod="openshift-ingress/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m" Nov 01 20:57:34.229077 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.202232 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ecec0b0-7e22-4803-891c-0a75173445ad-istio-data\") pod \"e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m\" (UID: \"7ecec0b0-7e22-4803-891c-0a75173445ad\") " pod="openshift-ingress/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m" Nov 01 20:57:34.229077 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.202294 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl6b6\" (UniqueName: \"kubernetes.io/projected/7ecec0b0-7e22-4803-891c-0a75173445ad-kube-api-access-vl6b6\") pod \"e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m\" (UID: \"7ecec0b0-7e22-4803-891c-0a75173445ad\") " pod="openshift-ingress/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m" Nov 01 20:57:34.229077 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.202350 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ecec0b0-7e22-4803-891c-0a75173445ad-workload-certs\") pod \"e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m\" (UID: \"7ecec0b0-7e22-4803-891c-0a75173445ad\") " pod="openshift-ingress/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m" Nov 01 20:57:34.229077 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.202402 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ecec0b0-7e22-4803-891c-0a75173445ad-credential-socket\") pod \"e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m\" (UID: \"7ecec0b0-7e22-4803-891c-0a75173445ad\") " pod="openshift-ingress/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m" Nov 01 20:57:34.229077 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.202513 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ecec0b0-7e22-4803-891c-0a75173445ad-workload-socket\") pod \"e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m\" (UID: \"7ecec0b0-7e22-4803-891c-0a75173445ad\") " pod="openshift-ingress/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m" Nov 01 20:57:34.229077 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.202542 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ecec0b0-7e22-4803-891c-0a75173445ad-istio-token\") pod \"e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m\" (UID: \"7ecec0b0-7e22-4803-891c-0a75173445ad\") " pod="openshift-ingress/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m" Nov 01 20:57:34.229077 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.202575 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ecec0b0-7e22-4803-891c-0a75173445ad-istio-podinfo\") pod \"e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m\" (UID: \"7ecec0b0-7e22-4803-891c-0a75173445ad\") " pod="openshift-ingress/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m" Nov 01 20:57:34.229077 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.202760 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95617e0d-4bf3-435d-86ec-b6c87a0996b9-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:34.229077 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.202787 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/388311ca-26b2-4bd9-b355-a5b4c7b459cb-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:34.229077 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.202806 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c757m\" (UniqueName: \"kubernetes.io/projected/95617e0d-4bf3-435d-86ec-b6c87a0996b9-kube-api-access-c757m\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:34.229077 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.202824 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95617e0d-4bf3-435d-86ec-b6c87a0996b9-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:34.229077 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.202841 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/388311ca-26b2-4bd9-b355-a5b4c7b459cb-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:34.229077 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.202862 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95617e0d-4bf3-435d-86ec-b6c87a0996b9-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:34.229077 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:57:34.202976 4484 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/388311ca-26b2-4bd9-b355-a5b4c7b459cb/volumes/kubernetes.io~projected/kube-api-access-zf5q9 Nov 01 20:57:34.229077 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.203050 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/388311ca-26b2-4bd9-b355-a5b4c7b459cb-kube-api-access-zf5q9" pod "388311ca-26b2-4bd9-b355-a5b4c7b459cb" (UID: "388311ca-26b2-4bd9-b355-a5b4c7b459cb"). InnerVolumeSpecName "kube-api-access-zf5q9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:57:34.316608 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.305228 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ecec0b0-7e22-4803-891c-0a75173445ad-istiod-ca-cert\") pod \"e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m\" (UID: \"7ecec0b0-7e22-4803-891c-0a75173445ad\") " pod="openshift-ingress/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m" Nov 01 20:57:34.316608 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.305367 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ecec0b0-7e22-4803-891c-0a75173445ad-istio-envoy\") pod \"e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m\" (UID: \"7ecec0b0-7e22-4803-891c-0a75173445ad\") " pod="openshift-ingress/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m" Nov 01 20:57:34.316608 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.305434 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ecec0b0-7e22-4803-891c-0a75173445ad-istio-data\") pod \"e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m\" (UID: \"7ecec0b0-7e22-4803-891c-0a75173445ad\") " pod="openshift-ingress/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m" Nov 01 20:57:34.316608 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.305496 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl6b6\" (UniqueName: \"kubernetes.io/projected/7ecec0b0-7e22-4803-891c-0a75173445ad-kube-api-access-vl6b6\") pod \"e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m\" (UID: \"7ecec0b0-7e22-4803-891c-0a75173445ad\") " pod="openshift-ingress/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m" Nov 01 20:57:34.316608 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.305547 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ecec0b0-7e22-4803-891c-0a75173445ad-workload-certs\") pod \"e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m\" (UID: \"7ecec0b0-7e22-4803-891c-0a75173445ad\") " pod="openshift-ingress/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m" Nov 01 20:57:34.316608 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.305594 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ecec0b0-7e22-4803-891c-0a75173445ad-credential-socket\") pod \"e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m\" (UID: \"7ecec0b0-7e22-4803-891c-0a75173445ad\") " pod="openshift-ingress/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m" Nov 01 20:57:34.316608 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.305687 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ecec0b0-7e22-4803-891c-0a75173445ad-workload-socket\") pod \"e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m\" (UID: \"7ecec0b0-7e22-4803-891c-0a75173445ad\") " pod="openshift-ingress/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m" Nov 01 20:57:34.316608 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.305715 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ecec0b0-7e22-4803-891c-0a75173445ad-istio-token\") pod \"e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m\" (UID: \"7ecec0b0-7e22-4803-891c-0a75173445ad\") " pod="openshift-ingress/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m" Nov 01 20:57:34.316608 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.305749 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ecec0b0-7e22-4803-891c-0a75173445ad-istio-podinfo\") pod \"e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m\" (UID: \"7ecec0b0-7e22-4803-891c-0a75173445ad\") " pod="openshift-ingress/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m" Nov 01 20:57:34.316608 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.305847 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf5q9\" (UniqueName: \"kubernetes.io/projected/388311ca-26b2-4bd9-b355-a5b4c7b459cb-kube-api-access-zf5q9\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:34.316608 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.314326 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ecec0b0-7e22-4803-891c-0a75173445ad-istiod-ca-cert\") pod \"e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m\" (UID: \"7ecec0b0-7e22-4803-891c-0a75173445ad\") " pod="openshift-ingress/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m" Nov 01 20:57:34.327864 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.320107 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ecec0b0-7e22-4803-891c-0a75173445ad-istio-envoy\") pod \"e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m\" (UID: \"7ecec0b0-7e22-4803-891c-0a75173445ad\") " pod="openshift-ingress/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m" Nov 01 20:57:34.327864 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.320453 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ecec0b0-7e22-4803-891c-0a75173445ad-istio-data\") pod \"e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m\" (UID: \"7ecec0b0-7e22-4803-891c-0a75173445ad\") " pod="openshift-ingress/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m" Nov 01 20:57:34.327864 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.320751 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ecec0b0-7e22-4803-891c-0a75173445ad-credential-socket\") pod \"e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m\" (UID: \"7ecec0b0-7e22-4803-891c-0a75173445ad\") " pod="openshift-ingress/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m" Nov 01 20:57:34.327864 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.321025 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ecec0b0-7e22-4803-891c-0a75173445ad-workload-certs\") pod \"e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m\" (UID: \"7ecec0b0-7e22-4803-891c-0a75173445ad\") " pod="openshift-ingress/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m" Nov 01 20:57:34.327864 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.321711 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ecec0b0-7e22-4803-891c-0a75173445ad-workload-socket\") pod \"e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m\" (UID: \"7ecec0b0-7e22-4803-891c-0a75173445ad\") " pod="openshift-ingress/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m" Nov 01 20:57:34.383914 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.383782 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:34.385601 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.385568 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ecec0b0-7e22-4803-891c-0a75173445ad-istio-podinfo\") pod \"e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m\" (UID: \"7ecec0b0-7e22-4803-891c-0a75173445ad\") " pod="openshift-ingress/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m" Nov 01 20:57:34.389041 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.388974 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9" Nov 01 20:57:34.390725 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.390658 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ecec0b0-7e22-4803-891c-0a75173445ad-istio-token\") pod \"e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m\" (UID: \"7ecec0b0-7e22-4803-891c-0a75173445ad\") " pod="openshift-ingress/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m" Nov 01 20:57:34.390851 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.390832 4484 scope.go:122] "RemoveContainer" containerID="9f1f72313446b073366578987cc4a9fef5c87d44bb84830c48a12af23e622def" Nov 01 20:57:34.391164 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.391137 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/388311ca-26b2-4bd9-b355-a5b4c7b459cb-istio-token" pod "388311ca-26b2-4bd9-b355-a5b4c7b459cb" (UID: "388311ca-26b2-4bd9-b355-a5b4c7b459cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:57:34.391353 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.391325 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl6b6\" (UniqueName: \"kubernetes.io/projected/7ecec0b0-7e22-4803-891c-0a75173445ad-kube-api-access-vl6b6\") pod \"e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m\" (UID: \"7ecec0b0-7e22-4803-891c-0a75173445ad\") " pod="openshift-ingress/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m" Nov 01 20:57:34.392054 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:57:34.391473 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f1f72313446b073366578987cc4a9fef5c87d44bb84830c48a12af23e622def\": container with ID starting with 9f1f72313446b073366578987cc4a9fef5c87d44bb84830c48a12af23e622def not found: ID does not exist" containerID="9f1f72313446b073366578987cc4a9fef5c87d44bb84830c48a12af23e622def" Nov 01 20:57:34.392054 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.391543 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f1f72313446b073366578987cc4a9fef5c87d44bb84830c48a12af23e622def"} err="failed to get container status \"9f1f72313446b073366578987cc4a9fef5c87d44bb84830c48a12af23e622def\": rpc error: code = NotFound desc = could not find container \"9f1f72313446b073366578987cc4a9fef5c87d44bb84830c48a12af23e622def\": container with ID starting with 9f1f72313446b073366578987cc4a9fef5c87d44bb84830c48a12af23e622def not found: ID does not exist" Nov 01 20:57:34.407848 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.406691 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d20fc4c-5175-4154-8669-2494376150ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d20fc4c-5175-4154-8669-2494376150ce-istio-data\") pod \"3d20fc4c-5175-4154-8669-2494376150ce\" (UID: \"3d20fc4c-5175-4154-8669-2494376150ce\") " Nov 01 20:57:34.407848 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.406811 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d20fc4c-5175-4154-8669-2494376150ce-istio-token\" (UniqueName: \"kubernetes.io/projected/3d20fc4c-5175-4154-8669-2494376150ce-istio-token\") pod \"3d20fc4c-5175-4154-8669-2494376150ce\" (UID: \"3d20fc4c-5175-4154-8669-2494376150ce\") " Nov 01 20:57:34.407848 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.406846 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d20fc4c-5175-4154-8669-2494376150ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d20fc4c-5175-4154-8669-2494376150ce-istio-envoy\") pod \"3d20fc4c-5175-4154-8669-2494376150ce\" (UID: \"3d20fc4c-5175-4154-8669-2494376150ce\") " Nov 01 20:57:34.407848 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.406895 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d20fc4c-5175-4154-8669-2494376150ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d20fc4c-5175-4154-8669-2494376150ce-workload-socket\") pod \"3d20fc4c-5175-4154-8669-2494376150ce\" (UID: \"3d20fc4c-5175-4154-8669-2494376150ce\") " Nov 01 20:57:34.407848 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.407083 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d20fc4c-5175-4154-8669-2494376150ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d20fc4c-5175-4154-8669-2494376150ce-istiod-ca-cert\") pod \"3d20fc4c-5175-4154-8669-2494376150ce\" (UID: \"3d20fc4c-5175-4154-8669-2494376150ce\") " Nov 01 20:57:34.407848 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.407139 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d20fc4c-5175-4154-8669-2494376150ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d20fc4c-5175-4154-8669-2494376150ce-credential-socket\") pod \"3d20fc4c-5175-4154-8669-2494376150ce\" (UID: \"3d20fc4c-5175-4154-8669-2494376150ce\") " Nov 01 20:57:34.407848 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.407239 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d20fc4c-5175-4154-8669-2494376150ce-kube-api-access-jgwlc\" (UniqueName: \"kubernetes.io/projected/3d20fc4c-5175-4154-8669-2494376150ce-kube-api-access-jgwlc\") pod \"3d20fc4c-5175-4154-8669-2494376150ce\" (UID: \"3d20fc4c-5175-4154-8669-2494376150ce\") " Nov 01 20:57:34.407848 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.407325 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d20fc4c-5175-4154-8669-2494376150ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d20fc4c-5175-4154-8669-2494376150ce-workload-certs\") pod \"3d20fc4c-5175-4154-8669-2494376150ce\" (UID: \"3d20fc4c-5175-4154-8669-2494376150ce\") " Nov 01 20:57:34.407848 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.407404 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d20fc4c-5175-4154-8669-2494376150ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d20fc4c-5175-4154-8669-2494376150ce-istio-podinfo\") pod \"3d20fc4c-5175-4154-8669-2494376150ce\" (UID: \"3d20fc4c-5175-4154-8669-2494376150ce\") " Nov 01 20:57:34.409554 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.409096 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/388311ca-26b2-4bd9-b355-a5b4c7b459cb-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:34.410095 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.409732 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d20fc4c-5175-4154-8669-2494376150ce-istio-data" pod "3d20fc4c-5175-4154-8669-2494376150ce" (UID: "3d20fc4c-5175-4154-8669-2494376150ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:57:34.419047 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.414939 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d20fc4c-5175-4154-8669-2494376150ce-credential-socket" pod "3d20fc4c-5175-4154-8669-2494376150ce" (UID: "3d20fc4c-5175-4154-8669-2494376150ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:57:34.419047 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.415895 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d20fc4c-5175-4154-8669-2494376150ce-workload-socket" pod "3d20fc4c-5175-4154-8669-2494376150ce" (UID: "3d20fc4c-5175-4154-8669-2494376150ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:57:34.419047 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.418254 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d20fc4c-5175-4154-8669-2494376150ce-workload-certs" pod "3d20fc4c-5175-4154-8669-2494376150ce" (UID: "3d20fc4c-5175-4154-8669-2494376150ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:57:34.419047 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.418666 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d20fc4c-5175-4154-8669-2494376150ce-kube-api-access-jgwlc" pod "3d20fc4c-5175-4154-8669-2494376150ce" (UID: "3d20fc4c-5175-4154-8669-2494376150ce"). InnerVolumeSpecName "kube-api-access-jgwlc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:57:34.419047 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.418964 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d20fc4c-5175-4154-8669-2494376150ce-istiod-ca-cert" pod "3d20fc4c-5175-4154-8669-2494376150ce" (UID: "3d20fc4c-5175-4154-8669-2494376150ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:57:34.452783 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.452467 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m" Nov 01 20:57:34.491839 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.464467 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d20fc4c-5175-4154-8669-2494376150ce-istio-podinfo" pod "3d20fc4c-5175-4154-8669-2494376150ce" (UID: "3d20fc4c-5175-4154-8669-2494376150ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:57:34.505401 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.505020 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d20fc4c-5175-4154-8669-2494376150ce-istio-token" pod "3d20fc4c-5175-4154-8669-2494376150ce" (UID: "3d20fc4c-5175-4154-8669-2494376150ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:57:34.514716 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.514682 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d20fc4c-5175-4154-8669-2494376150ce-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:34.514935 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.514917 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgwlc\" (UniqueName: \"kubernetes.io/projected/3d20fc4c-5175-4154-8669-2494376150ce-kube-api-access-jgwlc\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:34.515064 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.515050 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d20fc4c-5175-4154-8669-2494376150ce-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:34.515404 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.515388 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d20fc4c-5175-4154-8669-2494376150ce-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:34.515529 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.515513 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d20fc4c-5175-4154-8669-2494376150ce-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:34.515650 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.515637 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d20fc4c-5175-4154-8669-2494376150ce-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:34.515770 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.515756 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d20fc4c-5175-4154-8669-2494376150ce-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:34.515888 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.515874 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d20fc4c-5175-4154-8669-2494376150ce-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:34.556025 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.555164 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d20fc4c-5175-4154-8669-2494376150ce-istio-envoy" pod "3d20fc4c-5175-4154-8669-2494376150ce" (UID: "3d20fc4c-5175-4154-8669-2494376150ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:57:34.566033 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.565606 4484 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:57:34.566033 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.565669 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx" podUID="95617e0d-4bf3-435d-86ec-b6c87a0996b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:57:34.618086 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.617763 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d20fc4c-5175-4154-8669-2494376150ce-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:34.737388 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.737346 4484 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:57:34.737553 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.737417 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh" podUID="388311ca-26b2-4bd9-b355-a5b4c7b459cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:57:34.766109 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.766068 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh"] Nov 01 20:57:34.769238 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.769201 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-czlxh"] Nov 01 20:57:34.872934 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.872858 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx"] Nov 01 20:57:34.896888 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.896796 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9" event={"ID":"3d20fc4c-5175-4154-8669-2494376150ce","Type":"ContainerDied","Data":"2af5f98367dc2e08af7317fc7c30131c3c9e2c4ec75846299cf4ea4b2663198b"} Nov 01 20:57:34.897273 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.897250 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9" Nov 01 20:57:34.897636 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.897615 4484 scope.go:122] "RemoveContainer" containerID="c3cc07d5d8d25d52eabbaed8d6102a7c6ad4962435a83d0a0a2bed1d88d14207" Nov 01 20:57:34.901747 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.901720 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6" event={"ID":"5caf509e-0540-4ff6-a544-5b9e83bda042","Type":"ContainerStarted","Data":"b877616fcd50c65da91923af62eadbf103481a08c0b2aa30cab537aaf0efc123"} Nov 01 20:57:34.901899 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.901881 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6" event={"ID":"5caf509e-0540-4ff6-a544-5b9e83bda042","Type":"ContainerStarted","Data":"40c3d24202032567d872e6549d49f8072c003776a2a2ad8a89fb46ae0a5f485d"} Nov 01 20:57:34.903581 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:34.903557 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-gjkgx"] Nov 01 20:57:35.033020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:35.022904 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m"] Nov 01 20:57:35.078498 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:35.073886 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:57:35.078498 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:35.073977 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:57:35.078498 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:35.074044 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:57:35.087107 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:35.087059 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9"] Nov 01 20:57:35.088169 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:35.088128 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-kndl9"] Nov 01 20:57:35.167728 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:35.166074 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6" Nov 01 20:57:35.177090 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:35.177047 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6" Nov 01 20:57:35.216032 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:35.214853 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6" podStartSLOduration=4.214827461 podStartE2EDuration="4.214827461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:57:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:57:35.208441149 +0000 UTC m=+1605.986081297" watchObservedRunningTime="2026-11-01 20:57:35.214827461 +0000 UTC m=+1605.992467625" Nov 01 20:57:35.434207 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:35.434106 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="388311ca-26b2-4bd9-b355-a5b4c7b459cb" path="/var/lib/kubelet/pods/388311ca-26b2-4bd9-b355-a5b4c7b459cb/volumes" Nov 01 20:57:35.435244 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:35.435221 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d20fc4c-5175-4154-8669-2494376150ce" path="/var/lib/kubelet/pods/3d20fc4c-5175-4154-8669-2494376150ce/volumes" Nov 01 20:57:35.436028 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:35.436009 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95617e0d-4bf3-435d-86ec-b6c87a0996b9" path="/var/lib/kubelet/pods/95617e0d-4bf3-435d-86ec-b6c87a0996b9/volumes" Nov 01 20:57:35.916925 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:35.916890 4484 generic.go:356] "Generic (PLEG): container finished" podID="ce125f0b-43ab-40bf-ae39-5dc6e195ce39" containerID="f3adf3c2d14d3aac70cf5721d09c367dfad1b21a4e8b37014fbbf98c8b10e37f" exitCode=0 Nov 01 20:57:35.917503 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:35.916957 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr" event={"ID":"ce125f0b-43ab-40bf-ae39-5dc6e195ce39","Type":"ContainerDied","Data":"f3adf3c2d14d3aac70cf5721d09c367dfad1b21a4e8b37014fbbf98c8b10e37f"} Nov 01 20:57:35.917503 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:35.917060 4484 scope.go:122] "RemoveContainer" containerID="4207d5f3773e1840c07c8df01b337d98418cd7d3b1813cc705a5e1673dfa7760" Nov 01 20:57:35.919104 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:35.919069 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m" event={"ID":"7ecec0b0-7e22-4803-891c-0a75173445ad","Type":"ContainerStarted","Data":"205e260be2d92a416298051fc1b803f8de077fbe104b1b53c437f032521d2608"} Nov 01 20:57:35.919227 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:35.919119 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m" event={"ID":"7ecec0b0-7e22-4803-891c-0a75173445ad","Type":"ContainerStarted","Data":"16965be258b9ccf09afc7526afa6cb0b2c8034eea207516e2ddfe21ab1a06695"} Nov 01 20:57:35.919461 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:35.919438 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6" Nov 01 20:57:35.920462 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:35.920440 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6" Nov 01 20:57:36.041792 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:36.041731 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m" podStartSLOduration=3.041715391 podStartE2EDuration="3.041715391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:57:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:57:36.038386966 +0000 UTC m=+1606.816027102" watchObservedRunningTime="2026-11-01 20:57:36.041715391 +0000 UTC m=+1606.819355567" Nov 01 20:57:36.085039 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:36.084936 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr" Nov 01 20:57:36.192035 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:36.190160 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-istio-podinfo\") pod \"ce125f0b-43ab-40bf-ae39-5dc6e195ce39\" (UID: \"ce125f0b-43ab-40bf-ae39-5dc6e195ce39\") " Nov 01 20:57:36.192035 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:36.190336 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-istio-token\" (UniqueName: \"kubernetes.io/projected/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-istio-token\") pod \"ce125f0b-43ab-40bf-ae39-5dc6e195ce39\" (UID: \"ce125f0b-43ab-40bf-ae39-5dc6e195ce39\") " Nov 01 20:57:36.192035 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:36.190377 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-workload-socket\") pod \"ce125f0b-43ab-40bf-ae39-5dc6e195ce39\" (UID: \"ce125f0b-43ab-40bf-ae39-5dc6e195ce39\") " Nov 01 20:57:36.192035 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:36.190421 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-credential-socket\") pod \"ce125f0b-43ab-40bf-ae39-5dc6e195ce39\" (UID: \"ce125f0b-43ab-40bf-ae39-5dc6e195ce39\") " Nov 01 20:57:36.192035 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:36.190538 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-istiod-ca-cert\") pod \"ce125f0b-43ab-40bf-ae39-5dc6e195ce39\" (UID: \"ce125f0b-43ab-40bf-ae39-5dc6e195ce39\") " Nov 01 20:57:36.192035 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:36.190707 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-istio-envoy\") pod \"ce125f0b-43ab-40bf-ae39-5dc6e195ce39\" (UID: \"ce125f0b-43ab-40bf-ae39-5dc6e195ce39\") " Nov 01 20:57:36.192035 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:36.190734 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-istio-data\") pod \"ce125f0b-43ab-40bf-ae39-5dc6e195ce39\" (UID: \"ce125f0b-43ab-40bf-ae39-5dc6e195ce39\") " Nov 01 20:57:36.192035 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:36.190785 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-workload-certs\") pod \"ce125f0b-43ab-40bf-ae39-5dc6e195ce39\" (UID: \"ce125f0b-43ab-40bf-ae39-5dc6e195ce39\") " Nov 01 20:57:36.192035 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:36.190848 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-kube-api-access-r4mg4\" (UniqueName: \"kubernetes.io/projected/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-kube-api-access-r4mg4\") pod \"ce125f0b-43ab-40bf-ae39-5dc6e195ce39\" (UID: \"ce125f0b-43ab-40bf-ae39-5dc6e195ce39\") " Nov 01 20:57:36.207458 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:36.207385 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-istiod-ca-cert" pod "ce125f0b-43ab-40bf-ae39-5dc6e195ce39" (UID: "ce125f0b-43ab-40bf-ae39-5dc6e195ce39"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:57:36.216523 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:36.216451 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-credential-socket" pod "ce125f0b-43ab-40bf-ae39-5dc6e195ce39" (UID: "ce125f0b-43ab-40bf-ae39-5dc6e195ce39"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:57:36.226236 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:36.218198 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-workload-certs" pod "ce125f0b-43ab-40bf-ae39-5dc6e195ce39" (UID: "ce125f0b-43ab-40bf-ae39-5dc6e195ce39"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:57:36.226943 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:36.226674 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-istio-podinfo" pod "ce125f0b-43ab-40bf-ae39-5dc6e195ce39" (UID: "ce125f0b-43ab-40bf-ae39-5dc6e195ce39"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:57:36.226943 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:36.226875 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-workload-socket" pod "ce125f0b-43ab-40bf-ae39-5dc6e195ce39" (UID: "ce125f0b-43ab-40bf-ae39-5dc6e195ce39"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:57:36.226943 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:36.226924 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-istio-data" pod "ce125f0b-43ab-40bf-ae39-5dc6e195ce39" (UID: "ce125f0b-43ab-40bf-ae39-5dc6e195ce39"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:57:36.265728 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:36.265647 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-istio-token" pod "ce125f0b-43ab-40bf-ae39-5dc6e195ce39" (UID: "ce125f0b-43ab-40bf-ae39-5dc6e195ce39"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:57:36.265957 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:36.265922 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-istio-envoy" pod "ce125f0b-43ab-40bf-ae39-5dc6e195ce39" (UID: "ce125f0b-43ab-40bf-ae39-5dc6e195ce39"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:57:36.271518 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:36.271429 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-kube-api-access-r4mg4" pod "ce125f0b-43ab-40bf-ae39-5dc6e195ce39" (UID: "ce125f0b-43ab-40bf-ae39-5dc6e195ce39"). InnerVolumeSpecName "kube-api-access-r4mg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:57:36.312981 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:36.300786 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:36.312981 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:36.300825 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:36.312981 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:36.300881 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:36.312981 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:36.300896 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:36.312981 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:36.300911 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:36.312981 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:36.300926 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:36.312981 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:36.300939 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:36.312981 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:36.300952 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:36.312981 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:36.300966 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4mg4\" (UniqueName: \"kubernetes.io/projected/ce125f0b-43ab-40bf-ae39-5dc6e195ce39-kube-api-access-r4mg4\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:36.462072 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:36.453397 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m" Nov 01 20:57:36.480555 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:36.480219 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m" Nov 01 20:57:36.928852 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:36.928825 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr" Nov 01 20:57:36.929341 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:36.928859 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr" event={"ID":"ce125f0b-43ab-40bf-ae39-5dc6e195ce39","Type":"ContainerDied","Data":"a8814ff4889808a6ff123242c98a49b1a8f37fda57358a2c60f4d2024887ce5d"} Nov 01 20:57:36.929341 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:36.928921 4484 scope.go:122] "RemoveContainer" containerID="f3adf3c2d14d3aac70cf5721d09c367dfad1b21a4e8b37014fbbf98c8b10e37f" Nov 01 20:57:36.929492 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:36.929408 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m" Nov 01 20:57:36.930586 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:36.930566 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m" Nov 01 20:57:37.104038 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:37.103975 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr"] Nov 01 20:57:37.154506 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:37.154470 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-8ddfb06d-gw-openshift-default-7b966bd4bf-88hxr"] Nov 01 20:57:37.374814 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:37.374784 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce125f0b-43ab-40bf-ae39-5dc6e195ce39" path="/var/lib/kubelet/pods/ce125f0b-43ab-40bf-ae39-5dc6e195ce39/volumes" Nov 01 20:57:37.462192 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:37.462078 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:37.986158 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:37.986122 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:38.162473 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:38.162439 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:38.340072 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:38.340032 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:38.781897 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:38.781872 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg" Nov 01 20:57:38.876638 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:38.876544 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8137e500-e107-49d5-81ba-948c7ae986f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8137e500-e107-49d5-81ba-948c7ae986f1-workload-socket\") pod \"8137e500-e107-49d5-81ba-948c7ae986f1\" (UID: \"8137e500-e107-49d5-81ba-948c7ae986f1\") " Nov 01 20:57:38.876818 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:38.876704 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8137e500-e107-49d5-81ba-948c7ae986f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8137e500-e107-49d5-81ba-948c7ae986f1-istiod-ca-cert\") pod \"8137e500-e107-49d5-81ba-948c7ae986f1\" (UID: \"8137e500-e107-49d5-81ba-948c7ae986f1\") " Nov 01 20:57:38.876818 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:38.876802 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8137e500-e107-49d5-81ba-948c7ae986f1-kube-api-access-ppj26\" (UniqueName: \"kubernetes.io/projected/8137e500-e107-49d5-81ba-948c7ae986f1-kube-api-access-ppj26\") pod \"8137e500-e107-49d5-81ba-948c7ae986f1\" (UID: \"8137e500-e107-49d5-81ba-948c7ae986f1\") " Nov 01 20:57:38.876930 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:38.876811 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8137e500-e107-49d5-81ba-948c7ae986f1-workload-socket" pod "8137e500-e107-49d5-81ba-948c7ae986f1" (UID: "8137e500-e107-49d5-81ba-948c7ae986f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:57:38.876930 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:38.876913 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8137e500-e107-49d5-81ba-948c7ae986f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8137e500-e107-49d5-81ba-948c7ae986f1-workload-certs\") pod \"8137e500-e107-49d5-81ba-948c7ae986f1\" (UID: \"8137e500-e107-49d5-81ba-948c7ae986f1\") " Nov 01 20:57:38.877048 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:38.876961 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8137e500-e107-49d5-81ba-948c7ae986f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8137e500-e107-49d5-81ba-948c7ae986f1-istio-data\") pod \"8137e500-e107-49d5-81ba-948c7ae986f1\" (UID: \"8137e500-e107-49d5-81ba-948c7ae986f1\") " Nov 01 20:57:38.877048 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:38.877026 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8137e500-e107-49d5-81ba-948c7ae986f1-istio-token\" (UniqueName: \"kubernetes.io/projected/8137e500-e107-49d5-81ba-948c7ae986f1-istio-token\") pod \"8137e500-e107-49d5-81ba-948c7ae986f1\" (UID: \"8137e500-e107-49d5-81ba-948c7ae986f1\") " Nov 01 20:57:38.877160 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:38.877080 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8137e500-e107-49d5-81ba-948c7ae986f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8137e500-e107-49d5-81ba-948c7ae986f1-credential-socket\") pod \"8137e500-e107-49d5-81ba-948c7ae986f1\" (UID: \"8137e500-e107-49d5-81ba-948c7ae986f1\") " Nov 01 20:57:38.877160 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:38.877110 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8137e500-e107-49d5-81ba-948c7ae986f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8137e500-e107-49d5-81ba-948c7ae986f1-istio-podinfo\") pod \"8137e500-e107-49d5-81ba-948c7ae986f1\" (UID: \"8137e500-e107-49d5-81ba-948c7ae986f1\") " Nov 01 20:57:38.877266 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:38.877158 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8137e500-e107-49d5-81ba-948c7ae986f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8137e500-e107-49d5-81ba-948c7ae986f1-istio-envoy\") pod \"8137e500-e107-49d5-81ba-948c7ae986f1\" (UID: \"8137e500-e107-49d5-81ba-948c7ae986f1\") " Nov 01 20:57:38.877266 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:38.877234 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8137e500-e107-49d5-81ba-948c7ae986f1-workload-certs" pod "8137e500-e107-49d5-81ba-948c7ae986f1" (UID: "8137e500-e107-49d5-81ba-948c7ae986f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:57:38.877362 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:38.877277 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8137e500-e107-49d5-81ba-948c7ae986f1-istio-data" pod "8137e500-e107-49d5-81ba-948c7ae986f1" (UID: "8137e500-e107-49d5-81ba-948c7ae986f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:57:38.877482 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:38.877434 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8137e500-e107-49d5-81ba-948c7ae986f1-credential-socket" pod "8137e500-e107-49d5-81ba-948c7ae986f1" (UID: "8137e500-e107-49d5-81ba-948c7ae986f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:57:38.877552 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:38.877479 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8137e500-e107-49d5-81ba-948c7ae986f1-istiod-ca-cert" pod "8137e500-e107-49d5-81ba-948c7ae986f1" (UID: "8137e500-e107-49d5-81ba-948c7ae986f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:57:38.878071 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:38.878043 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8137e500-e107-49d5-81ba-948c7ae986f1-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:38.878071 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:38.878073 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8137e500-e107-49d5-81ba-948c7ae986f1-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:38.878260 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:38.878090 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8137e500-e107-49d5-81ba-948c7ae986f1-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:38.878260 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:38.878104 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8137e500-e107-49d5-81ba-948c7ae986f1-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:38.878260 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:38.878117 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8137e500-e107-49d5-81ba-948c7ae986f1-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:38.879421 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:38.879397 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8137e500-e107-49d5-81ba-948c7ae986f1-istio-podinfo" pod "8137e500-e107-49d5-81ba-948c7ae986f1" (UID: "8137e500-e107-49d5-81ba-948c7ae986f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:57:38.879421 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:38.879404 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8137e500-e107-49d5-81ba-948c7ae986f1-istio-envoy" pod "8137e500-e107-49d5-81ba-948c7ae986f1" (UID: "8137e500-e107-49d5-81ba-948c7ae986f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:57:38.879602 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:38.879579 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8137e500-e107-49d5-81ba-948c7ae986f1-kube-api-access-ppj26" pod "8137e500-e107-49d5-81ba-948c7ae986f1" (UID: "8137e500-e107-49d5-81ba-948c7ae986f1"). InnerVolumeSpecName "kube-api-access-ppj26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:57:38.879683 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:38.879644 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8137e500-e107-49d5-81ba-948c7ae986f1-istio-token" pod "8137e500-e107-49d5-81ba-948c7ae986f1" (UID: "8137e500-e107-49d5-81ba-948c7ae986f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:57:38.908767 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:38.908733 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:38.949947 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:38.949907 4484 generic.go:356] "Generic (PLEG): container finished" podID="8137e500-e107-49d5-81ba-948c7ae986f1" containerID="a2e6c67edb3b9a3cc7729e33e2f7e1f38315d93b3ee4df918886d9cb59ca5650" exitCode=0 Nov 01 20:57:38.950165 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:38.949964 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg" Nov 01 20:57:38.950165 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:38.950008 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg" event={"ID":"8137e500-e107-49d5-81ba-948c7ae986f1","Type":"ContainerDied","Data":"a2e6c67edb3b9a3cc7729e33e2f7e1f38315d93b3ee4df918886d9cb59ca5650"} Nov 01 20:57:38.950165 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:38.950103 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg" event={"ID":"8137e500-e107-49d5-81ba-948c7ae986f1","Type":"ContainerDied","Data":"fcbba7613da663d22034a2e494d85e47714b5884583ceee80803773ecc8f95d6"} Nov 01 20:57:38.950165 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:38.950131 4484 scope.go:122] "RemoveContainer" containerID="a2e6c67edb3b9a3cc7729e33e2f7e1f38315d93b3ee4df918886d9cb59ca5650" Nov 01 20:57:38.964619 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:38.964600 4484 scope.go:122] "RemoveContainer" containerID="c76773bf6f98025744a313005c3c15dc5dc2e8f3e252a02daf73199676275fb0" Nov 01 20:57:38.976411 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:38.976382 4484 scope.go:122] "RemoveContainer" containerID="a2e6c67edb3b9a3cc7729e33e2f7e1f38315d93b3ee4df918886d9cb59ca5650" Nov 01 20:57:38.976800 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:57:38.976771 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2e6c67edb3b9a3cc7729e33e2f7e1f38315d93b3ee4df918886d9cb59ca5650\": container with ID starting with a2e6c67edb3b9a3cc7729e33e2f7e1f38315d93b3ee4df918886d9cb59ca5650 not found: ID does not exist" containerID="a2e6c67edb3b9a3cc7729e33e2f7e1f38315d93b3ee4df918886d9cb59ca5650" Nov 01 20:57:38.976919 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:38.976815 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2e6c67edb3b9a3cc7729e33e2f7e1f38315d93b3ee4df918886d9cb59ca5650"} err="failed to get container status \"a2e6c67edb3b9a3cc7729e33e2f7e1f38315d93b3ee4df918886d9cb59ca5650\": rpc error: code = NotFound desc = could not find container \"a2e6c67edb3b9a3cc7729e33e2f7e1f38315d93b3ee4df918886d9cb59ca5650\": container with ID starting with a2e6c67edb3b9a3cc7729e33e2f7e1f38315d93b3ee4df918886d9cb59ca5650 not found: ID does not exist" Nov 01 20:57:38.976919 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:38.976841 4484 scope.go:122] "RemoveContainer" containerID="c76773bf6f98025744a313005c3c15dc5dc2e8f3e252a02daf73199676275fb0" Nov 01 20:57:38.977173 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:57:38.977148 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c76773bf6f98025744a313005c3c15dc5dc2e8f3e252a02daf73199676275fb0\": container with ID starting with c76773bf6f98025744a313005c3c15dc5dc2e8f3e252a02daf73199676275fb0 not found: ID does not exist" containerID="c76773bf6f98025744a313005c3c15dc5dc2e8f3e252a02daf73199676275fb0" Nov 01 20:57:38.977237 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:38.977177 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c76773bf6f98025744a313005c3c15dc5dc2e8f3e252a02daf73199676275fb0"} err="failed to get container status \"c76773bf6f98025744a313005c3c15dc5dc2e8f3e252a02daf73199676275fb0\": rpc error: code = NotFound desc = could not find container \"c76773bf6f98025744a313005c3c15dc5dc2e8f3e252a02daf73199676275fb0\": container with ID starting with c76773bf6f98025744a313005c3c15dc5dc2e8f3e252a02daf73199676275fb0 not found: ID does not exist" Nov 01 20:57:38.979843 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:38.979644 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppj26\" (UniqueName: \"kubernetes.io/projected/8137e500-e107-49d5-81ba-948c7ae986f1-kube-api-access-ppj26\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:38.979843 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:38.979671 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8137e500-e107-49d5-81ba-948c7ae986f1-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:38.979843 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:38.979687 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8137e500-e107-49d5-81ba-948c7ae986f1-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:38.979843 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:38.979702 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8137e500-e107-49d5-81ba-948c7ae986f1-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:39.128121 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:39.128042 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg"] Nov 01 20:57:39.129299 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:39.129272 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:39.132593 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:39.132566 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-74ab02fe-openshift-default-577bf6d567-wqvkg"] Nov 01 20:57:39.294796 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:39.294765 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:39.376408 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:39.376381 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8137e500-e107-49d5-81ba-948c7ae986f1" path="/var/lib/kubelet/pods/8137e500-e107-49d5-81ba-948c7ae986f1/volumes" Nov 01 20:57:41.333178 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:41.333130 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:41.577479 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:41.577440 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:41.947129 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:41.937341 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-de8145-9dcc65c7f-q6cff"] Nov 01 20:57:41.947129 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:41.938481 4484 cpu_manager.go:414] "containerMap: removing container" podUID="8137e500-e107-49d5-81ba-948c7ae986f1" containerName="istio-proxy" Nov 01 20:57:41.947129 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:41.938503 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8137e500-e107-49d5-81ba-948c7ae986f1" containerName="istio-proxy" Nov 01 20:57:41.947129 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:41.938521 4484 cpu_manager.go:414] "containerMap: removing container" podUID="8137e500-e107-49d5-81ba-948c7ae986f1" containerName="istio-proxy" Nov 01 20:57:41.947129 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:41.938528 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8137e500-e107-49d5-81ba-948c7ae986f1" containerName="istio-proxy" Nov 01 20:57:41.947129 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:41.938541 4484 cpu_manager.go:414] "containerMap: removing container" podUID="3d20fc4c-5175-4154-8669-2494376150ce" containerName="istio-proxy" Nov 01 20:57:41.947129 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:41.938548 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d20fc4c-5175-4154-8669-2494376150ce" containerName="istio-proxy" Nov 01 20:57:41.947129 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:41.938572 4484 cpu_manager.go:414] "containerMap: removing container" podUID="3d20fc4c-5175-4154-8669-2494376150ce" containerName="istio-proxy" Nov 01 20:57:41.947129 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:41.938579 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d20fc4c-5175-4154-8669-2494376150ce" containerName="istio-proxy" Nov 01 20:57:41.947129 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:41.938594 4484 cpu_manager.go:414] "containerMap: removing container" podUID="ce125f0b-43ab-40bf-ae39-5dc6e195ce39" containerName="istio-proxy" Nov 01 20:57:41.947129 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:41.938599 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce125f0b-43ab-40bf-ae39-5dc6e195ce39" containerName="istio-proxy" Nov 01 20:57:41.960294 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:41.960259 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="8137e500-e107-49d5-81ba-948c7ae986f1" containerName="istio-proxy" Nov 01 20:57:41.960294 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:41.960298 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="8137e500-e107-49d5-81ba-948c7ae986f1" containerName="istio-proxy" Nov 01 20:57:41.960513 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:41.960310 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d20fc4c-5175-4154-8669-2494376150ce" containerName="istio-proxy" Nov 01 20:57:41.960513 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:41.960325 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d20fc4c-5175-4154-8669-2494376150ce" containerName="istio-proxy" Nov 01 20:57:41.960513 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:41.960335 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce125f0b-43ab-40bf-ae39-5dc6e195ce39" containerName="istio-proxy" Nov 01 20:57:41.960513 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:41.960345 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce125f0b-43ab-40bf-ae39-5dc6e195ce39" containerName="istio-proxy" Nov 01 20:57:41.964930 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:41.964902 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-de8145-9dcc65c7f-q6cff" Nov 01 20:57:42.028128 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:42.019976 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-de8145\"" Nov 01 20:57:42.058027 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:42.055087 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d895fa0a-1661-4924-80fc-b05b07017767-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-de8145-9dcc65c7f-q6cff\" (UID: \"d895fa0a-1661-4924-80fc-b05b07017767\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-de8145-9dcc65c7f-q6cff" Nov 01 20:57:42.058027 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:42.055142 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j4m2\" (UniqueName: \"kubernetes.io/projected/d895fa0a-1661-4924-80fc-b05b07017767-kube-api-access-5j4m2\") pod \"payload-processing-e2e-shared-a-w0-de8145-9dcc65c7f-q6cff\" (UID: \"d895fa0a-1661-4924-80fc-b05b07017767\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-de8145-9dcc65c7f-q6cff" Nov 01 20:57:42.092134 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:42.090149 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-de8145-9dcc65c7f-q6cff"] Nov 01 20:57:42.092134 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:42.091028 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-de8145-56fb74df5-8nbgw"] Nov 01 20:57:42.092393 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:42.092344 4484 cpu_manager.go:414] "containerMap: removing container" podUID="ce125f0b-43ab-40bf-ae39-5dc6e195ce39" containerName="istio-proxy" Nov 01 20:57:42.092393 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:42.092366 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce125f0b-43ab-40bf-ae39-5dc6e195ce39" containerName="istio-proxy" Nov 01 20:57:42.121800 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:42.121763 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-de8145-dockercfg-lgnpf\"" Nov 01 20:57:42.158018 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:42.156372 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d895fa0a-1661-4924-80fc-b05b07017767-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-de8145-9dcc65c7f-q6cff\" (UID: \"d895fa0a-1661-4924-80fc-b05b07017767\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-de8145-9dcc65c7f-q6cff" Nov 01 20:57:42.158018 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:42.156425 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j4m2\" (UniqueName: \"kubernetes.io/projected/d895fa0a-1661-4924-80fc-b05b07017767-kube-api-access-5j4m2\") pod \"payload-processing-e2e-shared-a-w0-de8145-9dcc65c7f-q6cff\" (UID: \"d895fa0a-1661-4924-80fc-b05b07017767\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-de8145-9dcc65c7f-q6cff" Nov 01 20:57:42.167241 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:42.167206 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d895fa0a-1661-4924-80fc-b05b07017767-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-de8145-9dcc65c7f-q6cff\" (UID: \"d895fa0a-1661-4924-80fc-b05b07017767\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-de8145-9dcc65c7f-q6cff" Nov 01 20:57:42.169663 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:42.167838 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-de8145-56fb74df5-8nbgw" Nov 01 20:57:42.237893 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:42.237853 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:42.291508 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:42.291469 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr6kx\" (UniqueName: \"kubernetes.io/projected/6a7f6c58-4b9f-4c61-abe2-75e1f9fba161-kube-api-access-fr6kx\") pod \"payload-pre-processing-e2e-shared-a-w0-de8145-56fb74df5-8nbgw\" (UID: \"6a7f6c58-4b9f-4c61-abe2-75e1f9fba161\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-de8145-56fb74df5-8nbgw" Nov 01 20:57:42.293900 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:42.293871 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a7f6c58-4b9f-4c61-abe2-75e1f9fba161-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-de8145-56fb74df5-8nbgw\" (UID: \"6a7f6c58-4b9f-4c61-abe2-75e1f9fba161\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-de8145-56fb74df5-8nbgw" Nov 01 20:57:42.327550 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:42.326045 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-de8145-56fb74df5-8nbgw"] Nov 01 20:57:42.400753 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:42.400710 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j4m2\" (UniqueName: \"kubernetes.io/projected/d895fa0a-1661-4924-80fc-b05b07017767-kube-api-access-5j4m2\") pod \"payload-processing-e2e-shared-a-w0-de8145-9dcc65c7f-q6cff\" (UID: \"d895fa0a-1661-4924-80fc-b05b07017767\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-de8145-9dcc65c7f-q6cff" Nov 01 20:57:42.412923 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:42.408622 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr6kx\" (UniqueName: \"kubernetes.io/projected/6a7f6c58-4b9f-4c61-abe2-75e1f9fba161-kube-api-access-fr6kx\") pod \"payload-pre-processing-e2e-shared-a-w0-de8145-56fb74df5-8nbgw\" (UID: \"6a7f6c58-4b9f-4c61-abe2-75e1f9fba161\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-de8145-56fb74df5-8nbgw" Nov 01 20:57:42.412923 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:42.408723 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a7f6c58-4b9f-4c61-abe2-75e1f9fba161-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-de8145-56fb74df5-8nbgw\" (UID: \"6a7f6c58-4b9f-4c61-abe2-75e1f9fba161\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-de8145-56fb74df5-8nbgw" Nov 01 20:57:42.412923 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:42.411364 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a7f6c58-4b9f-4c61-abe2-75e1f9fba161-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-de8145-56fb74df5-8nbgw\" (UID: \"6a7f6c58-4b9f-4c61-abe2-75e1f9fba161\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-de8145-56fb74df5-8nbgw" Nov 01 20:57:42.469246 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:42.469214 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr6kx\" (UniqueName: \"kubernetes.io/projected/6a7f6c58-4b9f-4c61-abe2-75e1f9fba161-kube-api-access-fr6kx\") pod \"payload-pre-processing-e2e-shared-a-w0-de8145-56fb74df5-8nbgw\" (UID: \"6a7f6c58-4b9f-4c61-abe2-75e1f9fba161\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-de8145-56fb74df5-8nbgw" Nov 01 20:57:42.598978 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:42.598938 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-de8145-56fb74df5-8nbgw" Nov 01 20:57:42.678966 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:42.678936 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-de8145-9dcc65c7f-q6cff" Nov 01 20:57:42.999467 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:57:42.999437 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a7f6c58_4b9f_4c61_abe2_75e1f9fba161.slice/crio-b9a4df89ef9cf207537964aae57b118d2782e06d31292f4bfeedb2e15b2052eb WatchSource:0}: Error finding container b9a4df89ef9cf207537964aae57b118d2782e06d31292f4bfeedb2e15b2052eb: Status 404 returned error can't find the container with id b9a4df89ef9cf207537964aae57b118d2782e06d31292f4bfeedb2e15b2052eb Nov 01 20:57:42.999780 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:42.999758 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-de8145-56fb74df5-8nbgw"] Nov 01 20:57:43.040665 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:43.040634 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:43.123080 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:43.123046 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-de8145-9dcc65c7f-q6cff"] Nov 01 20:57:43.235830 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:43.235792 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-de8145-56fb74df5-8nbgw" event={"ID":"6a7f6c58-4b9f-4c61-abe2-75e1f9fba161","Type":"ContainerStarted","Data":"3c6fbb2cc68d4c8c7cda0d36276d88e0ac63579ea1c8c64ef70af6c17a5b7c05"} Nov 01 20:57:43.236033 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:43.235855 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-de8145-56fb74df5-8nbgw" event={"ID":"6a7f6c58-4b9f-4c61-abe2-75e1f9fba161","Type":"ContainerStarted","Data":"b9a4df89ef9cf207537964aae57b118d2782e06d31292f4bfeedb2e15b2052eb"} Nov 01 20:57:43.236033 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:43.235881 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-de8145-56fb74df5-8nbgw" Nov 01 20:57:43.237202 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:43.237177 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-de8145-9dcc65c7f-q6cff" event={"ID":"d895fa0a-1661-4924-80fc-b05b07017767","Type":"ContainerStarted","Data":"bb5eac3b4743f7469978d8972eaa33a522150afd1ced85f1ab48c3be95f7f130"} Nov 01 20:57:43.351462 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:43.351405 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-de8145-56fb74df5-8nbgw" podStartSLOduration=1.351390172 podStartE2EDuration="1.351390172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:57:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:57:43.349536856 +0000 UTC m=+1614.127176992" watchObservedRunningTime="2026-11-01 20:57:43.351390172 +0000 UTC m=+1614.129030306" Nov 01 20:57:44.200430 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:44.200392 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc"] Nov 01 20:57:44.200856 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:44.200664 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc" podUID="5142a1b6-ef5b-4381-ae7d-676a82db925a" containerName="istio-proxy" containerID="cri-o://1d1c0c13cacb2c3f418350d97268ea4fb0feebe629d99eca3f73bcb3af17ff7d" gracePeriod=30 Nov 01 20:57:44.250051 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:44.250008 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-de8145-9dcc65c7f-q6cff" event={"ID":"d895fa0a-1661-4924-80fc-b05b07017767","Type":"ContainerStarted","Data":"d3279eee1c0b0e85e2cdb48591a025e2b6d058d91bbc1aa7dc4a453a02e674f0"} Nov 01 20:57:44.250310 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:44.250092 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-de8145-9dcc65c7f-q6cff" Nov 01 20:57:44.446475 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:44.446420 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-de8145-9dcc65c7f-q6cff" podStartSLOduration=3.446405045 podStartE2EDuration="3.446405045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:57:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:57:44.442650912 +0000 UTC m=+1615.220291049" watchObservedRunningTime="2026-11-01 20:57:44.446405045 +0000 UTC m=+1615.224045179" Nov 01 20:57:45.768424 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:57:45.768387 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 20:57:46.332506 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:46.332474 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:46.566371 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:46.566303 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:46.727065 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:46.721501 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f620899-69lws"] Nov 01 20:57:46.888019 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:46.865248 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f620899-69lws"] Nov 01 20:57:46.888019 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:46.865374 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f620899-69lws" Nov 01 20:57:46.929935 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:46.929885 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 20:57:46.931260 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:46.931238 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:57:46.931576 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:46.931551 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:57:46.931772 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:46.931739 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:57:46.945263 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:46.943061 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a9b439-76df769c99-g6df9"] Nov 01 20:57:47.011643 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:47.003982 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a9b439-76df769c99-g6df9"] Nov 01 20:57:47.011643 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:47.004054 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-a9b439-76bf5ff49-qqkwc"] Nov 01 20:57:47.054030 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:47.012098 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a9b439-76df769c99-g6df9" Nov 01 20:57:47.054030 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:47.012829 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a9b439-76bf5ff49-qqkwc" Nov 01 20:57:47.113669 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:47.079689 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-a9b439-dockercfg-7wgq4\"" Nov 01 20:57:47.113669 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:47.079785 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-a9b439\"" Nov 01 20:57:47.113669 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:47.085297 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-a9b439-76bf5ff49-qqkwc"] Nov 01 20:57:47.113669 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:47.086459 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqm95\" (UniqueName: \"kubernetes.io/projected/8195e66f-4ed3-44f2-ab9c-3ff8881e428c-kube-api-access-jqm95\") pod \"payload-processing-e2e-shared-a-w4-a9b439-76bf5ff49-qqkwc\" (UID: \"8195e66f-4ed3-44f2-ab9c-3ff8881e428c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a9b439-76bf5ff49-qqkwc" Nov 01 20:57:47.113669 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:47.086532 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdmxs\" (UniqueName: \"kubernetes.io/projected/4e4385fa-bbd4-4fc5-b40d-43f27d0ae3ac-kube-api-access-rdmxs\") pod \"payload-pre-processing-e2e-shared-a-w4-a9b439-76df769c99-g6df9\" (UID: \"4e4385fa-bbd4-4fc5-b40d-43f27d0ae3ac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a9b439-76df769c99-g6df9" Nov 01 20:57:47.113669 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:47.086805 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8195e66f-4ed3-44f2-ab9c-3ff8881e428c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-a9b439-76bf5ff49-qqkwc\" (UID: \"8195e66f-4ed3-44f2-ab9c-3ff8881e428c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a9b439-76bf5ff49-qqkwc" Nov 01 20:57:47.113669 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:47.086880 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4385fa-bbd4-4fc5-b40d-43f27d0ae3ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-a9b439-76df769c99-g6df9\" (UID: \"4e4385fa-bbd4-4fc5-b40d-43f27d0ae3ac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a9b439-76df769c99-g6df9" Nov 01 20:57:47.113669 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:47.086908 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a7deca0-b10e-4b93-b0c8-9721b9db23b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f620899-69lws\" (UID: \"1a7deca0-b10e-4b93-b0c8-9721b9db23b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f620899-69lws" Nov 01 20:57:47.113669 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:47.087199 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxsl8\" (UniqueName: \"kubernetes.io/projected/1a7deca0-b10e-4b93-b0c8-9721b9db23b7-kube-api-access-nxsl8\") pod \"maas-api-revoke-sub-2f620899-69lws\" (UID: \"1a7deca0-b10e-4b93-b0c8-9721b9db23b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f620899-69lws" Nov 01 20:57:47.113669 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:47.087238 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a7deca0-b10e-4b93-b0c8-9721b9db23b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f620899-69lws\" (UID: \"1a7deca0-b10e-4b93-b0c8-9721b9db23b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f620899-69lws" Nov 01 20:57:47.225027 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:47.195846 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqm95\" (UniqueName: \"kubernetes.io/projected/8195e66f-4ed3-44f2-ab9c-3ff8881e428c-kube-api-access-jqm95\") pod \"payload-processing-e2e-shared-a-w4-a9b439-76bf5ff49-qqkwc\" (UID: \"8195e66f-4ed3-44f2-ab9c-3ff8881e428c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a9b439-76bf5ff49-qqkwc" Nov 01 20:57:47.225027 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:47.195933 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdmxs\" (UniqueName: \"kubernetes.io/projected/4e4385fa-bbd4-4fc5-b40d-43f27d0ae3ac-kube-api-access-rdmxs\") pod \"payload-pre-processing-e2e-shared-a-w4-a9b439-76df769c99-g6df9\" (UID: \"4e4385fa-bbd4-4fc5-b40d-43f27d0ae3ac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a9b439-76df769c99-g6df9" Nov 01 20:57:47.245636 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:47.225806 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8195e66f-4ed3-44f2-ab9c-3ff8881e428c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-a9b439-76bf5ff49-qqkwc\" (UID: \"8195e66f-4ed3-44f2-ab9c-3ff8881e428c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a9b439-76bf5ff49-qqkwc" Nov 01 20:57:47.245636 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:47.225922 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4385fa-bbd4-4fc5-b40d-43f27d0ae3ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-a9b439-76df769c99-g6df9\" (UID: \"4e4385fa-bbd4-4fc5-b40d-43f27d0ae3ac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a9b439-76df769c99-g6df9" Nov 01 20:57:47.245636 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:47.225949 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a7deca0-b10e-4b93-b0c8-9721b9db23b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f620899-69lws\" (UID: \"1a7deca0-b10e-4b93-b0c8-9721b9db23b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f620899-69lws" Nov 01 20:57:47.245636 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:47.226140 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxsl8\" (UniqueName: \"kubernetes.io/projected/1a7deca0-b10e-4b93-b0c8-9721b9db23b7-kube-api-access-nxsl8\") pod \"maas-api-revoke-sub-2f620899-69lws\" (UID: \"1a7deca0-b10e-4b93-b0c8-9721b9db23b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f620899-69lws" Nov 01 20:57:47.245636 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:47.226179 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a7deca0-b10e-4b93-b0c8-9721b9db23b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f620899-69lws\" (UID: \"1a7deca0-b10e-4b93-b0c8-9721b9db23b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f620899-69lws" Nov 01 20:57:47.245636 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:47.227222 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a7deca0-b10e-4b93-b0c8-9721b9db23b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f620899-69lws\" (UID: \"1a7deca0-b10e-4b93-b0c8-9721b9db23b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f620899-69lws" Nov 01 20:57:47.245636 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:47.228112 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8195e66f-4ed3-44f2-ab9c-3ff8881e428c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-a9b439-76bf5ff49-qqkwc\" (UID: \"8195e66f-4ed3-44f2-ab9c-3ff8881e428c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a9b439-76bf5ff49-qqkwc" Nov 01 20:57:47.245636 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:47.228918 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4385fa-bbd4-4fc5-b40d-43f27d0ae3ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-a9b439-76df769c99-g6df9\" (UID: \"4e4385fa-bbd4-4fc5-b40d-43f27d0ae3ac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a9b439-76df769c99-g6df9" Nov 01 20:57:47.396440 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:47.358776 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxsl8\" (UniqueName: \"kubernetes.io/projected/1a7deca0-b10e-4b93-b0c8-9721b9db23b7-kube-api-access-nxsl8\") pod \"maas-api-revoke-sub-2f620899-69lws\" (UID: \"1a7deca0-b10e-4b93-b0c8-9721b9db23b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f620899-69lws" Nov 01 20:57:47.456925 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:47.455535 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a7deca0-b10e-4b93-b0c8-9721b9db23b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f620899-69lws\" (UID: \"1a7deca0-b10e-4b93-b0c8-9721b9db23b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f620899-69lws" Nov 01 20:57:47.500945 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:47.500914 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqm95\" (UniqueName: \"kubernetes.io/projected/8195e66f-4ed3-44f2-ab9c-3ff8881e428c-kube-api-access-jqm95\") pod \"payload-processing-e2e-shared-a-w4-a9b439-76bf5ff49-qqkwc\" (UID: \"8195e66f-4ed3-44f2-ab9c-3ff8881e428c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a9b439-76bf5ff49-qqkwc" Nov 01 20:57:47.503397 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:47.503367 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdmxs\" (UniqueName: \"kubernetes.io/projected/4e4385fa-bbd4-4fc5-b40d-43f27d0ae3ac-kube-api-access-rdmxs\") pod \"payload-pre-processing-e2e-shared-a-w4-a9b439-76df769c99-g6df9\" (UID: \"4e4385fa-bbd4-4fc5-b40d-43f27d0ae3ac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a9b439-76df769c99-g6df9" Nov 01 20:57:47.517173 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:47.517135 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f620899-69lws" Nov 01 20:57:47.569717 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:47.569684 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a9b439-76bf5ff49-qqkwc" Nov 01 20:57:47.759871 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:47.759838 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a9b439-76df769c99-g6df9" Nov 01 20:57:47.904277 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:57:47.904241 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a7deca0_b10e_4b93_b0c8_9721b9db23b7.slice/crio-c64eaf71847662228d782842bd6765ea6cbca61a9392a532c3eaf0c78d744236 WatchSource:0}: Error finding container c64eaf71847662228d782842bd6765ea6cbca61a9392a532c3eaf0c78d744236: Status 404 returned error can't find the container with id c64eaf71847662228d782842bd6765ea6cbca61a9392a532c3eaf0c78d744236 Nov 01 20:57:47.905684 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:47.905658 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f620899-69lws"] Nov 01 20:57:48.012732 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:48.012709 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-a9b439-76bf5ff49-qqkwc"] Nov 01 20:57:48.106201 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:57:48.106161 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8195e66f_4ed3_44f2_ab9c_3ff8881e428c.slice/crio-ee956cc41e4c0396c39c6dd6b248183391d41725f29ed4664522f4f063888f6b WatchSource:0}: Error finding container ee956cc41e4c0396c39c6dd6b248183391d41725f29ed4664522f4f063888f6b: Status 404 returned error can't find the container with id ee956cc41e4c0396c39c6dd6b248183391d41725f29ed4664522f4f063888f6b Nov 01 20:57:48.141518 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:48.141491 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a9b439-76df769c99-g6df9"] Nov 01 20:57:48.141638 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:57:48.141489 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e4385fa_bbd4_4fc5_b40d_43f27d0ae3ac.slice/crio-ccbb04f375fea15624dc9dc24ae59b5c981fa3161df6b932fa8dbfa81ee45ab4 WatchSource:0}: Error finding container ccbb04f375fea15624dc9dc24ae59b5c981fa3161df6b932fa8dbfa81ee45ab4: Status 404 returned error can't find the container with id ccbb04f375fea15624dc9dc24ae59b5c981fa3161df6b932fa8dbfa81ee45ab4 Nov 01 20:57:48.381611 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:48.381576 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:48.506304 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:48.506265 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f620899-69lws" event={"ID":"1a7deca0-b10e-4b93-b0c8-9721b9db23b7","Type":"ContainerStarted","Data":"acc625b318bfa109bad380efb5b3ff9d5259e7261ce409704112f0380829e56b"} Nov 01 20:57:48.506662 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:48.506634 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f620899-69lws" event={"ID":"1a7deca0-b10e-4b93-b0c8-9721b9db23b7","Type":"ContainerStarted","Data":"c64eaf71847662228d782842bd6765ea6cbca61a9392a532c3eaf0c78d744236"} Nov 01 20:57:48.543167 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:48.543122 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a9b439-76bf5ff49-qqkwc" event={"ID":"8195e66f-4ed3-44f2-ab9c-3ff8881e428c","Type":"ContainerStarted","Data":"27d12065d89e35f17d959635b9447332d78abf1853fcefebd471169c49f116b0"} Nov 01 20:57:48.559233 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:48.559081 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a9b439-76bf5ff49-qqkwc" Nov 01 20:57:48.559530 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:48.559502 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a9b439-76bf5ff49-qqkwc" event={"ID":"8195e66f-4ed3-44f2-ab9c-3ff8881e428c","Type":"ContainerStarted","Data":"ee956cc41e4c0396c39c6dd6b248183391d41725f29ed4664522f4f063888f6b"} Nov 01 20:57:48.564100 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:48.564070 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a9b439-76df769c99-g6df9" event={"ID":"4e4385fa-bbd4-4fc5-b40d-43f27d0ae3ac","Type":"ContainerStarted","Data":"9d9577e8f2b4f42bca622d0740f5e8f5ec4f3738e67e48ad61e301037ba4e270"} Nov 01 20:57:48.564250 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:48.564114 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a9b439-76df769c99-g6df9" event={"ID":"4e4385fa-bbd4-4fc5-b40d-43f27d0ae3ac","Type":"ContainerStarted","Data":"ccbb04f375fea15624dc9dc24ae59b5c981fa3161df6b932fa8dbfa81ee45ab4"} Nov 01 20:57:48.564250 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:48.564215 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a9b439-76df769c99-g6df9" Nov 01 20:57:48.625598 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:48.625547 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f620899-69lws" podStartSLOduration=2.625533482 podStartE2EDuration="2.625533482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:57:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:57:48.622617472 +0000 UTC m=+1619.400257609" watchObservedRunningTime="2026-11-01 20:57:48.625533482 +0000 UTC m=+1619.403173822" Nov 01 20:57:48.741049 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:48.740954 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a9b439-76df769c99-g6df9" podStartSLOduration=2.740940995 podStartE2EDuration="2.740940995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:57:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:57:48.737240279 +0000 UTC m=+1619.514880413" watchObservedRunningTime="2026-11-01 20:57:48.740940995 +0000 UTC m=+1619.518581160" Nov 01 20:57:48.954451 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:48.949611 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a9b439-76bf5ff49-qqkwc" podStartSLOduration=2.949588327 podStartE2EDuration="2.949588327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:57:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:57:48.860591993 +0000 UTC m=+1619.638232129" watchObservedRunningTime="2026-11-01 20:57:48.949588327 +0000 UTC m=+1619.727228463" Nov 01 20:57:49.171417 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:49.164483 4484 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 20:57:49.171720 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:49.171674 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc" podUID="5142a1b6-ef5b-4381-ae7d-676a82db925a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 20:57:49.488176 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:49.488038 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:49.508172 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:49.508145 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc" Nov 01 20:57:49.573387 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:49.573355 4484 generic.go:356] "Generic (PLEG): container finished" podID="5142a1b6-ef5b-4381-ae7d-676a82db925a" containerID="1d1c0c13cacb2c3f418350d97268ea4fb0feebe629d99eca3f73bcb3af17ff7d" exitCode=0 Nov 01 20:57:49.573586 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:49.573404 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc" Nov 01 20:57:49.573586 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:49.573431 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc" event={"ID":"5142a1b6-ef5b-4381-ae7d-676a82db925a","Type":"ContainerDied","Data":"1d1c0c13cacb2c3f418350d97268ea4fb0feebe629d99eca3f73bcb3af17ff7d"} Nov 01 20:57:49.573586 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:49.573482 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc" event={"ID":"5142a1b6-ef5b-4381-ae7d-676a82db925a","Type":"ContainerDied","Data":"08785a5a6c5b39ac7abde82fe8ab3666d73093629911585be1f7ee5f2f06ae40"} Nov 01 20:57:49.573586 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:49.573503 4484 scope.go:122] "RemoveContainer" containerID="1d1c0c13cacb2c3f418350d97268ea4fb0feebe629d99eca3f73bcb3af17ff7d" Nov 01 20:57:49.575400 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:49.575373 4484 generic.go:356] "Generic (PLEG): container finished" podID="1a7deca0-b10e-4b93-b0c8-9721b9db23b7" containerID="acc625b318bfa109bad380efb5b3ff9d5259e7261ce409704112f0380829e56b" exitCode=0 Nov 01 20:57:49.575512 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:49.575452 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f620899-69lws" event={"ID":"1a7deca0-b10e-4b93-b0c8-9721b9db23b7","Type":"ContainerDied","Data":"acc625b318bfa109bad380efb5b3ff9d5259e7261ce409704112f0380829e56b"} Nov 01 20:57:49.586287 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:49.586270 4484 scope.go:122] "RemoveContainer" containerID="1d1c0c13cacb2c3f418350d97268ea4fb0feebe629d99eca3f73bcb3af17ff7d" Nov 01 20:57:49.586585 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:57:49.586563 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d1c0c13cacb2c3f418350d97268ea4fb0feebe629d99eca3f73bcb3af17ff7d\": container with ID starting with 1d1c0c13cacb2c3f418350d97268ea4fb0feebe629d99eca3f73bcb3af17ff7d not found: ID does not exist" containerID="1d1c0c13cacb2c3f418350d97268ea4fb0feebe629d99eca3f73bcb3af17ff7d" Nov 01 20:57:49.586646 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:49.586595 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d1c0c13cacb2c3f418350d97268ea4fb0feebe629d99eca3f73bcb3af17ff7d"} err="failed to get container status \"1d1c0c13cacb2c3f418350d97268ea4fb0feebe629d99eca3f73bcb3af17ff7d\": rpc error: code = NotFound desc = could not find container \"1d1c0c13cacb2c3f418350d97268ea4fb0feebe629d99eca3f73bcb3af17ff7d\": container with ID starting with 1d1c0c13cacb2c3f418350d97268ea4fb0feebe629d99eca3f73bcb3af17ff7d not found: ID does not exist" Nov 01 20:57:49.594036 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:49.593977 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5142a1b6-ef5b-4381-ae7d-676a82db925a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5142a1b6-ef5b-4381-ae7d-676a82db925a-workload-socket\") pod \"5142a1b6-ef5b-4381-ae7d-676a82db925a\" (UID: \"5142a1b6-ef5b-4381-ae7d-676a82db925a\") " Nov 01 20:57:49.594160 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:49.594107 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5142a1b6-ef5b-4381-ae7d-676a82db925a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5142a1b6-ef5b-4381-ae7d-676a82db925a-credential-socket\") pod \"5142a1b6-ef5b-4381-ae7d-676a82db925a\" (UID: \"5142a1b6-ef5b-4381-ae7d-676a82db925a\") " Nov 01 20:57:49.594160 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:49.594150 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5142a1b6-ef5b-4381-ae7d-676a82db925a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5142a1b6-ef5b-4381-ae7d-676a82db925a-istiod-ca-cert\") pod \"5142a1b6-ef5b-4381-ae7d-676a82db925a\" (UID: \"5142a1b6-ef5b-4381-ae7d-676a82db925a\") " Nov 01 20:57:49.594296 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:49.594269 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5142a1b6-ef5b-4381-ae7d-676a82db925a-kube-api-access-xwq9r\" (UniqueName: \"kubernetes.io/projected/5142a1b6-ef5b-4381-ae7d-676a82db925a-kube-api-access-xwq9r\") pod \"5142a1b6-ef5b-4381-ae7d-676a82db925a\" (UID: \"5142a1b6-ef5b-4381-ae7d-676a82db925a\") " Nov 01 20:57:49.594373 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:49.594268 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5142a1b6-ef5b-4381-ae7d-676a82db925a-workload-socket" pod "5142a1b6-ef5b-4381-ae7d-676a82db925a" (UID: "5142a1b6-ef5b-4381-ae7d-676a82db925a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:57:49.594431 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:49.594407 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5142a1b6-ef5b-4381-ae7d-676a82db925a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5142a1b6-ef5b-4381-ae7d-676a82db925a-workload-certs\") pod \"5142a1b6-ef5b-4381-ae7d-676a82db925a\" (UID: \"5142a1b6-ef5b-4381-ae7d-676a82db925a\") " Nov 01 20:57:49.594494 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:49.594438 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5142a1b6-ef5b-4381-ae7d-676a82db925a-credential-socket" pod "5142a1b6-ef5b-4381-ae7d-676a82db925a" (UID: "5142a1b6-ef5b-4381-ae7d-676a82db925a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:57:49.594494 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:49.594443 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5142a1b6-ef5b-4381-ae7d-676a82db925a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5142a1b6-ef5b-4381-ae7d-676a82db925a-istio-envoy\") pod \"5142a1b6-ef5b-4381-ae7d-676a82db925a\" (UID: \"5142a1b6-ef5b-4381-ae7d-676a82db925a\") " Nov 01 20:57:49.594621 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:49.594600 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5142a1b6-ef5b-4381-ae7d-676a82db925a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5142a1b6-ef5b-4381-ae7d-676a82db925a-istio-podinfo\") pod \"5142a1b6-ef5b-4381-ae7d-676a82db925a\" (UID: \"5142a1b6-ef5b-4381-ae7d-676a82db925a\") " Nov 01 20:57:49.594689 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:49.594652 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5142a1b6-ef5b-4381-ae7d-676a82db925a-workload-certs" pod "5142a1b6-ef5b-4381-ae7d-676a82db925a" (UID: "5142a1b6-ef5b-4381-ae7d-676a82db925a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:57:49.594689 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:49.594677 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5142a1b6-ef5b-4381-ae7d-676a82db925a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5142a1b6-ef5b-4381-ae7d-676a82db925a-istio-data\") pod \"5142a1b6-ef5b-4381-ae7d-676a82db925a\" (UID: \"5142a1b6-ef5b-4381-ae7d-676a82db925a\") " Nov 01 20:57:49.612851 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:49.594693 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5142a1b6-ef5b-4381-ae7d-676a82db925a-istiod-ca-cert" pod "5142a1b6-ef5b-4381-ae7d-676a82db925a" (UID: "5142a1b6-ef5b-4381-ae7d-676a82db925a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:57:49.612851 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:49.594729 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5142a1b6-ef5b-4381-ae7d-676a82db925a-istio-token\" (UniqueName: \"kubernetes.io/projected/5142a1b6-ef5b-4381-ae7d-676a82db925a-istio-token\") pod \"5142a1b6-ef5b-4381-ae7d-676a82db925a\" (UID: \"5142a1b6-ef5b-4381-ae7d-676a82db925a\") " Nov 01 20:57:49.612851 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:49.594948 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5142a1b6-ef5b-4381-ae7d-676a82db925a-istio-data" pod "5142a1b6-ef5b-4381-ae7d-676a82db925a" (UID: "5142a1b6-ef5b-4381-ae7d-676a82db925a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:57:49.612851 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:49.598029 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5142a1b6-ef5b-4381-ae7d-676a82db925a-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:49.612851 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:49.598053 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5142a1b6-ef5b-4381-ae7d-676a82db925a-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:49.612851 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:49.598067 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5142a1b6-ef5b-4381-ae7d-676a82db925a-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:49.612851 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:49.598079 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5142a1b6-ef5b-4381-ae7d-676a82db925a-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:49.612851 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:49.598094 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5142a1b6-ef5b-4381-ae7d-676a82db925a-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:49.612851 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:49.606589 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5142a1b6-ef5b-4381-ae7d-676a82db925a-kube-api-access-xwq9r" pod "5142a1b6-ef5b-4381-ae7d-676a82db925a" (UID: "5142a1b6-ef5b-4381-ae7d-676a82db925a"). InnerVolumeSpecName "kube-api-access-xwq9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:57:49.612851 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:49.607634 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5142a1b6-ef5b-4381-ae7d-676a82db925a-istio-podinfo" pod "5142a1b6-ef5b-4381-ae7d-676a82db925a" (UID: "5142a1b6-ef5b-4381-ae7d-676a82db925a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:57:49.617613 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:49.617555 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5142a1b6-ef5b-4381-ae7d-676a82db925a-istio-token" pod "5142a1b6-ef5b-4381-ae7d-676a82db925a" (UID: "5142a1b6-ef5b-4381-ae7d-676a82db925a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:57:49.622760 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:49.621872 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5142a1b6-ef5b-4381-ae7d-676a82db925a-istio-envoy" pod "5142a1b6-ef5b-4381-ae7d-676a82db925a" (UID: "5142a1b6-ef5b-4381-ae7d-676a82db925a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:57:49.699672 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:49.699635 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5142a1b6-ef5b-4381-ae7d-676a82db925a-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:49.699672 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:49.699667 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5142a1b6-ef5b-4381-ae7d-676a82db925a-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:49.699881 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:49.699682 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwq9r\" (UniqueName: \"kubernetes.io/projected/5142a1b6-ef5b-4381-ae7d-676a82db925a-kube-api-access-xwq9r\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:49.699881 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:49.699693 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5142a1b6-ef5b-4381-ae7d-676a82db925a-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:50.001361 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:50.001326 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc"] Nov 01 20:57:50.005866 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:50.005837 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-b95tc"] Nov 01 20:57:50.758356 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:50.758327 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f620899-69lws" Nov 01 20:57:50.819613 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:50.819576 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a7deca0-b10e-4b93-b0c8-9721b9db23b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a7deca0-b10e-4b93-b0c8-9721b9db23b7-maas-api-xks-ca\") pod \"1a7deca0-b10e-4b93-b0c8-9721b9db23b7\" (UID: \"1a7deca0-b10e-4b93-b0c8-9721b9db23b7\") " Nov 01 20:57:50.819820 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:50.819805 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a7deca0-b10e-4b93-b0c8-9721b9db23b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a7deca0-b10e-4b93-b0c8-9721b9db23b7-maas-api-openshift-ca\") pod \"1a7deca0-b10e-4b93-b0c8-9721b9db23b7\" (UID: \"1a7deca0-b10e-4b93-b0c8-9721b9db23b7\") " Nov 01 20:57:50.820048 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:50.819989 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a7deca0-b10e-4b93-b0c8-9721b9db23b7-kube-api-access-nxsl8\" (UniqueName: \"kubernetes.io/projected/1a7deca0-b10e-4b93-b0c8-9721b9db23b7-kube-api-access-nxsl8\") pod \"1a7deca0-b10e-4b93-b0c8-9721b9db23b7\" (UID: \"1a7deca0-b10e-4b93-b0c8-9721b9db23b7\") " Nov 01 20:57:50.820266 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:50.820243 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a7deca0-b10e-4b93-b0c8-9721b9db23b7-maas-api-openshift-ca" pod "1a7deca0-b10e-4b93-b0c8-9721b9db23b7" (UID: "1a7deca0-b10e-4b93-b0c8-9721b9db23b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:57:50.820661 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:50.820645 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a7deca0-b10e-4b93-b0c8-9721b9db23b7-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:50.822040 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:50.821966 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a7deca0-b10e-4b93-b0c8-9721b9db23b7-maas-api-xks-ca" pod "1a7deca0-b10e-4b93-b0c8-9721b9db23b7" (UID: "1a7deca0-b10e-4b93-b0c8-9721b9db23b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:57:50.822147 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:50.822052 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a7deca0-b10e-4b93-b0c8-9721b9db23b7-kube-api-access-nxsl8" pod "1a7deca0-b10e-4b93-b0c8-9721b9db23b7" (UID: "1a7deca0-b10e-4b93-b0c8-9721b9db23b7"). InnerVolumeSpecName "kube-api-access-nxsl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:57:50.922229 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:50.922143 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a7deca0-b10e-4b93-b0c8-9721b9db23b7-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:50.922229 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:50.922173 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxsl8\" (UniqueName: \"kubernetes.io/projected/1a7deca0-b10e-4b93-b0c8-9721b9db23b7-kube-api-access-nxsl8\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:57:51.375794 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:51.375760 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5142a1b6-ef5b-4381-ae7d-676a82db925a" path="/var/lib/kubelet/pods/5142a1b6-ef5b-4381-ae7d-676a82db925a/volumes" Nov 01 20:57:51.595056 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:51.594980 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f620899-69lws" Nov 01 20:57:51.595247 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:51.595034 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f620899-69lws" event={"ID":"1a7deca0-b10e-4b93-b0c8-9721b9db23b7","Type":"ContainerDied","Data":"c64eaf71847662228d782842bd6765ea6cbca61a9392a532c3eaf0c78d744236"} Nov 01 20:57:51.595247 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:51.595164 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c64eaf71847662228d782842bd6765ea6cbca61a9392a532c3eaf0c78d744236" Nov 01 20:57:53.679289 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:53.679252 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:54.253778 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:54.253746 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-de8145-56fb74df5-8nbgw" Nov 01 20:57:55.263160 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:55.263132 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-de8145-9dcc65c7f-q6cff" Nov 01 20:57:57.454131 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:57.454093 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:57.898979 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:57.898943 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt"] Nov 01 20:57:57.899188 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:57.899162 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt" podUID="7b446c1d-56a2-4b12-923e-2ca1874321ed" containerName="istio-proxy" containerID="cri-o://6d987a419cf3fe3d71bf6b2bdb40cfdae84adfabf76629186e44033cd121104c" gracePeriod=30 Nov 01 20:57:57.950090 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:57.950047 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz"] Nov 01 20:57:57.950333 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:57.950308 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz" podUID="e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed" containerName="istio-proxy" containerID="cri-o://dadc7b561815a1fdd3d45629dff96f4e18bc88a802cd7de7df5a693be1537f86" gracePeriod=30 Nov 01 20:57:57.954853 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:57.954826 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv"] Nov 01 20:57:57.955056 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:57.955024 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv" podUID="b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2" containerName="istio-proxy" containerID="cri-o://44b9d35619a9b318ba03543f0a159f06277941169a511aa5f57f7194abf9ddf4" gracePeriod=30 Nov 01 20:57:58.135035 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:58.134987 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:58.290861 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:58.282395 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr"] Nov 01 20:57:58.290861 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:58.282657 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr" podUID="35f67043-6238-4386-a37a-3c2dbed1fae9" containerName="istio-proxy" containerID="cri-o://f39fcfb768baa4a8f9d9656dcdcc776029817091ff2e33c1c2c9d2dfa9d06eec" gracePeriod=30 Nov 01 20:57:58.339098 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:58.338070 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r"] Nov 01 20:57:58.339098 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:58.338318 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r" podUID="74fb0cfd-c53c-49da-a2c8-02cdd62a79b7" containerName="istio-proxy" containerID="cri-o://2fa0e01f056ed2a498ec729af832d820d805c5377adf30d0f65b509f7a2a2614" gracePeriod=30 Nov 01 20:57:58.485037 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:58.452704 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk"] Nov 01 20:57:58.485037 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:58.453056 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk" podUID="c93e460f-14ce-40bc-93cf-51e81a750554" containerName="istio-proxy" containerID="cri-o://4c068cebff320e1ff75553f7197c0adbcd9d04c5033f91bbcec0a9062a2a960f" gracePeriod=30 Nov 01 20:57:59.305812 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:59.305774 4484 scope.go:122] "RemoveContainer" containerID="98eb9f67995f9d65e35ae193d13444057820e3d6c51777451704895e968334bb" Nov 01 20:57:59.317671 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:59.317649 4484 scope.go:122] "RemoveContainer" containerID="51a4e7edbce4d63e0c709cbd8e483cca4707110046613ebab1cac1bb5fb931c9" Nov 01 20:57:59.332286 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:59.332256 4484 scope.go:122] "RemoveContainer" containerID="cfa800d7f517ac0a0794e6e2fb7e4b0c6b5eadf581dcaa3d2f093bf066aa4e8f" Nov 01 20:57:59.343944 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:59.343924 4484 scope.go:122] "RemoveContainer" containerID="f68543a4be9c89f1cf16d5dd15cea72562f324c3063b356926c91efbdfb5ffcd" Nov 01 20:57:59.356405 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:59.356382 4484 scope.go:122] "RemoveContainer" containerID="11755af73def77184bacdf0b2e1be9d68fe4b5db0906f6aefb7486f58dd525ff" Nov 01 20:57:59.368181 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:59.368157 4484 scope.go:122] "RemoveContainer" containerID="723cd48916e6b719f4fc6ebf8c2a8b85dd05268a945a82ee0e827353d89364a3" Nov 01 20:57:59.382320 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:59.382296 4484 scope.go:122] "RemoveContainer" containerID="6d64ae249961d215d70fc1faf813ab0e7424a23286ea1d20e1f1fd2ca42da3d6" Nov 01 20:57:59.393083 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:59.393061 4484 scope.go:122] "RemoveContainer" containerID="bb3743c52e4388a3d036efae5c92ffd5c1316d3f7c3cf23926178116d3ba5708" Nov 01 20:57:59.403682 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:59.403659 4484 scope.go:122] "RemoveContainer" containerID="5f732772f0519f0c71593dc3d9fdf6db9f20929f0dea6ce334406305ef8c4d6d" Nov 01 20:57:59.414597 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:59.414575 4484 scope.go:122] "RemoveContainer" containerID="3cf46e97af4bfca2ed387dce8a73001b403e6a9a36219440497dd1d920d68748" Nov 01 20:57:59.427083 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:59.427064 4484 scope.go:122] "RemoveContainer" containerID="add0f520a069756283e725df7571c069a0d73b03c578da0b842cc67b4c39158e" Nov 01 20:57:59.437856 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:59.437832 4484 scope.go:122] "RemoveContainer" containerID="1db32fba5019b9cfc7f0c5ba1cf95832bd3d6331f9ddac7a642000d0c99958c9" Nov 01 20:57:59.449071 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:59.449052 4484 scope.go:122] "RemoveContainer" containerID="3aaf04809a886b4f96cd8303331e933e722909e72caab7599e702f8f696e7198" Nov 01 20:57:59.475801 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:59.475775 4484 scope.go:122] "RemoveContainer" containerID="7e1fe1e09d9023483aa7983f2815e0bf7d1c03612ec228ea1ce60b2dd73e877c" Nov 01 20:57:59.578575 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:59.578553 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a9b439-76df769c99-g6df9" Nov 01 20:57:59.767125 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:57:59.767093 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:00.028398 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:00.028354 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:00.211470 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:00.210587 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:00.286059 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:00.285937 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a5ab4417-jx9w5"] Nov 01 20:58:00.290580 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:00.290553 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a5ab4417-jx9w5"] Nov 01 20:58:00.588918 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:00.588834 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a9b439-76bf5ff49-qqkwc" Nov 01 20:58:00.692483 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:00.692425 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:01.375018 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:01.374960 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b4a05cb-3c59-420e-9aaf-9d57535fd45b" path="/var/lib/kubelet/pods/2b4a05cb-3c59-420e-9aaf-9d57535fd45b/volumes" Nov 01 20:58:01.882116 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:01.882075 4484 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 20:58:01.882523 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:01.882135 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk" podUID="c93e460f-14ce-40bc-93cf-51e81a750554" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 20:58:02.853051 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:02.852980 4484 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 20:58:02.853246 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:02.853071 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt" podUID="7b446c1d-56a2-4b12-923e-2ca1874321ed" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 20:58:02.866951 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:02.866911 4484 patch_prober.go:29] interesting pod/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 20:58:02.867176 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:02.866979 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r" podUID="74fb0cfd-c53c-49da-a2c8-02cdd62a79b7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 20:58:03.950098 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:03.947873 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:04.242569 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.242535 4484 generic.go:356] "Generic (PLEG): container finished" podID="74fb0cfd-c53c-49da-a2c8-02cdd62a79b7" containerID="2fa0e01f056ed2a498ec729af832d820d805c5377adf30d0f65b509f7a2a2614" exitCode=0 Nov 01 20:58:04.243089 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.243065 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r" event={"ID":"74fb0cfd-c53c-49da-a2c8-02cdd62a79b7","Type":"ContainerDied","Data":"2fa0e01f056ed2a498ec729af832d820d805c5377adf30d0f65b509f7a2a2614"} Nov 01 20:58:04.246355 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.246334 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv" event={"ID":"b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2","Type":"ContainerDied","Data":"44b9d35619a9b318ba03543f0a159f06277941169a511aa5f57f7194abf9ddf4"} Nov 01 20:58:04.246521 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.246198 4484 generic.go:356] "Generic (PLEG): container finished" podID="b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2" containerID="44b9d35619a9b318ba03543f0a159f06277941169a511aa5f57f7194abf9ddf4" exitCode=0 Nov 01 20:58:04.248739 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.248717 4484 generic.go:356] "Generic (PLEG): container finished" podID="c93e460f-14ce-40bc-93cf-51e81a750554" containerID="4c068cebff320e1ff75553f7197c0adbcd9d04c5033f91bbcec0a9062a2a960f" exitCode=0 Nov 01 20:58:04.249070 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.249053 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk" event={"ID":"c93e460f-14ce-40bc-93cf-51e81a750554","Type":"ContainerDied","Data":"4c068cebff320e1ff75553f7197c0adbcd9d04c5033f91bbcec0a9062a2a960f"} Nov 01 20:58:04.279021 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.273559 4484 generic.go:356] "Generic (PLEG): container finished" podID="e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed" containerID="dadc7b561815a1fdd3d45629dff96f4e18bc88a802cd7de7df5a693be1537f86" exitCode=0 Nov 01 20:58:04.279021 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.273670 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz" event={"ID":"e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed","Type":"ContainerDied","Data":"dadc7b561815a1fdd3d45629dff96f4e18bc88a802cd7de7df5a693be1537f86"} Nov 01 20:58:04.279272 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.279178 4484 generic.go:356] "Generic (PLEG): container finished" podID="35f67043-6238-4386-a37a-3c2dbed1fae9" containerID="f39fcfb768baa4a8f9d9656dcdcc776029817091ff2e33c1c2c9d2dfa9d06eec" exitCode=0 Nov 01 20:58:04.294036 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.279331 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr" event={"ID":"35f67043-6238-4386-a37a-3c2dbed1fae9","Type":"ContainerDied","Data":"f39fcfb768baa4a8f9d9656dcdcc776029817091ff2e33c1c2c9d2dfa9d06eec"} Nov 01 20:58:04.294036 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.279414 4484 scope.go:122] "RemoveContainer" containerID="61b1906e39baee0d36b4f4ec168741c9a02e049517234025034b92d856b82356" Nov 01 20:58:04.296747 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.296713 4484 generic.go:356] "Generic (PLEG): container finished" podID="7b446c1d-56a2-4b12-923e-2ca1874321ed" containerID="6d987a419cf3fe3d71bf6b2bdb40cfdae84adfabf76629186e44033cd121104c" exitCode=0 Nov 01 20:58:04.297102 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.297078 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt" event={"ID":"7b446c1d-56a2-4b12-923e-2ca1874321ed","Type":"ContainerDied","Data":"6d987a419cf3fe3d71bf6b2bdb40cfdae84adfabf76629186e44033cd121104c"} Nov 01 20:58:04.388195 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.384301 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt" Nov 01 20:58:04.668535 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.632226 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b446c1d-56a2-4b12-923e-2ca1874321ed-istio-token\" (UniqueName: \"kubernetes.io/projected/7b446c1d-56a2-4b12-923e-2ca1874321ed-istio-token\") pod \"7b446c1d-56a2-4b12-923e-2ca1874321ed\" (UID: \"7b446c1d-56a2-4b12-923e-2ca1874321ed\") " Nov 01 20:58:04.668535 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.632293 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b446c1d-56a2-4b12-923e-2ca1874321ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b446c1d-56a2-4b12-923e-2ca1874321ed-credential-socket\") pod \"7b446c1d-56a2-4b12-923e-2ca1874321ed\" (UID: \"7b446c1d-56a2-4b12-923e-2ca1874321ed\") " Nov 01 20:58:04.668535 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.632367 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b446c1d-56a2-4b12-923e-2ca1874321ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b446c1d-56a2-4b12-923e-2ca1874321ed-istiod-ca-cert\") pod \"7b446c1d-56a2-4b12-923e-2ca1874321ed\" (UID: \"7b446c1d-56a2-4b12-923e-2ca1874321ed\") " Nov 01 20:58:04.668535 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.632453 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b446c1d-56a2-4b12-923e-2ca1874321ed-kube-api-access-qjjg6\" (UniqueName: \"kubernetes.io/projected/7b446c1d-56a2-4b12-923e-2ca1874321ed-kube-api-access-qjjg6\") pod \"7b446c1d-56a2-4b12-923e-2ca1874321ed\" (UID: \"7b446c1d-56a2-4b12-923e-2ca1874321ed\") " Nov 01 20:58:04.668535 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.632496 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b446c1d-56a2-4b12-923e-2ca1874321ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b446c1d-56a2-4b12-923e-2ca1874321ed-workload-certs\") pod \"7b446c1d-56a2-4b12-923e-2ca1874321ed\" (UID: \"7b446c1d-56a2-4b12-923e-2ca1874321ed\") " Nov 01 20:58:04.668535 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.632535 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b446c1d-56a2-4b12-923e-2ca1874321ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b446c1d-56a2-4b12-923e-2ca1874321ed-workload-socket\") pod \"7b446c1d-56a2-4b12-923e-2ca1874321ed\" (UID: \"7b446c1d-56a2-4b12-923e-2ca1874321ed\") " Nov 01 20:58:04.668535 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.632621 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b446c1d-56a2-4b12-923e-2ca1874321ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b446c1d-56a2-4b12-923e-2ca1874321ed-istio-podinfo\") pod \"7b446c1d-56a2-4b12-923e-2ca1874321ed\" (UID: \"7b446c1d-56a2-4b12-923e-2ca1874321ed\") " Nov 01 20:58:04.668535 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.632665 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b446c1d-56a2-4b12-923e-2ca1874321ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b446c1d-56a2-4b12-923e-2ca1874321ed-istio-envoy\") pod \"7b446c1d-56a2-4b12-923e-2ca1874321ed\" (UID: \"7b446c1d-56a2-4b12-923e-2ca1874321ed\") " Nov 01 20:58:04.668535 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.632750 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b446c1d-56a2-4b12-923e-2ca1874321ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b446c1d-56a2-4b12-923e-2ca1874321ed-istio-data\") pod \"7b446c1d-56a2-4b12-923e-2ca1874321ed\" (UID: \"7b446c1d-56a2-4b12-923e-2ca1874321ed\") " Nov 01 20:58:04.668535 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.636832 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b446c1d-56a2-4b12-923e-2ca1874321ed-workload-certs" pod "7b446c1d-56a2-4b12-923e-2ca1874321ed" (UID: "7b446c1d-56a2-4b12-923e-2ca1874321ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:58:04.668535 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.638224 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b446c1d-56a2-4b12-923e-2ca1874321ed-workload-socket" pod "7b446c1d-56a2-4b12-923e-2ca1874321ed" (UID: "7b446c1d-56a2-4b12-923e-2ca1874321ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:58:04.718260 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.706587 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b446c1d-56a2-4b12-923e-2ca1874321ed-kube-api-access-qjjg6" pod "7b446c1d-56a2-4b12-923e-2ca1874321ed" (UID: "7b446c1d-56a2-4b12-923e-2ca1874321ed"). InnerVolumeSpecName "kube-api-access-qjjg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:58:04.730783 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.721469 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b446c1d-56a2-4b12-923e-2ca1874321ed-istio-envoy" pod "7b446c1d-56a2-4b12-923e-2ca1874321ed" (UID: "7b446c1d-56a2-4b12-923e-2ca1874321ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:58:04.740144 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.739433 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv" Nov 01 20:58:04.771092 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.754526 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjjg6\" (UniqueName: \"kubernetes.io/projected/7b446c1d-56a2-4b12-923e-2ca1874321ed-kube-api-access-qjjg6\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:04.774222 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.756515 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b446c1d-56a2-4b12-923e-2ca1874321ed-istiod-ca-cert" pod "7b446c1d-56a2-4b12-923e-2ca1874321ed" (UID: "7b446c1d-56a2-4b12-923e-2ca1874321ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:58:04.774222 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.774035 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b446c1d-56a2-4b12-923e-2ca1874321ed-credential-socket" pod "7b446c1d-56a2-4b12-923e-2ca1874321ed" (UID: "7b446c1d-56a2-4b12-923e-2ca1874321ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:58:04.774222 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.774085 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b446c1d-56a2-4b12-923e-2ca1874321ed-istio-data" pod "7b446c1d-56a2-4b12-923e-2ca1874321ed" (UID: "7b446c1d-56a2-4b12-923e-2ca1874321ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:58:04.780603 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.780527 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b446c1d-56a2-4b12-923e-2ca1874321ed-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:04.780603 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.780561 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b446c1d-56a2-4b12-923e-2ca1874321ed-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:04.780603 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.780577 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b446c1d-56a2-4b12-923e-2ca1874321ed-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:04.781714 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.781627 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz" Nov 01 20:58:04.831435 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.826661 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b446c1d-56a2-4b12-923e-2ca1874321ed-istio-token" pod "7b446c1d-56a2-4b12-923e-2ca1874321ed" (UID: "7b446c1d-56a2-4b12-923e-2ca1874321ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:58:04.833007 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.832922 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b446c1d-56a2-4b12-923e-2ca1874321ed-istio-podinfo" pod "7b446c1d-56a2-4b12-923e-2ca1874321ed" (UID: "7b446c1d-56a2-4b12-923e-2ca1874321ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:58:04.887151 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.884278 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r" Nov 01 20:58:04.887151 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.884853 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-istio-token\" (UniqueName: \"kubernetes.io/projected/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-istio-token\") pod \"b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2\" (UID: \"b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2\") " Nov 01 20:58:04.887151 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.884919 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-workload-certs\") pod \"b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2\" (UID: \"b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2\") " Nov 01 20:58:04.887151 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.884956 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-istio-envoy\") pod \"b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2\" (UID: \"b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2\") " Nov 01 20:58:04.887151 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.885028 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-workload-socket\") pod \"b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2\" (UID: \"b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2\") " Nov 01 20:58:04.887151 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.885081 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-istio-podinfo\") pod \"e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed\" (UID: \"e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed\") " Nov 01 20:58:04.887151 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.885111 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-workload-socket\") pod \"e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed\" (UID: \"e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed\") " Nov 01 20:58:04.887151 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.885144 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-workload-certs\") pod \"e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed\" (UID: \"e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed\") " Nov 01 20:58:04.887151 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.885237 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-istio-data\") pod \"e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed\" (UID: \"e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed\") " Nov 01 20:58:04.887151 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.885309 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-istio-envoy\") pod \"e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed\" (UID: \"e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed\") " Nov 01 20:58:04.887151 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.885342 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-istio-podinfo\") pod \"b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2\" (UID: \"b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2\") " Nov 01 20:58:04.887151 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.885392 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-istiod-ca-cert\") pod \"e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed\" (UID: \"e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed\") " Nov 01 20:58:04.887151 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.885436 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-kube-api-access-gx5np\" (UniqueName: \"kubernetes.io/projected/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-kube-api-access-gx5np\") pod \"e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed\" (UID: \"e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed\") " Nov 01 20:58:04.887151 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.885538 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-kube-api-access-zgn4w\" (UniqueName: \"kubernetes.io/projected/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-kube-api-access-zgn4w\") pod \"b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2\" (UID: \"b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2\") " Nov 01 20:58:04.887151 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.885665 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-istio-data\") pod \"b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2\" (UID: \"b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2\") " Nov 01 20:58:04.887151 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.885742 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-credential-socket\") pod \"e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed\" (UID: \"e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed\") " Nov 01 20:58:04.887151 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.885782 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-credential-socket\") pod \"b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2\" (UID: \"b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2\") " Nov 01 20:58:04.887151 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.885812 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-istio-token\" (UniqueName: \"kubernetes.io/projected/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-istio-token\") pod \"e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed\" (UID: \"e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed\") " Nov 01 20:58:04.887151 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.885848 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-istiod-ca-cert\") pod \"b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2\" (UID: \"b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2\") " Nov 01 20:58:04.887151 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.886795 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b446c1d-56a2-4b12-923e-2ca1874321ed-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:04.887151 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.886816 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b446c1d-56a2-4b12-923e-2ca1874321ed-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:04.887151 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.886831 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b446c1d-56a2-4b12-923e-2ca1874321ed-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:04.887151 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.886845 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b446c1d-56a2-4b12-923e-2ca1874321ed-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:04.887151 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.886860 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b446c1d-56a2-4b12-923e-2ca1874321ed-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:04.891670 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.891188 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-workload-socket" pod "b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2" (UID: "b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:58:04.891670 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.891215 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-credential-socket" pod "e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed" (UID: "e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:58:04.891670 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.891397 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-workload-certs" pod "b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2" (UID: "b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:58:04.891670 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.891457 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-istio-data" pod "b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2" (UID: "b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:58:04.892713 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.892450 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-istiod-ca-cert" pod "b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2" (UID: "b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:58:04.893174 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.892891 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-credential-socket" pod "b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2" (UID: "b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:58:04.893966 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.893730 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-istio-data" pod "e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed" (UID: "e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:58:04.894303 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.894284 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-workload-certs" pod "e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed" (UID: "e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:58:04.894460 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.894435 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-workload-socket" pod "e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed" (UID: "e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:58:04.895088 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.894817 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-istiod-ca-cert" pod "e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed" (UID: "e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:58:04.900529 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.900506 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-kube-api-access-gx5np" pod "e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed" (UID: "e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed"). InnerVolumeSpecName "kube-api-access-gx5np". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:58:04.900643 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.900571 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-istio-envoy" pod "b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2" (UID: "b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:58:04.902123 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.902094 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-istio-token" pod "b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2" (UID: "b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:58:04.902373 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.902351 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-istio-podinfo" pod "e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed" (UID: "e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:58:04.902508 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.902486 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-istio-token" pod "e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed" (UID: "e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:58:04.902598 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.902580 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-kube-api-access-zgn4w" pod "b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2" (UID: "b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2"). InnerVolumeSpecName "kube-api-access-zgn4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:58:04.902981 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.902810 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-istio-envoy" pod "e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed" (UID: "e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:58:04.904887 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.904432 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-istio-podinfo" pod "b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2" (UID: "b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:58:04.921754 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.918301 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk" Nov 01 20:58:04.923467 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:04.923446 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr" Nov 01 20:58:05.004403 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.004364 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c93e460f-14ce-40bc-93cf-51e81a750554-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c93e460f-14ce-40bc-93cf-51e81a750554-credential-socket\") pod \"c93e460f-14ce-40bc-93cf-51e81a750554\" (UID: \"c93e460f-14ce-40bc-93cf-51e81a750554\") " Nov 01 20:58:05.004877 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.004418 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-istio-envoy\") pod \"74fb0cfd-c53c-49da-a2c8-02cdd62a79b7\" (UID: \"74fb0cfd-c53c-49da-a2c8-02cdd62a79b7\") " Nov 01 20:58:05.004877 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.004452 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c93e460f-14ce-40bc-93cf-51e81a750554-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c93e460f-14ce-40bc-93cf-51e81a750554-workload-certs\") pod \"c93e460f-14ce-40bc-93cf-51e81a750554\" (UID: \"c93e460f-14ce-40bc-93cf-51e81a750554\") " Nov 01 20:58:05.004877 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.004481 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35f67043-6238-4386-a37a-3c2dbed1fae9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35f67043-6238-4386-a37a-3c2dbed1fae9-istio-envoy\") pod \"35f67043-6238-4386-a37a-3c2dbed1fae9\" (UID: \"35f67043-6238-4386-a37a-3c2dbed1fae9\") " Nov 01 20:58:05.004877 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.004510 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-istio-podinfo\") pod \"74fb0cfd-c53c-49da-a2c8-02cdd62a79b7\" (UID: \"74fb0cfd-c53c-49da-a2c8-02cdd62a79b7\") " Nov 01 20:58:05.004877 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.004555 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35f67043-6238-4386-a37a-3c2dbed1fae9-istio-token\" (UniqueName: \"kubernetes.io/projected/35f67043-6238-4386-a37a-3c2dbed1fae9-istio-token\") pod \"35f67043-6238-4386-a37a-3c2dbed1fae9\" (UID: \"35f67043-6238-4386-a37a-3c2dbed1fae9\") " Nov 01 20:58:05.004877 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.004584 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c93e460f-14ce-40bc-93cf-51e81a750554-istio-token\" (UniqueName: \"kubernetes.io/projected/c93e460f-14ce-40bc-93cf-51e81a750554-istio-token\") pod \"c93e460f-14ce-40bc-93cf-51e81a750554\" (UID: \"c93e460f-14ce-40bc-93cf-51e81a750554\") " Nov 01 20:58:05.004877 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.004621 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c93e460f-14ce-40bc-93cf-51e81a750554-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c93e460f-14ce-40bc-93cf-51e81a750554-istiod-ca-cert\") pod \"c93e460f-14ce-40bc-93cf-51e81a750554\" (UID: \"c93e460f-14ce-40bc-93cf-51e81a750554\") " Nov 01 20:58:05.004877 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.004656 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-workload-certs\") pod \"74fb0cfd-c53c-49da-a2c8-02cdd62a79b7\" (UID: \"74fb0cfd-c53c-49da-a2c8-02cdd62a79b7\") " Nov 01 20:58:05.004877 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.004703 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-kube-api-access-642kd\" (UniqueName: \"kubernetes.io/projected/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-kube-api-access-642kd\") pod \"74fb0cfd-c53c-49da-a2c8-02cdd62a79b7\" (UID: \"74fb0cfd-c53c-49da-a2c8-02cdd62a79b7\") " Nov 01 20:58:05.004877 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.004836 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c93e460f-14ce-40bc-93cf-51e81a750554-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c93e460f-14ce-40bc-93cf-51e81a750554-istio-envoy\") pod \"c93e460f-14ce-40bc-93cf-51e81a750554\" (UID: \"c93e460f-14ce-40bc-93cf-51e81a750554\") " Nov 01 20:58:05.004877 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.004872 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c93e460f-14ce-40bc-93cf-51e81a750554-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c93e460f-14ce-40bc-93cf-51e81a750554-istio-data\") pod \"c93e460f-14ce-40bc-93cf-51e81a750554\" (UID: \"c93e460f-14ce-40bc-93cf-51e81a750554\") " Nov 01 20:58:05.005608 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.004904 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-istiod-ca-cert\") pod \"74fb0cfd-c53c-49da-a2c8-02cdd62a79b7\" (UID: \"74fb0cfd-c53c-49da-a2c8-02cdd62a79b7\") " Nov 01 20:58:05.005608 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.004968 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35f67043-6238-4386-a37a-3c2dbed1fae9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35f67043-6238-4386-a37a-3c2dbed1fae9-istio-data\") pod \"35f67043-6238-4386-a37a-3c2dbed1fae9\" (UID: \"35f67043-6238-4386-a37a-3c2dbed1fae9\") " Nov 01 20:58:05.010544 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.008347 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35f67043-6238-4386-a37a-3c2dbed1fae9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f67043-6238-4386-a37a-3c2dbed1fae9-credential-socket\") pod \"35f67043-6238-4386-a37a-3c2dbed1fae9\" (UID: \"35f67043-6238-4386-a37a-3c2dbed1fae9\") " Nov 01 20:58:05.010544 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.008467 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35f67043-6238-4386-a37a-3c2dbed1fae9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35f67043-6238-4386-a37a-3c2dbed1fae9-istio-podinfo\") pod \"35f67043-6238-4386-a37a-3c2dbed1fae9\" (UID: \"35f67043-6238-4386-a37a-3c2dbed1fae9\") " Nov 01 20:58:05.010544 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.008521 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c93e460f-14ce-40bc-93cf-51e81a750554-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c93e460f-14ce-40bc-93cf-51e81a750554-workload-socket\") pod \"c93e460f-14ce-40bc-93cf-51e81a750554\" (UID: \"c93e460f-14ce-40bc-93cf-51e81a750554\") " Nov 01 20:58:05.010544 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.008555 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35f67043-6238-4386-a37a-3c2dbed1fae9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35f67043-6238-4386-a37a-3c2dbed1fae9-workload-certs\") pod \"35f67043-6238-4386-a37a-3c2dbed1fae9\" (UID: \"35f67043-6238-4386-a37a-3c2dbed1fae9\") " Nov 01 20:58:05.010544 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.008631 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-workload-socket\") pod \"74fb0cfd-c53c-49da-a2c8-02cdd62a79b7\" (UID: \"74fb0cfd-c53c-49da-a2c8-02cdd62a79b7\") " Nov 01 20:58:05.010544 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.008710 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c93e460f-14ce-40bc-93cf-51e81a750554-kube-api-access-p29xq\" (UniqueName: \"kubernetes.io/projected/c93e460f-14ce-40bc-93cf-51e81a750554-kube-api-access-p29xq\") pod \"c93e460f-14ce-40bc-93cf-51e81a750554\" (UID: \"c93e460f-14ce-40bc-93cf-51e81a750554\") " Nov 01 20:58:05.010544 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.008751 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-credential-socket\") pod \"74fb0cfd-c53c-49da-a2c8-02cdd62a79b7\" (UID: \"74fb0cfd-c53c-49da-a2c8-02cdd62a79b7\") " Nov 01 20:58:05.010544 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.008788 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35f67043-6238-4386-a37a-3c2dbed1fae9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35f67043-6238-4386-a37a-3c2dbed1fae9-istiod-ca-cert\") pod \"35f67043-6238-4386-a37a-3c2dbed1fae9\" (UID: \"35f67043-6238-4386-a37a-3c2dbed1fae9\") " Nov 01 20:58:05.010544 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.008858 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-istio-token\" (UniqueName: \"kubernetes.io/projected/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-istio-token\") pod \"74fb0cfd-c53c-49da-a2c8-02cdd62a79b7\" (UID: \"74fb0cfd-c53c-49da-a2c8-02cdd62a79b7\") " Nov 01 20:58:05.010544 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.008979 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c93e460f-14ce-40bc-93cf-51e81a750554-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c93e460f-14ce-40bc-93cf-51e81a750554-istio-podinfo\") pod \"c93e460f-14ce-40bc-93cf-51e81a750554\" (UID: \"c93e460f-14ce-40bc-93cf-51e81a750554\") " Nov 01 20:58:05.010544 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.009046 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-istio-data\") pod \"74fb0cfd-c53c-49da-a2c8-02cdd62a79b7\" (UID: \"74fb0cfd-c53c-49da-a2c8-02cdd62a79b7\") " Nov 01 20:58:05.010544 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.009090 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35f67043-6238-4386-a37a-3c2dbed1fae9-kube-api-access-ktmxx\" (UniqueName: \"kubernetes.io/projected/35f67043-6238-4386-a37a-3c2dbed1fae9-kube-api-access-ktmxx\") pod \"35f67043-6238-4386-a37a-3c2dbed1fae9\" (UID: \"35f67043-6238-4386-a37a-3c2dbed1fae9\") " Nov 01 20:58:05.010544 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.009123 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35f67043-6238-4386-a37a-3c2dbed1fae9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f67043-6238-4386-a37a-3c2dbed1fae9-workload-socket\") pod \"35f67043-6238-4386-a37a-3c2dbed1fae9\" (UID: \"35f67043-6238-4386-a37a-3c2dbed1fae9\") " Nov 01 20:58:05.028734 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.011735 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-workload-certs" pod "74fb0cfd-c53c-49da-a2c8-02cdd62a79b7" (UID: "74fb0cfd-c53c-49da-a2c8-02cdd62a79b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:58:05.028734 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.013787 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-istio-podinfo" pod "74fb0cfd-c53c-49da-a2c8-02cdd62a79b7" (UID: "74fb0cfd-c53c-49da-a2c8-02cdd62a79b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:58:05.028734 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.014736 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c93e460f-14ce-40bc-93cf-51e81a750554-credential-socket" pod "c93e460f-14ce-40bc-93cf-51e81a750554" (UID: "c93e460f-14ce-40bc-93cf-51e81a750554"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:58:05.028734 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.016668 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c93e460f-14ce-40bc-93cf-51e81a750554-istio-envoy" pod "c93e460f-14ce-40bc-93cf-51e81a750554" (UID: "c93e460f-14ce-40bc-93cf-51e81a750554"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:58:05.028734 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.018612 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-kube-api-access-642kd" pod "74fb0cfd-c53c-49da-a2c8-02cdd62a79b7" (UID: "74fb0cfd-c53c-49da-a2c8-02cdd62a79b7"). InnerVolumeSpecName "kube-api-access-642kd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:58:05.028734 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.018779 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f67043-6238-4386-a37a-3c2dbed1fae9-credential-socket" pod "35f67043-6238-4386-a37a-3c2dbed1fae9" (UID: "35f67043-6238-4386-a37a-3c2dbed1fae9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:58:05.028734 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.018893 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c93e460f-14ce-40bc-93cf-51e81a750554-istio-data" pod "c93e460f-14ce-40bc-93cf-51e81a750554" (UID: "c93e460f-14ce-40bc-93cf-51e81a750554"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:58:05.028734 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.019113 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-istiod-ca-cert" pod "74fb0cfd-c53c-49da-a2c8-02cdd62a79b7" (UID: "74fb0cfd-c53c-49da-a2c8-02cdd62a79b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:58:05.028734 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.019289 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f67043-6238-4386-a37a-3c2dbed1fae9-istio-data" pod "35f67043-6238-4386-a37a-3c2dbed1fae9" (UID: "35f67043-6238-4386-a37a-3c2dbed1fae9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:58:05.028734 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.021219 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-istio-token" pod "74fb0cfd-c53c-49da-a2c8-02cdd62a79b7" (UID: "74fb0cfd-c53c-49da-a2c8-02cdd62a79b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:58:05.028734 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.022845 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c93e460f-14ce-40bc-93cf-51e81a750554-kube-api-access-p29xq" pod "c93e460f-14ce-40bc-93cf-51e81a750554" (UID: "c93e460f-14ce-40bc-93cf-51e81a750554"). InnerVolumeSpecName "kube-api-access-p29xq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:58:05.028734 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.023028 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-credential-socket" pod "74fb0cfd-c53c-49da-a2c8-02cdd62a79b7" (UID: "74fb0cfd-c53c-49da-a2c8-02cdd62a79b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:58:05.028734 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.023333 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35f67043-6238-4386-a37a-3c2dbed1fae9-istiod-ca-cert" pod "35f67043-6238-4386-a37a-3c2dbed1fae9" (UID: "35f67043-6238-4386-a37a-3c2dbed1fae9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:58:05.028734 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.024628 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-workload-socket" pod "74fb0cfd-c53c-49da-a2c8-02cdd62a79b7" (UID: "74fb0cfd-c53c-49da-a2c8-02cdd62a79b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:58:05.028734 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.024781 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f67043-6238-4386-a37a-3c2dbed1fae9-workload-certs" pod "35f67043-6238-4386-a37a-3c2dbed1fae9" (UID: "35f67043-6238-4386-a37a-3c2dbed1fae9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:58:05.028734 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.025101 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-istio-data" pod "74fb0cfd-c53c-49da-a2c8-02cdd62a79b7" (UID: "74fb0cfd-c53c-49da-a2c8-02cdd62a79b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:58:05.028734 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.025421 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c93e460f-14ce-40bc-93cf-51e81a750554-workload-certs" pod "c93e460f-14ce-40bc-93cf-51e81a750554" (UID: "c93e460f-14ce-40bc-93cf-51e81a750554"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:58:05.028734 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.025441 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c93e460f-14ce-40bc-93cf-51e81a750554-workload-socket" pod "c93e460f-14ce-40bc-93cf-51e81a750554" (UID: "c93e460f-14ce-40bc-93cf-51e81a750554"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:58:05.028734 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.026858 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-istio-envoy" pod "74fb0cfd-c53c-49da-a2c8-02cdd62a79b7" (UID: "74fb0cfd-c53c-49da-a2c8-02cdd62a79b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:58:05.028734 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.027287 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f67043-6238-4386-a37a-3c2dbed1fae9-workload-socket" pod "35f67043-6238-4386-a37a-3c2dbed1fae9" (UID: "35f67043-6238-4386-a37a-3c2dbed1fae9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:58:05.029806 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.028896 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f67043-6238-4386-a37a-3c2dbed1fae9-istio-envoy" pod "35f67043-6238-4386-a37a-3c2dbed1fae9" (UID: "35f67043-6238-4386-a37a-3c2dbed1fae9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:58:05.029806 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.029375 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c93e460f-14ce-40bc-93cf-51e81a750554-istio-podinfo" pod "c93e460f-14ce-40bc-93cf-51e81a750554" (UID: "c93e460f-14ce-40bc-93cf-51e81a750554"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:58:05.029806 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.029389 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35f67043-6238-4386-a37a-3c2dbed1fae9-istio-token" pod "35f67043-6238-4386-a37a-3c2dbed1fae9" (UID: "35f67043-6238-4386-a37a-3c2dbed1fae9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:58:05.029806 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.029494 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c93e460f-14ce-40bc-93cf-51e81a750554-istio-token" pod "c93e460f-14ce-40bc-93cf-51e81a750554" (UID: "c93e460f-14ce-40bc-93cf-51e81a750554"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:58:05.029806 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.029577 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:05.029806 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.029656 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35f67043-6238-4386-a37a-3c2dbed1fae9-istio-podinfo" pod "35f67043-6238-4386-a37a-3c2dbed1fae9" (UID: "35f67043-6238-4386-a37a-3c2dbed1fae9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:58:05.029806 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.029698 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:05.029806 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.029714 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:05.029806 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.029731 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx5np\" (UniqueName: \"kubernetes.io/projected/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-kube-api-access-gx5np\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:05.029806 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.029751 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c93e460f-14ce-40bc-93cf-51e81a750554-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:05.029806 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.029779 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c93e460f-14ce-40bc-93cf-51e81a750554-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:05.029806 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.029793 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:05.029806 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.029806 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35f67043-6238-4386-a37a-3c2dbed1fae9-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:05.031425 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.029820 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f67043-6238-4386-a37a-3c2dbed1fae9-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:05.031425 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.029847 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgn4w\" (UniqueName: \"kubernetes.io/projected/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-kube-api-access-zgn4w\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:05.031425 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.029861 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c93e460f-14ce-40bc-93cf-51e81a750554-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:05.031425 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.029874 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35f67043-6238-4386-a37a-3c2dbed1fae9-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:05.031425 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.029889 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:05.031425 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.029903 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:05.031425 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.029929 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:05.031425 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.029945 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p29xq\" (UniqueName: \"kubernetes.io/projected/c93e460f-14ce-40bc-93cf-51e81a750554-kube-api-access-p29xq\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:05.031425 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.029959 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:05.031425 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.029973 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:05.031425 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.030011 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:05.031425 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.030031 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:05.031425 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.030043 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35f67043-6238-4386-a37a-3c2dbed1fae9-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:05.031425 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.030056 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:05.031425 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.030070 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:05.031425 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.030098 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:05.031425 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.030112 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:05.031425 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.030125 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:05.031425 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.030139 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:05.031425 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.030169 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f67043-6238-4386-a37a-3c2dbed1fae9-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:05.031425 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.030184 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:05.031425 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.030197 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:05.031425 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.030212 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:05.031425 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.030226 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c93e460f-14ce-40bc-93cf-51e81a750554-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:05.031425 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.030254 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:05.031425 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.030268 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c93e460f-14ce-40bc-93cf-51e81a750554-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:05.031425 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.030282 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35f67043-6238-4386-a37a-3c2dbed1fae9-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:05.031425 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.030295 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:05.031425 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.030320 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:05.031425 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.030334 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:05.031425 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.030349 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-642kd\" (UniqueName: \"kubernetes.io/projected/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7-kube-api-access-642kd\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:05.031425 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.030926 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35f67043-6238-4386-a37a-3c2dbed1fae9-kube-api-access-ktmxx" pod "35f67043-6238-4386-a37a-3c2dbed1fae9" (UID: "35f67043-6238-4386-a37a-3c2dbed1fae9"). InnerVolumeSpecName "kube-api-access-ktmxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:58:05.031425 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.031375 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c93e460f-14ce-40bc-93cf-51e81a750554-istiod-ca-cert" pod "c93e460f-14ce-40bc-93cf-51e81a750554" (UID: "c93e460f-14ce-40bc-93cf-51e81a750554"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:58:05.132067 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.132030 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c93e460f-14ce-40bc-93cf-51e81a750554-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:05.132067 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.132070 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktmxx\" (UniqueName: \"kubernetes.io/projected/35f67043-6238-4386-a37a-3c2dbed1fae9-kube-api-access-ktmxx\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:05.132332 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.132087 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35f67043-6238-4386-a37a-3c2dbed1fae9-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:05.132332 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.132095 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c93e460f-14ce-40bc-93cf-51e81a750554-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:05.132332 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.132107 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c93e460f-14ce-40bc-93cf-51e81a750554-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:05.132332 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.132123 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35f67043-6238-4386-a37a-3c2dbed1fae9-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:05.309472 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.309346 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr" Nov 01 20:58:05.309472 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.309368 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr" event={"ID":"35f67043-6238-4386-a37a-3c2dbed1fae9","Type":"ContainerDied","Data":"2cb0d83910ce159451bf940ebc68d0f121925c28c04b269f78788a7a2d96f276"} Nov 01 20:58:05.309472 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.309437 4484 scope.go:122] "RemoveContainer" containerID="f39fcfb768baa4a8f9d9656dcdcc776029817091ff2e33c1c2c9d2dfa9d06eec" Nov 01 20:58:05.312829 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.312535 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt" event={"ID":"7b446c1d-56a2-4b12-923e-2ca1874321ed","Type":"ContainerDied","Data":"cf17d18f551abff3aec52c8dcfe8ec476e3867ab3addb8f4aae0acc3878e2f9c"} Nov 01 20:58:05.312829 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.312610 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt" Nov 01 20:58:05.315487 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.315462 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r" event={"ID":"74fb0cfd-c53c-49da-a2c8-02cdd62a79b7","Type":"ContainerDied","Data":"22776dc59cdbcda85af1703be5bd2b3386b7345ec3ecc584ad6e4a16f16a24bc"} Nov 01 20:58:05.319671 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.319645 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r" Nov 01 20:58:05.327675 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.323356 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv" event={"ID":"b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2","Type":"ContainerDied","Data":"6081548bc3c04db34cc47a2118604924edd900bd104a412465b8c6d44b1832d6"} Nov 01 20:58:05.327675 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.323454 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv" Nov 01 20:58:05.334214 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.334191 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk" Nov 01 20:58:05.334381 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.334352 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk" event={"ID":"c93e460f-14ce-40bc-93cf-51e81a750554","Type":"ContainerDied","Data":"99d4940fab79c223381690b66aa27dcae74aced2fd8f8932a42e4104b0c5e6f4"} Nov 01 20:58:05.338275 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.336618 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz" event={"ID":"e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed","Type":"ContainerDied","Data":"82293654e03aff0b5a607849770eb46daf3ed64eaddef271fa6351272cfea9e9"} Nov 01 20:58:05.338275 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.336765 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz" Nov 01 20:58:05.373373 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.373352 4484 scope.go:122] "RemoveContainer" containerID="6d987a419cf3fe3d71bf6b2bdb40cfdae84adfabf76629186e44033cd121104c" Nov 01 20:58:05.390573 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.390489 4484 scope.go:122] "RemoveContainer" containerID="2fa0e01f056ed2a498ec729af832d820d805c5377adf30d0f65b509f7a2a2614" Nov 01 20:58:05.403910 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.403817 4484 scope.go:122] "RemoveContainer" containerID="44b9d35619a9b318ba03543f0a159f06277941169a511aa5f57f7194abf9ddf4" Nov 01 20:58:05.420799 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.420733 4484 scope.go:122] "RemoveContainer" containerID="4c068cebff320e1ff75553f7197c0adbcd9d04c5033f91bbcec0a9062a2a960f" Nov 01 20:58:05.439263 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.439236 4484 scope.go:122] "RemoveContainer" containerID="dadc7b561815a1fdd3d45629dff96f4e18bc88a802cd7de7df5a693be1537f86" Nov 01 20:58:05.509224 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.509192 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt"] Nov 01 20:58:05.515704 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.515668 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-zttrt"] Nov 01 20:58:05.526315 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.526244 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:05.663069 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.654514 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv"] Nov 01 20:58:05.674400 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.674362 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-lsbvv"] Nov 01 20:58:05.755313 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.755275 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk"] Nov 01 20:58:05.759218 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.759193 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-6wrqk"] Nov 01 20:58:05.872819 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.872788 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz"] Nov 01 20:58:05.876888 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.876860 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-tt6hz"] Nov 01 20:58:05.988286 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.988259 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr"] Nov 01 20:58:05.992260 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:05.992228 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-gkhjr"] Nov 01 20:58:06.099985 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:06.099955 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r"] Nov 01 20:58:06.103901 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:06.103874 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-cc24r"] Nov 01 20:58:06.732568 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:06.732530 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:06.932809 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:06.932774 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:07.019329 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:07.019298 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2"] Nov 01 20:58:07.019539 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:07.019519 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2" podUID="71cb71c6-14d7-4e6d-a269-22b3b66fe170" containerName="istio-proxy" containerID="cri-o://cd11d06566365a70138161b43321e9b9c4c45deb5dc7f2cf42208dc04e355338" gracePeriod=30 Nov 01 20:58:07.037222 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:07.037171 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h"] Nov 01 20:58:07.037453 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:07.037424 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h" podUID="3b91fbd7-5fe8-42de-b6be-fcbf67244747" containerName="istio-proxy" containerID="cri-o://e83d80fb7c64778872459faedd92c0daf10744f5a1c69d26dbd0648d54b98295" gracePeriod=30 Nov 01 20:58:07.137190 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:07.137115 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:07.376882 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:07.376803 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35f67043-6238-4386-a37a-3c2dbed1fae9" path="/var/lib/kubelet/pods/35f67043-6238-4386-a37a-3c2dbed1fae9/volumes" Nov 01 20:58:07.377592 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:07.377570 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74fb0cfd-c53c-49da-a2c8-02cdd62a79b7" path="/var/lib/kubelet/pods/74fb0cfd-c53c-49da-a2c8-02cdd62a79b7/volumes" Nov 01 20:58:07.378129 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:07.378114 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b446c1d-56a2-4b12-923e-2ca1874321ed" path="/var/lib/kubelet/pods/7b446c1d-56a2-4b12-923e-2ca1874321ed/volumes" Nov 01 20:58:07.378919 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:07.378903 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2" path="/var/lib/kubelet/pods/b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2/volumes" Nov 01 20:58:07.379399 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:07.379384 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c93e460f-14ce-40bc-93cf-51e81a750554" path="/var/lib/kubelet/pods/c93e460f-14ce-40bc-93cf-51e81a750554/volumes" Nov 01 20:58:07.379848 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:07.379832 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed" path="/var/lib/kubelet/pods/e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed/volumes" Nov 01 20:58:07.943150 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:07.943117 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:08.397807 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:08.397775 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl"] Nov 01 20:58:08.398555 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:08.398538 4484 cpu_manager.go:414] "containerMap: removing container" podUID="1a7deca0-b10e-4b93-b0c8-9721b9db23b7" containerName="revoke-keys" Nov 01 20:58:08.398613 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:08.398556 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a7deca0-b10e-4b93-b0c8-9721b9db23b7" containerName="revoke-keys" Nov 01 20:58:08.398613 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:08.398569 4484 cpu_manager.go:414] "containerMap: removing container" podUID="35f67043-6238-4386-a37a-3c2dbed1fae9" containerName="istio-proxy" Nov 01 20:58:08.398613 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:08.398574 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35f67043-6238-4386-a37a-3c2dbed1fae9" containerName="istio-proxy" Nov 01 20:58:08.398613 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:08.398606 4484 cpu_manager.go:414] "containerMap: removing container" podUID="74fb0cfd-c53c-49da-a2c8-02cdd62a79b7" containerName="istio-proxy" Nov 01 20:58:08.398740 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:08.398615 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74fb0cfd-c53c-49da-a2c8-02cdd62a79b7" containerName="istio-proxy" Nov 01 20:58:08.398740 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:08.398631 4484 cpu_manager.go:414] "containerMap: removing container" podUID="c93e460f-14ce-40bc-93cf-51e81a750554" containerName="istio-proxy" Nov 01 20:58:08.398740 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:08.398637 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c93e460f-14ce-40bc-93cf-51e81a750554" containerName="istio-proxy" Nov 01 20:58:08.398740 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:08.398645 4484 cpu_manager.go:414] "containerMap: removing container" podUID="e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed" containerName="istio-proxy" Nov 01 20:58:08.398740 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:08.398649 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed" containerName="istio-proxy" Nov 01 20:58:08.398740 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:08.398658 4484 cpu_manager.go:414] "containerMap: removing container" podUID="7b446c1d-56a2-4b12-923e-2ca1874321ed" containerName="istio-proxy" Nov 01 20:58:08.398740 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:08.398663 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b446c1d-56a2-4b12-923e-2ca1874321ed" containerName="istio-proxy" Nov 01 20:58:08.398740 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:08.398671 4484 cpu_manager.go:414] "containerMap: removing container" podUID="5142a1b6-ef5b-4381-ae7d-676a82db925a" containerName="istio-proxy" Nov 01 20:58:08.398740 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:08.398676 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5142a1b6-ef5b-4381-ae7d-676a82db925a" containerName="istio-proxy" Nov 01 20:58:08.398740 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:08.398687 4484 cpu_manager.go:414] "containerMap: removing container" podUID="b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2" containerName="istio-proxy" Nov 01 20:58:08.398740 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:08.398692 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2" containerName="istio-proxy" Nov 01 20:58:08.399655 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:08.399628 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="35f67043-6238-4386-a37a-3c2dbed1fae9" containerName="istio-proxy" Nov 01 20:58:08.399761 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:08.399672 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="74fb0cfd-c53c-49da-a2c8-02cdd62a79b7" containerName="istio-proxy" Nov 01 20:58:08.399761 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:08.399682 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="c93e460f-14ce-40bc-93cf-51e81a750554" containerName="istio-proxy" Nov 01 20:58:08.399761 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:08.399694 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="e10a5bf9-c1fd-49ac-a1e8-27301f3f8eed" containerName="istio-proxy" Nov 01 20:58:08.399761 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:08.399723 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b446c1d-56a2-4b12-923e-2ca1874321ed" containerName="istio-proxy" Nov 01 20:58:08.399761 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:08.399742 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="5142a1b6-ef5b-4381-ae7d-676a82db925a" containerName="istio-proxy" Nov 01 20:58:08.399761 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:08.399761 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="b48c9d14-e8b3-40d0-b3bd-ad35da7c1ac2" containerName="istio-proxy" Nov 01 20:58:08.400085 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:08.399776 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a7deca0-b10e-4b93-b0c8-9721b9db23b7" containerName="revoke-keys" Nov 01 20:58:08.400085 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:08.399803 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="35f67043-6238-4386-a37a-3c2dbed1fae9" containerName="istio-proxy" Nov 01 20:58:08.407622 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:08.407595 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl" Nov 01 20:58:08.555055 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:08.549472 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/836acdb1-4e39-472f-ba72-a07d61f977bf-credential-socket\") pod \"e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl\" (UID: \"836acdb1-4e39-472f-ba72-a07d61f977bf\") " pod="openshift-ingress/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl" Nov 01 20:58:08.555055 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:08.549640 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnzn6\" (UniqueName: \"kubernetes.io/projected/836acdb1-4e39-472f-ba72-a07d61f977bf-kube-api-access-bnzn6\") pod \"e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl\" (UID: \"836acdb1-4e39-472f-ba72-a07d61f977bf\") " pod="openshift-ingress/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl" Nov 01 20:58:08.555055 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:08.549710 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/836acdb1-4e39-472f-ba72-a07d61f977bf-istio-podinfo\") pod \"e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl\" (UID: \"836acdb1-4e39-472f-ba72-a07d61f977bf\") " pod="openshift-ingress/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl" Nov 01 20:58:08.555055 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:08.549738 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/836acdb1-4e39-472f-ba72-a07d61f977bf-workload-certs\") pod \"e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl\" (UID: \"836acdb1-4e39-472f-ba72-a07d61f977bf\") " pod="openshift-ingress/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl" Nov 01 20:58:08.555055 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:08.550017 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/836acdb1-4e39-472f-ba72-a07d61f977bf-istiod-ca-cert\") pod \"e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl\" (UID: \"836acdb1-4e39-472f-ba72-a07d61f977bf\") " pod="openshift-ingress/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl" Nov 01 20:58:08.608334 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:08.556239 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-b27e66-openshift-default-dockercfg-q4wtw\"" Nov 01 20:58:08.618012 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:08.571308 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/836acdb1-4e39-472f-ba72-a07d61f977bf-istio-data\") pod \"e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl\" (UID: \"836acdb1-4e39-472f-ba72-a07d61f977bf\") " pod="openshift-ingress/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl" Nov 01 20:58:08.622400 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:08.622365 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/836acdb1-4e39-472f-ba72-a07d61f977bf-workload-socket\") pod \"e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl\" (UID: \"836acdb1-4e39-472f-ba72-a07d61f977bf\") " pod="openshift-ingress/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl" Nov 01 20:58:08.624030 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:08.574321 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:08.624644 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:08.624619 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/836acdb1-4e39-472f-ba72-a07d61f977bf-istio-envoy\") pod \"e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl\" (UID: \"836acdb1-4e39-472f-ba72-a07d61f977bf\") " pod="openshift-ingress/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl" Nov 01 20:58:08.624808 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:08.624791 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/836acdb1-4e39-472f-ba72-a07d61f977bf-istio-token\") pod \"e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl\" (UID: \"836acdb1-4e39-472f-ba72-a07d61f977bf\") " pod="openshift-ingress/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl" Nov 01 20:58:08.645064 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:08.642072 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl"] Nov 01 20:58:08.787064 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:08.763845 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/836acdb1-4e39-472f-ba72-a07d61f977bf-istiod-ca-cert\") pod \"e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl\" (UID: \"836acdb1-4e39-472f-ba72-a07d61f977bf\") " pod="openshift-ingress/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl" Nov 01 20:58:08.787064 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:08.763950 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/836acdb1-4e39-472f-ba72-a07d61f977bf-istio-data\") pod \"e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl\" (UID: \"836acdb1-4e39-472f-ba72-a07d61f977bf\") " pod="openshift-ingress/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl" Nov 01 20:58:08.787064 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:08.764084 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/836acdb1-4e39-472f-ba72-a07d61f977bf-workload-socket\") pod \"e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl\" (UID: \"836acdb1-4e39-472f-ba72-a07d61f977bf\") " pod="openshift-ingress/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl" Nov 01 20:58:08.787064 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:08.764183 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/836acdb1-4e39-472f-ba72-a07d61f977bf-istio-envoy\") pod \"e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl\" (UID: \"836acdb1-4e39-472f-ba72-a07d61f977bf\") " pod="openshift-ingress/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl" Nov 01 20:58:08.787064 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:08.764248 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/836acdb1-4e39-472f-ba72-a07d61f977bf-istio-token\") pod \"e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl\" (UID: \"836acdb1-4e39-472f-ba72-a07d61f977bf\") " pod="openshift-ingress/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl" Nov 01 20:58:08.787064 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:08.764296 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/836acdb1-4e39-472f-ba72-a07d61f977bf-credential-socket\") pod \"e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl\" (UID: \"836acdb1-4e39-472f-ba72-a07d61f977bf\") " pod="openshift-ingress/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl" Nov 01 20:58:08.787064 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:08.764323 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnzn6\" (UniqueName: \"kubernetes.io/projected/836acdb1-4e39-472f-ba72-a07d61f977bf-kube-api-access-bnzn6\") pod \"e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl\" (UID: \"836acdb1-4e39-472f-ba72-a07d61f977bf\") " pod="openshift-ingress/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl" Nov 01 20:58:08.787064 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:08.764371 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/836acdb1-4e39-472f-ba72-a07d61f977bf-istio-podinfo\") pod \"e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl\" (UID: \"836acdb1-4e39-472f-ba72-a07d61f977bf\") " pod="openshift-ingress/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl" Nov 01 20:58:08.787064 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:08.764398 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/836acdb1-4e39-472f-ba72-a07d61f977bf-workload-certs\") pod \"e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl\" (UID: \"836acdb1-4e39-472f-ba72-a07d61f977bf\") " pod="openshift-ingress/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl" Nov 01 20:58:08.787064 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:08.764955 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/836acdb1-4e39-472f-ba72-a07d61f977bf-workload-certs\") pod \"e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl\" (UID: \"836acdb1-4e39-472f-ba72-a07d61f977bf\") " pod="openshift-ingress/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl" Nov 01 20:58:08.787064 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:08.776724 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/836acdb1-4e39-472f-ba72-a07d61f977bf-istio-envoy\") pod \"e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl\" (UID: \"836acdb1-4e39-472f-ba72-a07d61f977bf\") " pod="openshift-ingress/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl" Nov 01 20:58:08.787064 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:08.785359 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/836acdb1-4e39-472f-ba72-a07d61f977bf-istiod-ca-cert\") pod \"e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl\" (UID: \"836acdb1-4e39-472f-ba72-a07d61f977bf\") " pod="openshift-ingress/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl" Nov 01 20:58:08.787064 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:08.785671 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/836acdb1-4e39-472f-ba72-a07d61f977bf-istio-data\") pod \"e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl\" (UID: \"836acdb1-4e39-472f-ba72-a07d61f977bf\") " pod="openshift-ingress/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl" Nov 01 20:58:08.787064 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:08.785924 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/836acdb1-4e39-472f-ba72-a07d61f977bf-workload-socket\") pod \"e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl\" (UID: \"836acdb1-4e39-472f-ba72-a07d61f977bf\") " pod="openshift-ingress/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl" Nov 01 20:58:08.787064 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:08.786211 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/836acdb1-4e39-472f-ba72-a07d61f977bf-credential-socket\") pod \"e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl\" (UID: \"836acdb1-4e39-472f-ba72-a07d61f977bf\") " pod="openshift-ingress/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl" Nov 01 20:58:08.837096 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:08.837040 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/836acdb1-4e39-472f-ba72-a07d61f977bf-istio-podinfo\") pod \"e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl\" (UID: \"836acdb1-4e39-472f-ba72-a07d61f977bf\") " pod="openshift-ingress/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl" Nov 01 20:58:08.856669 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:08.856626 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnzn6\" (UniqueName: \"kubernetes.io/projected/836acdb1-4e39-472f-ba72-a07d61f977bf-kube-api-access-bnzn6\") pod \"e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl\" (UID: \"836acdb1-4e39-472f-ba72-a07d61f977bf\") " pod="openshift-ingress/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl" Nov 01 20:58:08.859924 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:08.859863 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/836acdb1-4e39-472f-ba72-a07d61f977bf-istio-token\") pod \"e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl\" (UID: \"836acdb1-4e39-472f-ba72-a07d61f977bf\") " pod="openshift-ingress/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl" Nov 01 20:58:09.023300 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:09.023258 4484 patch_prober.go:29] interesting pod/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 20:58:09.023509 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:09.023337 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2" podUID="71cb71c6-14d7-4e6d-a269-22b3b66fe170" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 20:58:09.026479 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:09.026443 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:09.036431 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:09.036379 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl" Nov 01 20:58:09.208021 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:09.206128 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:09.554096 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:09.554059 4484 patch_prober.go:29] interesting pod/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 20:58:09.554493 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:09.554115 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h" podUID="3b91fbd7-5fe8-42de-b6be-fcbf67244747" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 20:58:09.561103 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:58:09.561073 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod836acdb1_4e39_472f_ba72_a07d61f977bf.slice/crio-0224f3bba9f169c15d5cc0f2cfa3c1294a84f005fcf0539df7a6faf804dd94f1 WatchSource:0}: Error finding container 0224f3bba9f169c15d5cc0f2cfa3c1294a84f005fcf0539df7a6faf804dd94f1: Status 404 returned error can't find the container with id 0224f3bba9f169c15d5cc0f2cfa3c1294a84f005fcf0539df7a6faf804dd94f1 Nov 01 20:58:09.561705 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:09.561687 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl"] Nov 01 20:58:09.563961 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:09.563926 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:58:09.564062 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:09.564008 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:58:09.564062 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:09.564045 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:58:09.573927 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:09.573888 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:10.110023 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:10.105784 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822"] Nov 01 20:58:10.111331 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:10.111296 4484 cpu_manager.go:414] "containerMap: removing container" podUID="35f67043-6238-4386-a37a-3c2dbed1fae9" containerName="istio-proxy" Nov 01 20:58:10.111506 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:10.111492 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35f67043-6238-4386-a37a-3c2dbed1fae9" containerName="istio-proxy" Nov 01 20:58:10.135027 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:10.134619 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822" Nov 01 20:58:10.207096 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:10.207054 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lrw2\" (UniqueName: \"kubernetes.io/projected/4048751a-034d-4ab2-937f-6161505e4166-kube-api-access-9lrw2\") pod \"e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822\" (UID: \"4048751a-034d-4ab2-937f-6161505e4166\") " pod="openshift-ingress/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822" Nov 01 20:58:10.250017 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:10.220100 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4048751a-034d-4ab2-937f-6161505e4166-istio-data\") pod \"e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822\" (UID: \"4048751a-034d-4ab2-937f-6161505e4166\") " pod="openshift-ingress/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822" Nov 01 20:58:10.250385 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:10.250341 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4048751a-034d-4ab2-937f-6161505e4166-istio-token\") pod \"e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822\" (UID: \"4048751a-034d-4ab2-937f-6161505e4166\") " pod="openshift-ingress/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822" Nov 01 20:58:10.265027 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:10.261134 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4048751a-034d-4ab2-937f-6161505e4166-credential-socket\") pod \"e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822\" (UID: \"4048751a-034d-4ab2-937f-6161505e4166\") " pod="openshift-ingress/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822" Nov 01 20:58:10.290790 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:10.277168 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4048751a-034d-4ab2-937f-6161505e4166-istiod-ca-cert\") pod \"e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822\" (UID: \"4048751a-034d-4ab2-937f-6161505e4166\") " pod="openshift-ingress/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822" Nov 01 20:58:10.290790 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:10.277230 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4048751a-034d-4ab2-937f-6161505e4166-workload-certs\") pod \"e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822\" (UID: \"4048751a-034d-4ab2-937f-6161505e4166\") " pod="openshift-ingress/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822" Nov 01 20:58:10.290790 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:10.277371 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4048751a-034d-4ab2-937f-6161505e4166-workload-socket\") pod \"e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822\" (UID: \"4048751a-034d-4ab2-937f-6161505e4166\") " pod="openshift-ingress/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822" Nov 01 20:58:10.290790 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:10.277514 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4048751a-034d-4ab2-937f-6161505e4166-istio-envoy\") pod \"e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822\" (UID: \"4048751a-034d-4ab2-937f-6161505e4166\") " pod="openshift-ingress/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822" Nov 01 20:58:10.290790 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:10.277567 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4048751a-034d-4ab2-937f-6161505e4166-istio-podinfo\") pod \"e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822\" (UID: \"4048751a-034d-4ab2-937f-6161505e4166\") " pod="openshift-ingress/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822" Nov 01 20:58:10.290790 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:10.249354 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-df700a-openshift-default-dockercfg-4kcdv\"" Nov 01 20:58:10.290790 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:10.254634 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:10.303250 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:10.297171 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822"] Nov 01 20:58:10.380069 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:10.379378 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4048751a-034d-4ab2-937f-6161505e4166-workload-socket\") pod \"e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822\" (UID: \"4048751a-034d-4ab2-937f-6161505e4166\") " pod="openshift-ingress/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822" Nov 01 20:58:10.380069 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:10.379481 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4048751a-034d-4ab2-937f-6161505e4166-istio-envoy\") pod \"e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822\" (UID: \"4048751a-034d-4ab2-937f-6161505e4166\") " pod="openshift-ingress/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822" Nov 01 20:58:10.380069 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:10.379535 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4048751a-034d-4ab2-937f-6161505e4166-istio-podinfo\") pod \"e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822\" (UID: \"4048751a-034d-4ab2-937f-6161505e4166\") " pod="openshift-ingress/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822" Nov 01 20:58:10.380069 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:10.379617 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lrw2\" (UniqueName: \"kubernetes.io/projected/4048751a-034d-4ab2-937f-6161505e4166-kube-api-access-9lrw2\") pod \"e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822\" (UID: \"4048751a-034d-4ab2-937f-6161505e4166\") " pod="openshift-ingress/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822" Nov 01 20:58:10.380069 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:10.379664 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4048751a-034d-4ab2-937f-6161505e4166-istio-data\") pod \"e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822\" (UID: \"4048751a-034d-4ab2-937f-6161505e4166\") " pod="openshift-ingress/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822" Nov 01 20:58:10.380069 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:10.379719 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4048751a-034d-4ab2-937f-6161505e4166-istio-token\") pod \"e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822\" (UID: \"4048751a-034d-4ab2-937f-6161505e4166\") " pod="openshift-ingress/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822" Nov 01 20:58:10.380069 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:10.379780 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4048751a-034d-4ab2-937f-6161505e4166-credential-socket\") pod \"e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822\" (UID: \"4048751a-034d-4ab2-937f-6161505e4166\") " pod="openshift-ingress/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822" Nov 01 20:58:10.380069 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:10.379812 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4048751a-034d-4ab2-937f-6161505e4166-istiod-ca-cert\") pod \"e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822\" (UID: \"4048751a-034d-4ab2-937f-6161505e4166\") " pod="openshift-ingress/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822" Nov 01 20:58:10.380069 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:10.379844 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4048751a-034d-4ab2-937f-6161505e4166-workload-certs\") pod \"e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822\" (UID: \"4048751a-034d-4ab2-937f-6161505e4166\") " pod="openshift-ingress/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822" Nov 01 20:58:10.380658 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:10.380337 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4048751a-034d-4ab2-937f-6161505e4166-workload-certs\") pod \"e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822\" (UID: \"4048751a-034d-4ab2-937f-6161505e4166\") " pod="openshift-ingress/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822" Nov 01 20:58:10.380658 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:10.380557 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4048751a-034d-4ab2-937f-6161505e4166-workload-socket\") pod \"e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822\" (UID: \"4048751a-034d-4ab2-937f-6161505e4166\") " pod="openshift-ingress/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822" Nov 01 20:58:10.390537 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:10.385292 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4048751a-034d-4ab2-937f-6161505e4166-istio-data\") pod \"e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822\" (UID: \"4048751a-034d-4ab2-937f-6161505e4166\") " pod="openshift-ingress/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822" Nov 01 20:58:10.390537 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:10.385549 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4048751a-034d-4ab2-937f-6161505e4166-credential-socket\") pod \"e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822\" (UID: \"4048751a-034d-4ab2-937f-6161505e4166\") " pod="openshift-ingress/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822" Nov 01 20:58:10.390537 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:10.386178 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4048751a-034d-4ab2-937f-6161505e4166-istiod-ca-cert\") pod \"e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822\" (UID: \"4048751a-034d-4ab2-937f-6161505e4166\") " pod="openshift-ingress/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822" Nov 01 20:58:10.423075 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:10.422315 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4048751a-034d-4ab2-937f-6161505e4166-istio-podinfo\") pod \"e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822\" (UID: \"4048751a-034d-4ab2-937f-6161505e4166\") " pod="openshift-ingress/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822" Nov 01 20:58:10.448228 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:10.448129 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4048751a-034d-4ab2-937f-6161505e4166-istio-envoy\") pod \"e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822\" (UID: \"4048751a-034d-4ab2-937f-6161505e4166\") " pod="openshift-ingress/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822" Nov 01 20:58:10.456549 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:10.455274 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl" event={"ID":"836acdb1-4e39-472f-ba72-a07d61f977bf","Type":"ContainerStarted","Data":"2bd1c28a8eb5bb55790b360ae048808b234637ab065e345fb7be2188b90906b1"} Nov 01 20:58:10.456549 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:10.455355 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl" event={"ID":"836acdb1-4e39-472f-ba72-a07d61f977bf","Type":"ContainerStarted","Data":"0224f3bba9f169c15d5cc0f2cfa3c1294a84f005fcf0539df7a6faf804dd94f1"} Nov 01 20:58:10.462340 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:10.462306 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4048751a-034d-4ab2-937f-6161505e4166-istio-token\") pod \"e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822\" (UID: \"4048751a-034d-4ab2-937f-6161505e4166\") " pod="openshift-ingress/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822" Nov 01 20:58:10.463440 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:10.463222 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lrw2\" (UniqueName: \"kubernetes.io/projected/4048751a-034d-4ab2-937f-6161505e4166-kube-api-access-9lrw2\") pod \"e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822\" (UID: \"4048751a-034d-4ab2-937f-6161505e4166\") " pod="openshift-ingress/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822" Nov 01 20:58:10.516747 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:10.515385 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822" Nov 01 20:58:10.577499 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:10.577445 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl" podStartSLOduration=2.577430035 podStartE2EDuration="2.577430035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:58:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:58:10.576393284 +0000 UTC m=+1641.354033431" watchObservedRunningTime="2026-11-01 20:58:10.577430035 +0000 UTC m=+1641.355070169" Nov 01 20:58:10.999953 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:10.994792 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:11.019739 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:11.019698 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822"] Nov 01 20:58:11.024349 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:58:11.024316 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4048751a_034d_4ab2_937f_6161505e4166.slice/crio-de1e21d1da6c4685d968730c466b672600e5c1c820fcee804503625619455ad8 WatchSource:0}: Error finding container de1e21d1da6c4685d968730c466b672600e5c1c820fcee804503625619455ad8: Status 404 returned error can't find the container with id de1e21d1da6c4685d968730c466b672600e5c1c820fcee804503625619455ad8 Nov 01 20:58:11.027102 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:11.027060 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:58:11.027196 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:11.027132 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:58:11.027196 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:11.027162 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:58:11.037173 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:11.037134 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl" Nov 01 20:58:11.046165 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:11.046142 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl" Nov 01 20:58:11.199677 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:11.194795 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:11.465622 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:11.465526 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822" event={"ID":"4048751a-034d-4ab2-937f-6161505e4166","Type":"ContainerStarted","Data":"20c6f9f9c84adcb86aad2f2a31f2cfad160d309e9a25f92974e5c257bf5bfc04"} Nov 01 20:58:11.465622 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:11.465581 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822" event={"ID":"4048751a-034d-4ab2-937f-6161505e4166","Type":"ContainerStarted","Data":"de1e21d1da6c4685d968730c466b672600e5c1c820fcee804503625619455ad8"} Nov 01 20:58:11.465952 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:11.465932 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl" Nov 01 20:58:11.466987 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:11.466963 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl" Nov 01 20:58:11.516311 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:11.516279 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822" Nov 01 20:58:11.586884 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:11.586831 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822" podStartSLOduration=1.586817004 podStartE2EDuration="1.586817004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:58:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:58:11.584458352 +0000 UTC m=+1642.362098488" watchObservedRunningTime="2026-11-01 20:58:11.586817004 +0000 UTC m=+1642.364457228" Nov 01 20:58:12.362070 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.362040 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2" Nov 01 20:58:12.382573 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.382544 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h" Nov 01 20:58:12.459060 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.459025 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/71cb71c6-14d7-4e6d-a269-22b3b66fe170-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71cb71c6-14d7-4e6d-a269-22b3b66fe170-istio-podinfo\") pod \"71cb71c6-14d7-4e6d-a269-22b3b66fe170\" (UID: \"71cb71c6-14d7-4e6d-a269-22b3b66fe170\") " Nov 01 20:58:12.459256 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.459098 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71cb71c6-14d7-4e6d-a269-22b3b66fe170-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71cb71c6-14d7-4e6d-a269-22b3b66fe170-istio-data\") pod \"71cb71c6-14d7-4e6d-a269-22b3b66fe170\" (UID: \"71cb71c6-14d7-4e6d-a269-22b3b66fe170\") " Nov 01 20:58:12.459256 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.459119 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71cb71c6-14d7-4e6d-a269-22b3b66fe170-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71cb71c6-14d7-4e6d-a269-22b3b66fe170-istio-envoy\") pod \"71cb71c6-14d7-4e6d-a269-22b3b66fe170\" (UID: \"71cb71c6-14d7-4e6d-a269-22b3b66fe170\") " Nov 01 20:58:12.459256 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.459183 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71cb71c6-14d7-4e6d-a269-22b3b66fe170-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71cb71c6-14d7-4e6d-a269-22b3b66fe170-workload-socket\") pod \"71cb71c6-14d7-4e6d-a269-22b3b66fe170\" (UID: \"71cb71c6-14d7-4e6d-a269-22b3b66fe170\") " Nov 01 20:58:12.459256 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.459233 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71cb71c6-14d7-4e6d-a269-22b3b66fe170-kube-api-access-98jc5\" (UniqueName: \"kubernetes.io/projected/71cb71c6-14d7-4e6d-a269-22b3b66fe170-kube-api-access-98jc5\") pod \"71cb71c6-14d7-4e6d-a269-22b3b66fe170\" (UID: \"71cb71c6-14d7-4e6d-a269-22b3b66fe170\") " Nov 01 20:58:12.459487 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.459262 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71cb71c6-14d7-4e6d-a269-22b3b66fe170-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71cb71c6-14d7-4e6d-a269-22b3b66fe170-istiod-ca-cert\") pod \"71cb71c6-14d7-4e6d-a269-22b3b66fe170\" (UID: \"71cb71c6-14d7-4e6d-a269-22b3b66fe170\") " Nov 01 20:58:12.459487 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.459357 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71cb71c6-14d7-4e6d-a269-22b3b66fe170-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71cb71c6-14d7-4e6d-a269-22b3b66fe170-workload-certs\") pod \"71cb71c6-14d7-4e6d-a269-22b3b66fe170\" (UID: \"71cb71c6-14d7-4e6d-a269-22b3b66fe170\") " Nov 01 20:58:12.459487 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.459384 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71cb71c6-14d7-4e6d-a269-22b3b66fe170-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71cb71c6-14d7-4e6d-a269-22b3b66fe170-credential-socket\") pod \"71cb71c6-14d7-4e6d-a269-22b3b66fe170\" (UID: \"71cb71c6-14d7-4e6d-a269-22b3b66fe170\") " Nov 01 20:58:12.459487 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.459419 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71cb71c6-14d7-4e6d-a269-22b3b66fe170-istio-data" pod "71cb71c6-14d7-4e6d-a269-22b3b66fe170" (UID: "71cb71c6-14d7-4e6d-a269-22b3b66fe170"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:58:12.459487 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.459446 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71cb71c6-14d7-4e6d-a269-22b3b66fe170-istio-token\" (UniqueName: \"kubernetes.io/projected/71cb71c6-14d7-4e6d-a269-22b3b66fe170-istio-token\") pod \"71cb71c6-14d7-4e6d-a269-22b3b66fe170\" (UID: \"71cb71c6-14d7-4e6d-a269-22b3b66fe170\") " Nov 01 20:58:12.459741 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.459578 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71cb71c6-14d7-4e6d-a269-22b3b66fe170-workload-certs" pod "71cb71c6-14d7-4e6d-a269-22b3b66fe170" (UID: "71cb71c6-14d7-4e6d-a269-22b3b66fe170"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:58:12.459741 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.459590 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71cb71c6-14d7-4e6d-a269-22b3b66fe170-workload-socket" pod "71cb71c6-14d7-4e6d-a269-22b3b66fe170" (UID: "71cb71c6-14d7-4e6d-a269-22b3b66fe170"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:58:12.459741 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.459690 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71cb71c6-14d7-4e6d-a269-22b3b66fe170-istiod-ca-cert" pod "71cb71c6-14d7-4e6d-a269-22b3b66fe170" (UID: "71cb71c6-14d7-4e6d-a269-22b3b66fe170"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:58:12.459741 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.459697 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71cb71c6-14d7-4e6d-a269-22b3b66fe170-credential-socket" pod "71cb71c6-14d7-4e6d-a269-22b3b66fe170" (UID: "71cb71c6-14d7-4e6d-a269-22b3b66fe170"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:58:12.460460 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.460362 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71cb71c6-14d7-4e6d-a269-22b3b66fe170-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:12.460460 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.460394 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71cb71c6-14d7-4e6d-a269-22b3b66fe170-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:12.460460 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.460410 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71cb71c6-14d7-4e6d-a269-22b3b66fe170-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:12.460460 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.460424 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71cb71c6-14d7-4e6d-a269-22b3b66fe170-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:12.460460 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.460438 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71cb71c6-14d7-4e6d-a269-22b3b66fe170-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:12.462096 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.462073 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71cb71c6-14d7-4e6d-a269-22b3b66fe170-kube-api-access-98jc5" pod "71cb71c6-14d7-4e6d-a269-22b3b66fe170" (UID: "71cb71c6-14d7-4e6d-a269-22b3b66fe170"). InnerVolumeSpecName "kube-api-access-98jc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:58:12.462174 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.462142 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71cb71c6-14d7-4e6d-a269-22b3b66fe170-istio-envoy" pod "71cb71c6-14d7-4e6d-a269-22b3b66fe170" (UID: "71cb71c6-14d7-4e6d-a269-22b3b66fe170"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:58:12.462211 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.462178 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71cb71c6-14d7-4e6d-a269-22b3b66fe170-istio-podinfo" pod "71cb71c6-14d7-4e6d-a269-22b3b66fe170" (UID: "71cb71c6-14d7-4e6d-a269-22b3b66fe170"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:58:12.462255 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.462212 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71cb71c6-14d7-4e6d-a269-22b3b66fe170-istio-token" pod "71cb71c6-14d7-4e6d-a269-22b3b66fe170" (UID: "71cb71c6-14d7-4e6d-a269-22b3b66fe170"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:58:12.475948 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.475915 4484 generic.go:356] "Generic (PLEG): container finished" podID="3b91fbd7-5fe8-42de-b6be-fcbf67244747" containerID="e83d80fb7c64778872459faedd92c0daf10744f5a1c69d26dbd0648d54b98295" exitCode=0 Nov 01 20:58:12.476132 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.475963 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h" Nov 01 20:58:12.476132 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.476010 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h" event={"ID":"3b91fbd7-5fe8-42de-b6be-fcbf67244747","Type":"ContainerDied","Data":"e83d80fb7c64778872459faedd92c0daf10744f5a1c69d26dbd0648d54b98295"} Nov 01 20:58:12.476132 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.476062 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h" event={"ID":"3b91fbd7-5fe8-42de-b6be-fcbf67244747","Type":"ContainerDied","Data":"fa13d552b60e43d3b8686d009ed320c6b164acfbe20f502fb04ab9a929f9fc9f"} Nov 01 20:58:12.476132 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.476084 4484 scope.go:122] "RemoveContainer" containerID="e83d80fb7c64778872459faedd92c0daf10744f5a1c69d26dbd0648d54b98295" Nov 01 20:58:12.477443 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.477426 4484 generic.go:356] "Generic (PLEG): container finished" podID="71cb71c6-14d7-4e6d-a269-22b3b66fe170" containerID="cd11d06566365a70138161b43321e9b9c4c45deb5dc7f2cf42208dc04e355338" exitCode=0 Nov 01 20:58:12.477534 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.477463 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2" Nov 01 20:58:12.477534 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.477516 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2" event={"ID":"71cb71c6-14d7-4e6d-a269-22b3b66fe170","Type":"ContainerDied","Data":"cd11d06566365a70138161b43321e9b9c4c45deb5dc7f2cf42208dc04e355338"} Nov 01 20:58:12.477610 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.477559 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2" event={"ID":"71cb71c6-14d7-4e6d-a269-22b3b66fe170","Type":"ContainerDied","Data":"fca6534bb45ef07689ef179134de8549695b7cee94014ae2dd0b98ae85e54e8b"} Nov 01 20:58:12.489110 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.489083 4484 scope.go:122] "RemoveContainer" containerID="a03bdfbf3eeb002f5f1455a0bc0571dec18e0e0f9891e4e54033d8ef3a95366c" Nov 01 20:58:12.501893 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.501868 4484 scope.go:122] "RemoveContainer" containerID="e83d80fb7c64778872459faedd92c0daf10744f5a1c69d26dbd0648d54b98295" Nov 01 20:58:12.502307 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:58:12.502286 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e83d80fb7c64778872459faedd92c0daf10744f5a1c69d26dbd0648d54b98295\": container with ID starting with e83d80fb7c64778872459faedd92c0daf10744f5a1c69d26dbd0648d54b98295 not found: ID does not exist" containerID="e83d80fb7c64778872459faedd92c0daf10744f5a1c69d26dbd0648d54b98295" Nov 01 20:58:12.502417 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.502318 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e83d80fb7c64778872459faedd92c0daf10744f5a1c69d26dbd0648d54b98295"} err="failed to get container status \"e83d80fb7c64778872459faedd92c0daf10744f5a1c69d26dbd0648d54b98295\": rpc error: code = NotFound desc = could not find container \"e83d80fb7c64778872459faedd92c0daf10744f5a1c69d26dbd0648d54b98295\": container with ID starting with e83d80fb7c64778872459faedd92c0daf10744f5a1c69d26dbd0648d54b98295 not found: ID does not exist" Nov 01 20:58:12.502417 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.502345 4484 scope.go:122] "RemoveContainer" containerID="a03bdfbf3eeb002f5f1455a0bc0571dec18e0e0f9891e4e54033d8ef3a95366c" Nov 01 20:58:12.502673 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:58:12.502655 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a03bdfbf3eeb002f5f1455a0bc0571dec18e0e0f9891e4e54033d8ef3a95366c\": container with ID starting with a03bdfbf3eeb002f5f1455a0bc0571dec18e0e0f9891e4e54033d8ef3a95366c not found: ID does not exist" containerID="a03bdfbf3eeb002f5f1455a0bc0571dec18e0e0f9891e4e54033d8ef3a95366c" Nov 01 20:58:12.502755 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.502679 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a03bdfbf3eeb002f5f1455a0bc0571dec18e0e0f9891e4e54033d8ef3a95366c"} err="failed to get container status \"a03bdfbf3eeb002f5f1455a0bc0571dec18e0e0f9891e4e54033d8ef3a95366c\": rpc error: code = NotFound desc = could not find container \"a03bdfbf3eeb002f5f1455a0bc0571dec18e0e0f9891e4e54033d8ef3a95366c\": container with ID starting with a03bdfbf3eeb002f5f1455a0bc0571dec18e0e0f9891e4e54033d8ef3a95366c not found: ID does not exist" Nov 01 20:58:12.502755 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.502697 4484 scope.go:122] "RemoveContainer" containerID="cd11d06566365a70138161b43321e9b9c4c45deb5dc7f2cf42208dc04e355338" Nov 01 20:58:12.516572 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.516543 4484 scope.go:122] "RemoveContainer" containerID="cd11d06566365a70138161b43321e9b9c4c45deb5dc7f2cf42208dc04e355338" Nov 01 20:58:12.516954 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:58:12.516929 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd11d06566365a70138161b43321e9b9c4c45deb5dc7f2cf42208dc04e355338\": container with ID starting with cd11d06566365a70138161b43321e9b9c4c45deb5dc7f2cf42208dc04e355338 not found: ID does not exist" containerID="cd11d06566365a70138161b43321e9b9c4c45deb5dc7f2cf42208dc04e355338" Nov 01 20:58:12.517088 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.516965 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd11d06566365a70138161b43321e9b9c4c45deb5dc7f2cf42208dc04e355338"} err="failed to get container status \"cd11d06566365a70138161b43321e9b9c4c45deb5dc7f2cf42208dc04e355338\": rpc error: code = NotFound desc = could not find container \"cd11d06566365a70138161b43321e9b9c4c45deb5dc7f2cf42208dc04e355338\": container with ID starting with cd11d06566365a70138161b43321e9b9c4c45deb5dc7f2cf42208dc04e355338 not found: ID does not exist" Nov 01 20:58:12.524146 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.524118 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822" Nov 01 20:58:12.561727 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.561686 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b91fbd7-5fe8-42de-b6be-fcbf67244747-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b91fbd7-5fe8-42de-b6be-fcbf67244747-istio-data\") pod \"3b91fbd7-5fe8-42de-b6be-fcbf67244747\" (UID: \"3b91fbd7-5fe8-42de-b6be-fcbf67244747\") " Nov 01 20:58:12.561902 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.561786 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b91fbd7-5fe8-42de-b6be-fcbf67244747-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b91fbd7-5fe8-42de-b6be-fcbf67244747-workload-socket\") pod \"3b91fbd7-5fe8-42de-b6be-fcbf67244747\" (UID: \"3b91fbd7-5fe8-42de-b6be-fcbf67244747\") " Nov 01 20:58:12.561902 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.561835 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b91fbd7-5fe8-42de-b6be-fcbf67244747-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b91fbd7-5fe8-42de-b6be-fcbf67244747-istiod-ca-cert\") pod \"3b91fbd7-5fe8-42de-b6be-fcbf67244747\" (UID: \"3b91fbd7-5fe8-42de-b6be-fcbf67244747\") " Nov 01 20:58:12.562073 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.561898 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b91fbd7-5fe8-42de-b6be-fcbf67244747-kube-api-access-bn7m8\" (UniqueName: \"kubernetes.io/projected/3b91fbd7-5fe8-42de-b6be-fcbf67244747-kube-api-access-bn7m8\") pod \"3b91fbd7-5fe8-42de-b6be-fcbf67244747\" (UID: \"3b91fbd7-5fe8-42de-b6be-fcbf67244747\") " Nov 01 20:58:12.562146 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.562084 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b91fbd7-5fe8-42de-b6be-fcbf67244747-istio-token\" (UniqueName: \"kubernetes.io/projected/3b91fbd7-5fe8-42de-b6be-fcbf67244747-istio-token\") pod \"3b91fbd7-5fe8-42de-b6be-fcbf67244747\" (UID: \"3b91fbd7-5fe8-42de-b6be-fcbf67244747\") " Nov 01 20:58:12.562146 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.562087 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b91fbd7-5fe8-42de-b6be-fcbf67244747-workload-socket" pod "3b91fbd7-5fe8-42de-b6be-fcbf67244747" (UID: "3b91fbd7-5fe8-42de-b6be-fcbf67244747"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:58:12.562146 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.561984 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b91fbd7-5fe8-42de-b6be-fcbf67244747-istio-data" pod "3b91fbd7-5fe8-42de-b6be-fcbf67244747" (UID: "3b91fbd7-5fe8-42de-b6be-fcbf67244747"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:58:12.562287 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.562167 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b91fbd7-5fe8-42de-b6be-fcbf67244747-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b91fbd7-5fe8-42de-b6be-fcbf67244747-istio-podinfo\") pod \"3b91fbd7-5fe8-42de-b6be-fcbf67244747\" (UID: \"3b91fbd7-5fe8-42de-b6be-fcbf67244747\") " Nov 01 20:58:12.562341 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.562308 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b91fbd7-5fe8-42de-b6be-fcbf67244747-istiod-ca-cert" pod "3b91fbd7-5fe8-42de-b6be-fcbf67244747" (UID: "3b91fbd7-5fe8-42de-b6be-fcbf67244747"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:58:12.562341 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.562322 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b91fbd7-5fe8-42de-b6be-fcbf67244747-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b91fbd7-5fe8-42de-b6be-fcbf67244747-workload-certs\") pod \"3b91fbd7-5fe8-42de-b6be-fcbf67244747\" (UID: \"3b91fbd7-5fe8-42de-b6be-fcbf67244747\") " Nov 01 20:58:12.562437 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.562411 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b91fbd7-5fe8-42de-b6be-fcbf67244747-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b91fbd7-5fe8-42de-b6be-fcbf67244747-credential-socket\") pod \"3b91fbd7-5fe8-42de-b6be-fcbf67244747\" (UID: \"3b91fbd7-5fe8-42de-b6be-fcbf67244747\") " Nov 01 20:58:12.562496 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.562447 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b91fbd7-5fe8-42de-b6be-fcbf67244747-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b91fbd7-5fe8-42de-b6be-fcbf67244747-istio-envoy\") pod \"3b91fbd7-5fe8-42de-b6be-fcbf67244747\" (UID: \"3b91fbd7-5fe8-42de-b6be-fcbf67244747\") " Nov 01 20:58:12.562571 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.562548 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b91fbd7-5fe8-42de-b6be-fcbf67244747-workload-certs" pod "3b91fbd7-5fe8-42de-b6be-fcbf67244747" (UID: "3b91fbd7-5fe8-42de-b6be-fcbf67244747"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:58:12.562786 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.562753 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b91fbd7-5fe8-42de-b6be-fcbf67244747-credential-socket" pod "3b91fbd7-5fe8-42de-b6be-fcbf67244747" (UID: "3b91fbd7-5fe8-42de-b6be-fcbf67244747"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:58:12.563717 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.563694 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b91fbd7-5fe8-42de-b6be-fcbf67244747-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:12.563836 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.563722 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71cb71c6-14d7-4e6d-a269-22b3b66fe170-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:12.563836 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.563737 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b91fbd7-5fe8-42de-b6be-fcbf67244747-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:12.563836 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.563751 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b91fbd7-5fe8-42de-b6be-fcbf67244747-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:12.563836 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.563765 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b91fbd7-5fe8-42de-b6be-fcbf67244747-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:12.563836 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.563778 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b91fbd7-5fe8-42de-b6be-fcbf67244747-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:12.563836 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.563791 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71cb71c6-14d7-4e6d-a269-22b3b66fe170-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:12.563836 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.563806 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71cb71c6-14d7-4e6d-a269-22b3b66fe170-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:12.563836 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.563820 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98jc5\" (UniqueName: \"kubernetes.io/projected/71cb71c6-14d7-4e6d-a269-22b3b66fe170-kube-api-access-98jc5\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:12.564616 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.564555 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b91fbd7-5fe8-42de-b6be-fcbf67244747-istio-envoy" pod "3b91fbd7-5fe8-42de-b6be-fcbf67244747" (UID: "3b91fbd7-5fe8-42de-b6be-fcbf67244747"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:58:12.564706 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.564612 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b91fbd7-5fe8-42de-b6be-fcbf67244747-istio-token" pod "3b91fbd7-5fe8-42de-b6be-fcbf67244747" (UID: "3b91fbd7-5fe8-42de-b6be-fcbf67244747"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:58:12.564820 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.564803 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b91fbd7-5fe8-42de-b6be-fcbf67244747-kube-api-access-bn7m8" pod "3b91fbd7-5fe8-42de-b6be-fcbf67244747" (UID: "3b91fbd7-5fe8-42de-b6be-fcbf67244747"). InnerVolumeSpecName "kube-api-access-bn7m8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:58:12.564874 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.564860 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b91fbd7-5fe8-42de-b6be-fcbf67244747-istio-podinfo" pod "3b91fbd7-5fe8-42de-b6be-fcbf67244747" (UID: "3b91fbd7-5fe8-42de-b6be-fcbf67244747"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:58:12.666275 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.666231 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn7m8\" (UniqueName: \"kubernetes.io/projected/3b91fbd7-5fe8-42de-b6be-fcbf67244747-kube-api-access-bn7m8\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:12.666275 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.666277 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b91fbd7-5fe8-42de-b6be-fcbf67244747-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:12.666822 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.666293 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b91fbd7-5fe8-42de-b6be-fcbf67244747-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:12.666822 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.666307 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b91fbd7-5fe8-42de-b6be-fcbf67244747-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:58:12.890117 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.890036 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2"] Nov 01 20:58:12.894197 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:12.894167 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-6nzw2"] Nov 01 20:58:13.001783 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:13.001605 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h"] Nov 01 20:58:13.010738 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:13.010698 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a5ab4417-openshift-default-86cdc589-8hn7h"] Nov 01 20:58:13.373294 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:13.373258 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b91fbd7-5fe8-42de-b6be-fcbf67244747" path="/var/lib/kubelet/pods/3b91fbd7-5fe8-42de-b6be-fcbf67244747/volumes" Nov 01 20:58:13.373822 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:13.373809 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71cb71c6-14d7-4e6d-a269-22b3b66fe170" path="/var/lib/kubelet/pods/71cb71c6-14d7-4e6d-a269-22b3b66fe170/volumes" Nov 01 20:58:13.487902 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:13.487868 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822" Nov 01 20:58:13.489414 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:13.489394 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822" Nov 01 20:58:13.522836 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:13.522803 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:13.699808 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:13.699700 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:13.861063 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:13.861027 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:14.359354 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:14.359318 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:14.530574 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:14.530537 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:14.724865 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:14.724761 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:15.768507 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:58:15.768471 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 20:58:16.269924 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:16.269866 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:16.825374 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:16.825342 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:17.023035 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:17.021970 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:17.238073 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:17.232838 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b27e66-5db66b9467-z8rv7"] Nov 01 20:58:17.238073 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:17.234431 4484 cpu_manager.go:414] "containerMap: removing container" podUID="71cb71c6-14d7-4e6d-a269-22b3b66fe170" containerName="istio-proxy" Nov 01 20:58:17.238073 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:17.234462 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71cb71c6-14d7-4e6d-a269-22b3b66fe170" containerName="istio-proxy" Nov 01 20:58:17.238073 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:17.234489 4484 cpu_manager.go:414] "containerMap: removing container" podUID="3b91fbd7-5fe8-42de-b6be-fcbf67244747" containerName="istio-proxy" Nov 01 20:58:17.238073 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:17.234499 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b91fbd7-5fe8-42de-b6be-fcbf67244747" containerName="istio-proxy" Nov 01 20:58:17.238073 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:17.234575 4484 cpu_manager.go:414] "containerMap: removing container" podUID="3b91fbd7-5fe8-42de-b6be-fcbf67244747" containerName="istio-proxy" Nov 01 20:58:17.238073 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:17.234582 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b91fbd7-5fe8-42de-b6be-fcbf67244747" containerName="istio-proxy" Nov 01 20:58:17.238073 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:17.234869 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b91fbd7-5fe8-42de-b6be-fcbf67244747" containerName="istio-proxy" Nov 01 20:58:17.238073 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:17.234891 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="71cb71c6-14d7-4e6d-a269-22b3b66fe170" containerName="istio-proxy" Nov 01 20:58:17.254020 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:17.241934 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b27e66-5db66b9467-z8rv7" Nov 01 20:58:17.334601 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:17.327601 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b27e66-5db66b9467-z8rv7"] Nov 01 20:58:17.334601 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:17.330444 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-b27e66-dockercfg-pmvfq\"" Nov 01 20:58:17.334601 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:17.330516 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-b27e66\"" Nov 01 20:58:17.662442 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:17.605833 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd9f8\" (UniqueName: \"kubernetes.io/projected/4b167c30-a633-471e-a8f4-7f6933c81f0d-kube-api-access-sd9f8\") pod \"payload-pre-processing-e2e-api-a-b27e66-5db66b9467-z8rv7\" (UID: \"4b167c30-a633-471e-a8f4-7f6933c81f0d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b27e66-5db66b9467-z8rv7" Nov 01 20:58:17.662442 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:17.605989 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b167c30-a633-471e-a8f4-7f6933c81f0d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b27e66-5db66b9467-z8rv7\" (UID: \"4b167c30-a633-471e-a8f4-7f6933c81f0d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b27e66-5db66b9467-z8rv7" Nov 01 20:58:17.662442 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:17.651060 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b27e66-6df49bd599-n7wtm"] Nov 01 20:58:17.720016 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:17.719876 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b91fbd7-5fe8-42de-b6be-fcbf67244747" containerName="istio-proxy" Nov 01 20:58:17.725638 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:17.721888 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd9f8\" (UniqueName: \"kubernetes.io/projected/4b167c30-a633-471e-a8f4-7f6933c81f0d-kube-api-access-sd9f8\") pod \"payload-pre-processing-e2e-api-a-b27e66-5db66b9467-z8rv7\" (UID: \"4b167c30-a633-471e-a8f4-7f6933c81f0d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b27e66-5db66b9467-z8rv7" Nov 01 20:58:17.725638 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:17.722065 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b167c30-a633-471e-a8f4-7f6933c81f0d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b27e66-5db66b9467-z8rv7\" (UID: \"4b167c30-a633-471e-a8f4-7f6933c81f0d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b27e66-5db66b9467-z8rv7" Nov 01 20:58:17.725638 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:17.723577 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b167c30-a633-471e-a8f4-7f6933c81f0d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b27e66-5db66b9467-z8rv7\" (UID: \"4b167c30-a633-471e-a8f4-7f6933c81f0d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b27e66-5db66b9467-z8rv7" Nov 01 20:58:17.737640 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:17.737607 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b27e66-6df49bd599-n7wtm" Nov 01 20:58:17.744201 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:17.743777 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b27e66-6df49bd599-n7wtm"] Nov 01 20:58:17.782157 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:17.782123 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd9f8\" (UniqueName: \"kubernetes.io/projected/4b167c30-a633-471e-a8f4-7f6933c81f0d-kube-api-access-sd9f8\") pod \"payload-pre-processing-e2e-api-a-b27e66-5db66b9467-z8rv7\" (UID: \"4b167c30-a633-471e-a8f4-7f6933c81f0d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b27e66-5db66b9467-z8rv7" Nov 01 20:58:17.823878 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:17.823839 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ce576d7-7d1e-4b92-ad28-654f4426099c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b27e66-6df49bd599-n7wtm\" (UID: \"3ce576d7-7d1e-4b92-ad28-654f4426099c\") " pod="openshift-ingress/payload-processing-e2e-api-a-b27e66-6df49bd599-n7wtm" Nov 01 20:58:17.824155 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:17.824130 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2cbh\" (UniqueName: \"kubernetes.io/projected/3ce576d7-7d1e-4b92-ad28-654f4426099c-kube-api-access-p2cbh\") pod \"payload-processing-e2e-api-a-b27e66-6df49bd599-n7wtm\" (UID: \"3ce576d7-7d1e-4b92-ad28-654f4426099c\") " pod="openshift-ingress/payload-processing-e2e-api-a-b27e66-6df49bd599-n7wtm" Nov 01 20:58:17.925414 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:17.925327 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2cbh\" (UniqueName: \"kubernetes.io/projected/3ce576d7-7d1e-4b92-ad28-654f4426099c-kube-api-access-p2cbh\") pod \"payload-processing-e2e-api-a-b27e66-6df49bd599-n7wtm\" (UID: \"3ce576d7-7d1e-4b92-ad28-654f4426099c\") " pod="openshift-ingress/payload-processing-e2e-api-a-b27e66-6df49bd599-n7wtm" Nov 01 20:58:17.926015 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:17.925433 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ce576d7-7d1e-4b92-ad28-654f4426099c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b27e66-6df49bd599-n7wtm\" (UID: \"3ce576d7-7d1e-4b92-ad28-654f4426099c\") " pod="openshift-ingress/payload-processing-e2e-api-a-b27e66-6df49bd599-n7wtm" Nov 01 20:58:17.926362 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:17.926338 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ce576d7-7d1e-4b92-ad28-654f4426099c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b27e66-6df49bd599-n7wtm\" (UID: \"3ce576d7-7d1e-4b92-ad28-654f4426099c\") " pod="openshift-ingress/payload-processing-e2e-api-a-b27e66-6df49bd599-n7wtm" Nov 01 20:58:17.957262 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:17.957229 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b27e66-5db66b9467-z8rv7" Nov 01 20:58:17.983591 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:17.983562 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2cbh\" (UniqueName: \"kubernetes.io/projected/3ce576d7-7d1e-4b92-ad28-654f4426099c-kube-api-access-p2cbh\") pod \"payload-processing-e2e-api-a-b27e66-6df49bd599-n7wtm\" (UID: \"3ce576d7-7d1e-4b92-ad28-654f4426099c\") " pod="openshift-ingress/payload-processing-e2e-api-a-b27e66-6df49bd599-n7wtm" Nov 01 20:58:18.064116 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:18.064081 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b27e66-6df49bd599-n7wtm" Nov 01 20:58:18.335512 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:18.335485 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b27e66-5db66b9467-z8rv7"] Nov 01 20:58:18.335635 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:58:18.335624 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b167c30_a633_471e_a8f4_7f6933c81f0d.slice/crio-ade80a9aeafab87ad4f3960a2c1d556f636db4d0f7100af78286f707f196d77d WatchSource:0}: Error finding container ade80a9aeafab87ad4f3960a2c1d556f636db4d0f7100af78286f707f196d77d: Status 404 returned error can't find the container with id ade80a9aeafab87ad4f3960a2c1d556f636db4d0f7100af78286f707f196d77d Nov 01 20:58:18.447152 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:58:18.447123 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ce576d7_7d1e_4b92_ad28_654f4426099c.slice/crio-0342e5ab57e55af4fd8e7c80c20cddf130726d81d53e3d63e5c06eaa7e2107dc WatchSource:0}: Error finding container 0342e5ab57e55af4fd8e7c80c20cddf130726d81d53e3d63e5c06eaa7e2107dc: Status 404 returned error can't find the container with id 0342e5ab57e55af4fd8e7c80c20cddf130726d81d53e3d63e5c06eaa7e2107dc Nov 01 20:58:18.447600 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:18.447577 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b27e66-6df49bd599-n7wtm"] Nov 01 20:58:18.575532 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:18.575496 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b27e66-5db66b9467-z8rv7" event={"ID":"4b167c30-a633-471e-a8f4-7f6933c81f0d","Type":"ContainerStarted","Data":"b77bfc23b1adb869151c34434aa56228107e2ea6994a1f5995919c61803d29c0"} Nov 01 20:58:18.575726 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:18.575565 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b27e66-5db66b9467-z8rv7" Nov 01 20:58:18.575726 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:18.575591 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b27e66-5db66b9467-z8rv7" event={"ID":"4b167c30-a633-471e-a8f4-7f6933c81f0d","Type":"ContainerStarted","Data":"ade80a9aeafab87ad4f3960a2c1d556f636db4d0f7100af78286f707f196d77d"} Nov 01 20:58:18.577284 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:18.577252 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b27e66-6df49bd599-n7wtm" event={"ID":"3ce576d7-7d1e-4b92-ad28-654f4426099c","Type":"ContainerStarted","Data":"955b611b7c742efb3af01bbac2d28300101fc5426f619758ef87fe45a54f45c6"} Nov 01 20:58:18.577430 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:18.577299 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b27e66-6df49bd599-n7wtm" event={"ID":"3ce576d7-7d1e-4b92-ad28-654f4426099c","Type":"ContainerStarted","Data":"0342e5ab57e55af4fd8e7c80c20cddf130726d81d53e3d63e5c06eaa7e2107dc"} Nov 01 20:58:18.577430 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:18.577382 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-b27e66-6df49bd599-n7wtm" Nov 01 20:58:18.691600 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:18.691495 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b27e66-5db66b9467-z8rv7" podStartSLOduration=1.691481452 podStartE2EDuration="1.691481452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:58:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:58:18.689600646 +0000 UTC m=+1649.467240792" watchObservedRunningTime="2026-11-01 20:58:18.691481452 +0000 UTC m=+1649.469121586" Nov 01 20:58:18.833341 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:18.832363 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-b27e66-6df49bd599-n7wtm" podStartSLOduration=1.832338371 podStartE2EDuration="1.832338371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:58:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:58:18.828863583 +0000 UTC m=+1649.606503730" watchObservedRunningTime="2026-11-01 20:58:18.832338371 +0000 UTC m=+1649.609978506" Nov 01 20:58:20.048770 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:20.048729 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-964f4a4a-x2lw6"] Nov 01 20:58:20.098873 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:20.098838 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-964f4a4a-x2lw6"] Nov 01 20:58:21.374498 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:21.374466 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="529a9279-5840-45be-af91-529f4f5df586" path="/var/lib/kubelet/pods/529a9279-5840-45be-af91-529f4f5df586/volumes" Nov 01 20:58:21.606207 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:21.606175 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:21.862899 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:21.862865 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:22.111873 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:22.110920 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:22.252551 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:22.252498 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-df700a-789b6c9547-g9ps7"] Nov 01 20:58:22.259413 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:22.259388 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-df700a-789b6c9547-g9ps7" Nov 01 20:58:22.309539 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:22.309504 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-df700a-789b6c9547-g9ps7"] Nov 01 20:58:22.321227 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:22.321187 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx8vd\" (UniqueName: \"kubernetes.io/projected/5b0aa3a9-007c-45e7-a21b-3fd0f2f8dbf7-kube-api-access-gx8vd\") pod \"payload-pre-processing-e2e-shared-b-w1-df700a-789b6c9547-g9ps7\" (UID: \"5b0aa3a9-007c-45e7-a21b-3fd0f2f8dbf7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-df700a-789b6c9547-g9ps7" Nov 01 20:58:22.321455 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:22.321429 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b0aa3a9-007c-45e7-a21b-3fd0f2f8dbf7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-df700a-789b6c9547-g9ps7\" (UID: \"5b0aa3a9-007c-45e7-a21b-3fd0f2f8dbf7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-df700a-789b6c9547-g9ps7" Nov 01 20:58:22.456137 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:22.456106 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-df700a\"" Nov 01 20:58:22.456858 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:22.456839 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-df700a-dockercfg-tpcjn\"" Nov 01 20:58:22.468044 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:22.460296 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx8vd\" (UniqueName: \"kubernetes.io/projected/5b0aa3a9-007c-45e7-a21b-3fd0f2f8dbf7-kube-api-access-gx8vd\") pod \"payload-pre-processing-e2e-shared-b-w1-df700a-789b6c9547-g9ps7\" (UID: \"5b0aa3a9-007c-45e7-a21b-3fd0f2f8dbf7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-df700a-789b6c9547-g9ps7" Nov 01 20:58:22.468044 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:22.460457 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b0aa3a9-007c-45e7-a21b-3fd0f2f8dbf7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-df700a-789b6c9547-g9ps7\" (UID: \"5b0aa3a9-007c-45e7-a21b-3fd0f2f8dbf7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-df700a-789b6c9547-g9ps7" Nov 01 20:58:22.570389 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:22.557217 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b0aa3a9-007c-45e7-a21b-3fd0f2f8dbf7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-df700a-789b6c9547-g9ps7\" (UID: \"5b0aa3a9-007c-45e7-a21b-3fd0f2f8dbf7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-df700a-789b6c9547-g9ps7" Nov 01 20:58:22.596636 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:22.596236 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-df700a-788d57d789-n6jlx"] Nov 01 20:58:22.606787 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:22.606760 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-df700a-788d57d789-n6jlx" Nov 01 20:58:22.638653 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:22.630456 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-df700a-788d57d789-n6jlx"] Nov 01 20:58:22.659264 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:22.657418 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx8vd\" (UniqueName: \"kubernetes.io/projected/5b0aa3a9-007c-45e7-a21b-3fd0f2f8dbf7-kube-api-access-gx8vd\") pod \"payload-pre-processing-e2e-shared-b-w1-df700a-789b6c9547-g9ps7\" (UID: \"5b0aa3a9-007c-45e7-a21b-3fd0f2f8dbf7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-df700a-789b6c9547-g9ps7" Nov 01 20:58:22.792543 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:22.792509 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxhrv\" (UniqueName: \"kubernetes.io/projected/37709aaf-dd72-454f-9131-5334513b7c2a-kube-api-access-xxhrv\") pod \"payload-processing-e2e-shared-b-w1-df700a-788d57d789-n6jlx\" (UID: \"37709aaf-dd72-454f-9131-5334513b7c2a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-df700a-788d57d789-n6jlx" Nov 01 20:58:22.792729 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:22.792588 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37709aaf-dd72-454f-9131-5334513b7c2a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-df700a-788d57d789-n6jlx\" (UID: \"37709aaf-dd72-454f-9131-5334513b7c2a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-df700a-788d57d789-n6jlx" Nov 01 20:58:22.876172 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:22.876093 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-df700a-789b6c9547-g9ps7" Nov 01 20:58:22.894501 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:22.894471 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37709aaf-dd72-454f-9131-5334513b7c2a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-df700a-788d57d789-n6jlx\" (UID: \"37709aaf-dd72-454f-9131-5334513b7c2a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-df700a-788d57d789-n6jlx" Nov 01 20:58:22.894712 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:22.894698 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxhrv\" (UniqueName: \"kubernetes.io/projected/37709aaf-dd72-454f-9131-5334513b7c2a-kube-api-access-xxhrv\") pod \"payload-processing-e2e-shared-b-w1-df700a-788d57d789-n6jlx\" (UID: \"37709aaf-dd72-454f-9131-5334513b7c2a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-df700a-788d57d789-n6jlx" Nov 01 20:58:22.895399 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:22.895377 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37709aaf-dd72-454f-9131-5334513b7c2a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-df700a-788d57d789-n6jlx\" (UID: \"37709aaf-dd72-454f-9131-5334513b7c2a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-df700a-788d57d789-n6jlx" Nov 01 20:58:22.957781 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:22.957751 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxhrv\" (UniqueName: \"kubernetes.io/projected/37709aaf-dd72-454f-9131-5334513b7c2a-kube-api-access-xxhrv\") pod \"payload-processing-e2e-shared-b-w1-df700a-788d57d789-n6jlx\" (UID: \"37709aaf-dd72-454f-9131-5334513b7c2a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-df700a-788d57d789-n6jlx" Nov 01 20:58:22.981076 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:22.981036 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-df700a-788d57d789-n6jlx" Nov 01 20:58:23.254835 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:58:23.254804 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b0aa3a9_007c_45e7_a21b_3fd0f2f8dbf7.slice/crio-0197eceb99bfacfbfc67b8750bf896090e173d469508fda8930023166fc9d488 WatchSource:0}: Error finding container 0197eceb99bfacfbfc67b8750bf896090e173d469508fda8930023166fc9d488: Status 404 returned error can't find the container with id 0197eceb99bfacfbfc67b8750bf896090e173d469508fda8930023166fc9d488 Nov 01 20:58:23.255290 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:23.255267 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-df700a-789b6c9547-g9ps7"] Nov 01 20:58:23.363983 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:58:23.363951 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37709aaf_dd72_454f_9131_5334513b7c2a.slice/crio-989d0fa577b2ad523a46922156a0b6ab03d7a821de983f3cd09ba36cdae72b2b WatchSource:0}: Error finding container 989d0fa577b2ad523a46922156a0b6ab03d7a821de983f3cd09ba36cdae72b2b: Status 404 returned error can't find the container with id 989d0fa577b2ad523a46922156a0b6ab03d7a821de983f3cd09ba36cdae72b2b Nov 01 20:58:23.373548 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:23.373524 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-df700a-788d57d789-n6jlx"] Nov 01 20:58:23.648952 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:23.648855 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-df700a-789b6c9547-g9ps7" event={"ID":"5b0aa3a9-007c-45e7-a21b-3fd0f2f8dbf7","Type":"ContainerStarted","Data":"1f70884702ed43d3e600d5300d5b56f68cd10d8d87417d5b27a9738ac21ddd99"} Nov 01 20:58:23.648952 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:23.648921 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-df700a-789b6c9547-g9ps7" event={"ID":"5b0aa3a9-007c-45e7-a21b-3fd0f2f8dbf7","Type":"ContainerStarted","Data":"0197eceb99bfacfbfc67b8750bf896090e173d469508fda8930023166fc9d488"} Nov 01 20:58:23.648952 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:23.648951 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-df700a-789b6c9547-g9ps7" Nov 01 20:58:23.650966 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:23.650936 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-df700a-788d57d789-n6jlx" event={"ID":"37709aaf-dd72-454f-9131-5334513b7c2a","Type":"ContainerStarted","Data":"857571fd4d8abedfa9086909cf9ced28262842dd4bc18e92cdb45ded376bb1aa"} Nov 01 20:58:23.651153 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:23.650985 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-df700a-788d57d789-n6jlx" event={"ID":"37709aaf-dd72-454f-9131-5334513b7c2a","Type":"ContainerStarted","Data":"989d0fa577b2ad523a46922156a0b6ab03d7a821de983f3cd09ba36cdae72b2b"} Nov 01 20:58:23.651153 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:23.651041 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-df700a-788d57d789-n6jlx" Nov 01 20:58:23.765440 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:23.765389 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-df700a-789b6c9547-g9ps7" podStartSLOduration=1.7653740359999999 podStartE2EDuration="1.765374036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:58:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:58:23.763539957 +0000 UTC m=+1654.541180092" watchObservedRunningTime="2026-11-01 20:58:23.765374036 +0000 UTC m=+1654.543014170" Nov 01 20:58:23.889034 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:23.880776 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-df700a-788d57d789-n6jlx" podStartSLOduration=1.880754381 podStartE2EDuration="1.880754381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:58:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:58:23.879957444 +0000 UTC m=+1654.657597579" watchObservedRunningTime="2026-11-01 20:58:23.880754381 +0000 UTC m=+1654.658394517" Nov 01 20:58:26.752555 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:26.752517 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:26.957166 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:26.957135 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:29.587475 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:29.587442 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-b27e66-6df49bd599-n7wtm" Nov 01 20:58:29.587901 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:29.587520 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b27e66-5db66b9467-z8rv7" Nov 01 20:58:32.383444 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:32.383329 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:34.664268 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:34.664238 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-df700a-788d57d789-n6jlx" Nov 01 20:58:34.664726 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:34.664662 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-df700a-789b6c9547-g9ps7" Nov 01 20:58:37.515305 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:37.515268 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:37.686459 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:37.686420 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:37.846294 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:37.846207 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:38.142013 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:38.141866 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:38.531436 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:38.531402 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:39.179024 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:39.178857 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk"] Nov 01 20:58:39.228024 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:39.226986 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk" Nov 01 20:58:39.277018 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:39.255831 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk"] Nov 01 20:58:39.305547 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:39.305514 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-bf3fa2-openshift-default-dockercfg-sxsv8\"" Nov 01 20:58:39.317863 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:39.317832 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:39.398670 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:39.360272 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b77b5e66-a42b-4800-a34f-d415b42913d1-istio-envoy\") pod \"e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk\" (UID: \"b77b5e66-a42b-4800-a34f-d415b42913d1\") " pod="openshift-ingress/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk" Nov 01 20:58:39.432343 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:39.432260 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b77b5e66-a42b-4800-a34f-d415b42913d1-istiod-ca-cert\") pod \"e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk\" (UID: \"b77b5e66-a42b-4800-a34f-d415b42913d1\") " pod="openshift-ingress/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk" Nov 01 20:58:39.433549 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:39.433520 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b77b5e66-a42b-4800-a34f-d415b42913d1-credential-socket\") pod \"e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk\" (UID: \"b77b5e66-a42b-4800-a34f-d415b42913d1\") " pod="openshift-ingress/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk" Nov 01 20:58:39.433908 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:39.433883 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf6b5\" (UniqueName: \"kubernetes.io/projected/b77b5e66-a42b-4800-a34f-d415b42913d1-kube-api-access-kf6b5\") pod \"e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk\" (UID: \"b77b5e66-a42b-4800-a34f-d415b42913d1\") " pod="openshift-ingress/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk" Nov 01 20:58:39.434157 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:39.434135 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b77b5e66-a42b-4800-a34f-d415b42913d1-istio-token\") pod \"e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk\" (UID: \"b77b5e66-a42b-4800-a34f-d415b42913d1\") " pod="openshift-ingress/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk" Nov 01 20:58:39.434492 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:39.434471 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b77b5e66-a42b-4800-a34f-d415b42913d1-istio-data\") pod \"e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk\" (UID: \"b77b5e66-a42b-4800-a34f-d415b42913d1\") " pod="openshift-ingress/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk" Nov 01 20:58:39.434715 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:39.434677 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b77b5e66-a42b-4800-a34f-d415b42913d1-istio-podinfo\") pod \"e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk\" (UID: \"b77b5e66-a42b-4800-a34f-d415b42913d1\") " pod="openshift-ingress/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk" Nov 01 20:58:39.434939 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:39.434919 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b77b5e66-a42b-4800-a34f-d415b42913d1-workload-certs\") pod \"e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk\" (UID: \"b77b5e66-a42b-4800-a34f-d415b42913d1\") " pod="openshift-ingress/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk" Nov 01 20:58:39.435226 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:39.435207 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b77b5e66-a42b-4800-a34f-d415b42913d1-workload-socket\") pod \"e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk\" (UID: \"b77b5e66-a42b-4800-a34f-d415b42913d1\") " pod="openshift-ingress/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk" Nov 01 20:58:39.507142 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:39.507098 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:39.554018 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:39.544696 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b77b5e66-a42b-4800-a34f-d415b42913d1-istio-data\") pod \"e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk\" (UID: \"b77b5e66-a42b-4800-a34f-d415b42913d1\") " pod="openshift-ingress/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk" Nov 01 20:58:39.554018 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:39.544760 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b77b5e66-a42b-4800-a34f-d415b42913d1-istio-podinfo\") pod \"e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk\" (UID: \"b77b5e66-a42b-4800-a34f-d415b42913d1\") " pod="openshift-ingress/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk" Nov 01 20:58:39.554018 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:39.544809 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b77b5e66-a42b-4800-a34f-d415b42913d1-workload-certs\") pod \"e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk\" (UID: \"b77b5e66-a42b-4800-a34f-d415b42913d1\") " pod="openshift-ingress/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk" Nov 01 20:58:39.554018 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:39.544930 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b77b5e66-a42b-4800-a34f-d415b42913d1-workload-socket\") pod \"e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk\" (UID: \"b77b5e66-a42b-4800-a34f-d415b42913d1\") " pod="openshift-ingress/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk" Nov 01 20:58:39.554018 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:39.545062 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b77b5e66-a42b-4800-a34f-d415b42913d1-istio-envoy\") pod \"e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk\" (UID: \"b77b5e66-a42b-4800-a34f-d415b42913d1\") " pod="openshift-ingress/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk" Nov 01 20:58:39.554018 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:39.545108 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b77b5e66-a42b-4800-a34f-d415b42913d1-istiod-ca-cert\") pod \"e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk\" (UID: \"b77b5e66-a42b-4800-a34f-d415b42913d1\") " pod="openshift-ingress/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk" Nov 01 20:58:39.554018 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:39.545212 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b77b5e66-a42b-4800-a34f-d415b42913d1-credential-socket\") pod \"e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk\" (UID: \"b77b5e66-a42b-4800-a34f-d415b42913d1\") " pod="openshift-ingress/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk" Nov 01 20:58:39.554018 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:39.545278 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf6b5\" (UniqueName: \"kubernetes.io/projected/b77b5e66-a42b-4800-a34f-d415b42913d1-kube-api-access-kf6b5\") pod \"e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk\" (UID: \"b77b5e66-a42b-4800-a34f-d415b42913d1\") " pod="openshift-ingress/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk" Nov 01 20:58:39.554018 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:39.545311 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b77b5e66-a42b-4800-a34f-d415b42913d1-istio-token\") pod \"e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk\" (UID: \"b77b5e66-a42b-4800-a34f-d415b42913d1\") " pod="openshift-ingress/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk" Nov 01 20:58:39.554018 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:39.551414 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b77b5e66-a42b-4800-a34f-d415b42913d1-workload-certs\") pod \"e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk\" (UID: \"b77b5e66-a42b-4800-a34f-d415b42913d1\") " pod="openshift-ingress/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk" Nov 01 20:58:39.559015 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:39.557418 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b77b5e66-a42b-4800-a34f-d415b42913d1-workload-socket\") pod \"e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk\" (UID: \"b77b5e66-a42b-4800-a34f-d415b42913d1\") " pod="openshift-ingress/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk" Nov 01 20:58:39.559015 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:39.558532 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b77b5e66-a42b-4800-a34f-d415b42913d1-istiod-ca-cert\") pod \"e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk\" (UID: \"b77b5e66-a42b-4800-a34f-d415b42913d1\") " pod="openshift-ingress/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk" Nov 01 20:58:39.559015 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:39.558791 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b77b5e66-a42b-4800-a34f-d415b42913d1-credential-socket\") pod \"e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk\" (UID: \"b77b5e66-a42b-4800-a34f-d415b42913d1\") " pod="openshift-ingress/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk" Nov 01 20:58:39.574019 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:39.569204 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b77b5e66-a42b-4800-a34f-d415b42913d1-istio-data\") pod \"e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk\" (UID: \"b77b5e66-a42b-4800-a34f-d415b42913d1\") " pod="openshift-ingress/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk" Nov 01 20:58:39.587271 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:39.585371 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b77b5e66-a42b-4800-a34f-d415b42913d1-istio-envoy\") pod \"e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk\" (UID: \"b77b5e66-a42b-4800-a34f-d415b42913d1\") " pod="openshift-ingress/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk" Nov 01 20:58:39.590705 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:39.590670 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b77b5e66-a42b-4800-a34f-d415b42913d1-istio-podinfo\") pod \"e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk\" (UID: \"b77b5e66-a42b-4800-a34f-d415b42913d1\") " pod="openshift-ingress/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk" Nov 01 20:58:39.603581 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:39.603554 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b77b5e66-a42b-4800-a34f-d415b42913d1-istio-token\") pod \"e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk\" (UID: \"b77b5e66-a42b-4800-a34f-d415b42913d1\") " pod="openshift-ingress/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk" Nov 01 20:58:39.617894 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:39.617866 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf6b5\" (UniqueName: \"kubernetes.io/projected/b77b5e66-a42b-4800-a34f-d415b42913d1-kube-api-access-kf6b5\") pod \"e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk\" (UID: \"b77b5e66-a42b-4800-a34f-d415b42913d1\") " pod="openshift-ingress/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk" Nov 01 20:58:39.872952 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:39.872917 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk" Nov 01 20:58:40.361582 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:40.361552 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk"] Nov 01 20:58:40.361742 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:58:40.361643 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb77b5e66_a42b_4800_a34f_d415b42913d1.slice/crio-bfb7d3e450cffee87dc6f9068aac1007f012e8b40f25cd79f86ee53750eb0d82 WatchSource:0}: Error finding container bfb7d3e450cffee87dc6f9068aac1007f012e8b40f25cd79f86ee53750eb0d82: Status 404 returned error can't find the container with id bfb7d3e450cffee87dc6f9068aac1007f012e8b40f25cd79f86ee53750eb0d82 Nov 01 20:58:40.364434 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:40.364402 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:58:40.364527 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:40.364463 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:58:40.364527 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:40.364493 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:58:40.913973 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:40.913936 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk" event={"ID":"b77b5e66-a42b-4800-a34f-d415b42913d1","Type":"ContainerStarted","Data":"4afcb3d14e0d6c7c7e19fce05ade68290b13463bc828c8fb42f69d54a03804dd"} Nov 01 20:58:40.914527 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:40.914027 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk" event={"ID":"b77b5e66-a42b-4800-a34f-d415b42913d1","Type":"ContainerStarted","Data":"bfb7d3e450cffee87dc6f9068aac1007f012e8b40f25cd79f86ee53750eb0d82"} Nov 01 20:58:41.084528 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:41.084462 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk" podStartSLOduration=2.08444782 podStartE2EDuration="2.08444782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:58:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:58:41.082358269 +0000 UTC m=+1671.859998407" watchObservedRunningTime="2026-11-01 20:58:41.08444782 +0000 UTC m=+1671.862087960" Nov 01 20:58:41.873943 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:41.873911 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk" Nov 01 20:58:41.880061 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:41.880037 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk" Nov 01 20:58:41.921279 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:41.921249 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk" Nov 01 20:58:41.922139 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:41.922119 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk" Nov 01 20:58:42.470406 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:42.470370 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:42.981335 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:42.981303 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:44.510541 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:44.510509 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:44.711803 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:44.711767 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:44.928636 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:44.928560 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:45.768418 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:58:45.768382 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 20:58:47.779575 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:47.779513 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:48.009449 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:48.009396 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:48.259462 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:48.259427 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:48.348063 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:48.335824 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bf3fa2-5f4bb58487-jrpbx"] Nov 01 20:58:48.611538 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:48.588233 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:48.611538 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:48.606557 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bf3fa2-5f4bb58487-jrpbx"] Nov 01 20:58:48.611538 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:48.607569 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bf3fa2-5f4bb58487-jrpbx" Nov 01 20:58:48.666069 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:48.665988 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-bf3fa2\"" Nov 01 20:58:48.666535 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:48.666514 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-bf3fa2-dockercfg-zml8k\"" Nov 01 20:58:48.786946 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:48.786901 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-bf3fa2-79cb8b8f-w5wfb"] Nov 01 20:58:48.827320 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:48.827283 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-bf3fa2-79cb8b8f-w5wfb"] Nov 01 20:58:48.827687 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:48.827666 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bf3fa2-79cb8b8f-w5wfb" Nov 01 20:58:48.886577 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:48.886490 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:48.921451 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:48.921406 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltsrb\" (UniqueName: \"kubernetes.io/projected/84fa582f-9eaf-4d54-9684-87e33f3f8b81-kube-api-access-ltsrb\") pod \"payload-pre-processing-e2e-shared-b-w3-bf3fa2-5f4bb58487-jrpbx\" (UID: \"84fa582f-9eaf-4d54-9684-87e33f3f8b81\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bf3fa2-5f4bb58487-jrpbx" Nov 01 20:58:48.921651 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:48.921555 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aca0497-55c4-4de2-8a73-5a72ba9054b2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-bf3fa2-79cb8b8f-w5wfb\" (UID: \"1aca0497-55c4-4de2-8a73-5a72ba9054b2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bf3fa2-79cb8b8f-w5wfb" Nov 01 20:58:48.921651 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:48.921606 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84fa582f-9eaf-4d54-9684-87e33f3f8b81-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-bf3fa2-5f4bb58487-jrpbx\" (UID: \"84fa582f-9eaf-4d54-9684-87e33f3f8b81\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bf3fa2-5f4bb58487-jrpbx" Nov 01 20:58:48.921651 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:48.921627 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp5n2\" (UniqueName: \"kubernetes.io/projected/1aca0497-55c4-4de2-8a73-5a72ba9054b2-kube-api-access-kp5n2\") pod \"payload-processing-e2e-shared-b-w3-bf3fa2-79cb8b8f-w5wfb\" (UID: \"1aca0497-55c4-4de2-8a73-5a72ba9054b2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bf3fa2-79cb8b8f-w5wfb" Nov 01 20:58:49.023570 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:49.023530 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aca0497-55c4-4de2-8a73-5a72ba9054b2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-bf3fa2-79cb8b8f-w5wfb\" (UID: \"1aca0497-55c4-4de2-8a73-5a72ba9054b2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bf3fa2-79cb8b8f-w5wfb" Nov 01 20:58:49.023726 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:49.023597 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84fa582f-9eaf-4d54-9684-87e33f3f8b81-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-bf3fa2-5f4bb58487-jrpbx\" (UID: \"84fa582f-9eaf-4d54-9684-87e33f3f8b81\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bf3fa2-5f4bb58487-jrpbx" Nov 01 20:58:49.023726 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:49.023620 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp5n2\" (UniqueName: \"kubernetes.io/projected/1aca0497-55c4-4de2-8a73-5a72ba9054b2-kube-api-access-kp5n2\") pod \"payload-processing-e2e-shared-b-w3-bf3fa2-79cb8b8f-w5wfb\" (UID: \"1aca0497-55c4-4de2-8a73-5a72ba9054b2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bf3fa2-79cb8b8f-w5wfb" Nov 01 20:58:49.023838 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:49.023818 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltsrb\" (UniqueName: \"kubernetes.io/projected/84fa582f-9eaf-4d54-9684-87e33f3f8b81-kube-api-access-ltsrb\") pod \"payload-pre-processing-e2e-shared-b-w3-bf3fa2-5f4bb58487-jrpbx\" (UID: \"84fa582f-9eaf-4d54-9684-87e33f3f8b81\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bf3fa2-5f4bb58487-jrpbx" Nov 01 20:58:49.024327 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:49.024299 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aca0497-55c4-4de2-8a73-5a72ba9054b2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-bf3fa2-79cb8b8f-w5wfb\" (UID: \"1aca0497-55c4-4de2-8a73-5a72ba9054b2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bf3fa2-79cb8b8f-w5wfb" Nov 01 20:58:49.024327 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:49.024309 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84fa582f-9eaf-4d54-9684-87e33f3f8b81-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-bf3fa2-5f4bb58487-jrpbx\" (UID: \"84fa582f-9eaf-4d54-9684-87e33f3f8b81\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bf3fa2-5f4bb58487-jrpbx" Nov 01 20:58:49.082584 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:49.082554 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp5n2\" (UniqueName: \"kubernetes.io/projected/1aca0497-55c4-4de2-8a73-5a72ba9054b2-kube-api-access-kp5n2\") pod \"payload-processing-e2e-shared-b-w3-bf3fa2-79cb8b8f-w5wfb\" (UID: \"1aca0497-55c4-4de2-8a73-5a72ba9054b2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bf3fa2-79cb8b8f-w5wfb" Nov 01 20:58:49.082751 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:49.082655 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltsrb\" (UniqueName: \"kubernetes.io/projected/84fa582f-9eaf-4d54-9684-87e33f3f8b81-kube-api-access-ltsrb\") pod \"payload-pre-processing-e2e-shared-b-w3-bf3fa2-5f4bb58487-jrpbx\" (UID: \"84fa582f-9eaf-4d54-9684-87e33f3f8b81\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bf3fa2-5f4bb58487-jrpbx" Nov 01 20:58:49.154655 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:49.154566 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bf3fa2-79cb8b8f-w5wfb" Nov 01 20:58:49.349366 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:49.349327 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bf3fa2-5f4bb58487-jrpbx" Nov 01 20:58:49.537196 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:58:49.537154 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aca0497_55c4_4de2_8a73_5a72ba9054b2.slice/crio-793dec16ac7d3d6b1c29326ecbb430847ef82326c13cb6b051c5b8e0ebd74b3b WatchSource:0}: Error finding container 793dec16ac7d3d6b1c29326ecbb430847ef82326c13cb6b051c5b8e0ebd74b3b: Status 404 returned error can't find the container with id 793dec16ac7d3d6b1c29326ecbb430847ef82326c13cb6b051c5b8e0ebd74b3b Nov 01 20:58:49.538046 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:49.538015 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-bf3fa2-79cb8b8f-w5wfb"] Nov 01 20:58:49.731514 ip-10-0-141-35 kubenswrapper[4484]: W1101 20:58:49.731474 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84fa582f_9eaf_4d54_9684_87e33f3f8b81.slice/crio-e8dc7c77b30cfebc282b5e3c06af00d8abbe30d6b846134946d4f34850b59bab WatchSource:0}: Error finding container e8dc7c77b30cfebc282b5e3c06af00d8abbe30d6b846134946d4f34850b59bab: Status 404 returned error can't find the container with id e8dc7c77b30cfebc282b5e3c06af00d8abbe30d6b846134946d4f34850b59bab Nov 01 20:58:49.731761 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:49.731737 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bf3fa2-5f4bb58487-jrpbx"] Nov 01 20:58:50.046043 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:50.045979 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bf3fa2-5f4bb58487-jrpbx" event={"ID":"84fa582f-9eaf-4d54-9684-87e33f3f8b81","Type":"ContainerStarted","Data":"c3d0df010b25403eda246c44b289954a35c4bd0e39d76fd94b11d915951d7b91"} Nov 01 20:58:50.046510 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:50.046078 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bf3fa2-5f4bb58487-jrpbx" Nov 01 20:58:50.046510 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:50.046102 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bf3fa2-5f4bb58487-jrpbx" event={"ID":"84fa582f-9eaf-4d54-9684-87e33f3f8b81","Type":"ContainerStarted","Data":"e8dc7c77b30cfebc282b5e3c06af00d8abbe30d6b846134946d4f34850b59bab"} Nov 01 20:58:50.047853 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:50.047825 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bf3fa2-79cb8b8f-w5wfb" event={"ID":"1aca0497-55c4-4de2-8a73-5a72ba9054b2","Type":"ContainerStarted","Data":"878105ee2f079c980b4969f87d59112adea8e0795fde80d710b749e15ad6af99"} Nov 01 20:58:50.048032 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:50.047873 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bf3fa2-79cb8b8f-w5wfb" event={"ID":"1aca0497-55c4-4de2-8a73-5a72ba9054b2","Type":"ContainerStarted","Data":"793dec16ac7d3d6b1c29326ecbb430847ef82326c13cb6b051c5b8e0ebd74b3b"} Nov 01 20:58:50.048032 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:50.047900 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bf3fa2-79cb8b8f-w5wfb" Nov 01 20:58:50.165320 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:50.165269 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bf3fa2-5f4bb58487-jrpbx" podStartSLOduration=2.165256142 podStartE2EDuration="2.165256142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:58:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:58:50.160785437 +0000 UTC m=+1680.938425573" watchObservedRunningTime="2026-11-01 20:58:50.165256142 +0000 UTC m=+1680.942896314" Nov 01 20:58:50.331070 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:50.326737 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bf3fa2-79cb8b8f-w5wfb" podStartSLOduration=2.326713188 podStartE2EDuration="2.326713188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:58:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:58:50.308809315 +0000 UTC m=+1681.086449470" watchObservedRunningTime="2026-11-01 20:58:50.326713188 +0000 UTC m=+1681.104353324" Nov 01 20:58:53.253888 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:53.253850 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:57.984476 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:57.984442 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:59.631641 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:59.631610 4484 scope.go:122] "RemoveContainer" containerID="8fe384a79f7056515d0875e05970efc6cbd12689ab7976abdb12ea42a8b62561" Nov 01 20:58:59.643594 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:59.643570 4484 scope.go:122] "RemoveContainer" containerID="ce7d2b53ac68525c99d7d72c5b10bf655d9c0a765449a01a483aee28ba2a97c7" Nov 01 20:58:59.654732 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:58:59.654708 4484 scope.go:122] "RemoveContainer" containerID="e79e50a51d320f3f3bb0913849f41f19841df17da62834a7bc24cf0dfc9ff40d" Nov 01 20:59:01.061982 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:01.061953 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bf3fa2-5f4bb58487-jrpbx" Nov 01 20:59:01.062434 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:01.062048 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bf3fa2-79cb8b8f-w5wfb" Nov 01 20:59:02.050196 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:02.050151 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c232c5f-b4r48"] Nov 01 20:59:02.098006 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:02.097953 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c232c5f-b4r48"] Nov 01 20:59:02.750180 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:02.750128 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:59:03.374701 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:03.374655 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="743ddb5c-ae93-4e30-bd06-f44401ea0d68" path="/var/lib/kubelet/pods/743ddb5c-ae93-4e30-bd06-f44401ea0d68/volumes" Nov 01 20:59:07.513750 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:07.513592 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:59:12.252315 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:12.252065 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:59:15.768887 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:59:15.768846 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 20:59:17.103025 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:17.097777 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:59:20.920832 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:20.920795 4484 patch_prober.go:29] interesting pod/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Nov 01 20:59:20.921285 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:20.920866 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6" podUID="5caf509e-0540-4ff6-a544-5b9e83bda042" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Nov 01 20:59:21.432527 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:21.432438 4484 generic.go:356] "Generic (PLEG): container finished" podID="5caf509e-0540-4ff6-a544-5b9e83bda042" containerID="b877616fcd50c65da91923af62eadbf103481a08c0b2aa30cab537aaf0efc123" exitCode=0 Nov 01 20:59:21.432527 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:21.432501 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6" event={"ID":"5caf509e-0540-4ff6-a544-5b9e83bda042","Type":"ContainerDied","Data":"b877616fcd50c65da91923af62eadbf103481a08c0b2aa30cab537aaf0efc123"} Nov 01 20:59:21.433473 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:21.432933 4484 scope.go:122] "RemoveContainer" containerID="b877616fcd50c65da91923af62eadbf103481a08c0b2aa30cab537aaf0efc123" Nov 01 20:59:21.434709 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:21.434679 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:59:21.434783 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:21.434750 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:59:21.434819 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:21.434786 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:59:21.799208 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:21.799168 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:59:21.944360 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:21.939074 4484 patch_prober.go:29] interesting pod/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Nov 01 20:59:21.944360 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:21.939135 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m" podUID="7ecec0b0-7e22-4803-891c-0a75173445ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Nov 01 20:59:21.966892 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:21.966854 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:59:22.121593 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:22.121495 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:59:22.443294 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:22.443207 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6" event={"ID":"5caf509e-0540-4ff6-a544-5b9e83bda042","Type":"ContainerStarted","Data":"9dd7eb3f4bdea480e82cdd337391ab8250f45239e5f09eca70544e8c27650b0e"} Nov 01 20:59:22.444874 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:22.444850 4484 generic.go:356] "Generic (PLEG): container finished" podID="7ecec0b0-7e22-4803-891c-0a75173445ad" containerID="205e260be2d92a416298051fc1b803f8de077fbe104b1b53c437f032521d2608" exitCode=0 Nov 01 20:59:22.445018 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:22.444924 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m" event={"ID":"7ecec0b0-7e22-4803-891c-0a75173445ad","Type":"ContainerDied","Data":"205e260be2d92a416298051fc1b803f8de077fbe104b1b53c437f032521d2608"} Nov 01 20:59:22.445356 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:22.445341 4484 scope.go:122] "RemoveContainer" containerID="205e260be2d92a416298051fc1b803f8de077fbe104b1b53c437f032521d2608" Nov 01 20:59:22.447030 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:22.446975 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:59:22.447116 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:22.447054 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:59:22.447116 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:22.447093 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:59:23.171112 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:23.166206 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6" Nov 01 20:59:23.171112 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:23.170126 4484 patch_prober.go:29] interesting pod/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Nov 01 20:59:23.171112 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:23.170189 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6" podUID="5caf509e-0540-4ff6-a544-5b9e83bda042" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Nov 01 20:59:23.463890 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:23.463789 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m" event={"ID":"7ecec0b0-7e22-4803-891c-0a75173445ad","Type":"ContainerStarted","Data":"2f7560be7d7ec4d112867c55ef155c749075b51ed446557b8d4fba8334ff3e6d"} Nov 01 20:59:24.166976 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:24.166938 4484 patch_prober.go:29] interesting pod/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Nov 01 20:59:24.167189 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:24.167030 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6" podUID="5caf509e-0540-4ff6-a544-5b9e83bda042" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Nov 01 20:59:24.453495 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:24.453402 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m" Nov 01 20:59:24.454188 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:24.454158 4484 patch_prober.go:29] interesting pod/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Nov 01 20:59:24.454386 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:24.454360 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m" podUID="7ecec0b0-7e22-4803-891c-0a75173445ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Nov 01 20:59:25.175418 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:25.175380 4484 patch_prober.go:29] interesting pod/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Nov 01 20:59:25.175723 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:25.175695 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6" podUID="5caf509e-0540-4ff6-a544-5b9e83bda042" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Nov 01 20:59:25.454046 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:25.453897 4484 patch_prober.go:29] interesting pod/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Nov 01 20:59:25.454682 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:25.454637 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m" podUID="7ecec0b0-7e22-4803-891c-0a75173445ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Nov 01 20:59:26.166110 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:26.166077 4484 patch_prober.go:29] interesting pod/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Nov 01 20:59:26.166330 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:26.166149 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6" podUID="5caf509e-0540-4ff6-a544-5b9e83bda042" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Nov 01 20:59:26.454079 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:26.453377 4484 patch_prober.go:29] interesting pod/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Nov 01 20:59:26.454079 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:26.453455 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m" podUID="7ecec0b0-7e22-4803-891c-0a75173445ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Nov 01 20:59:26.643021 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:26.642839 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:59:26.818044 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:26.817988 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:59:27.123104 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:27.122941 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:59:27.166241 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:27.166208 4484 patch_prober.go:29] interesting pod/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Nov 01 20:59:27.166443 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:27.166261 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6" podUID="5caf509e-0540-4ff6-a544-5b9e83bda042" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Nov 01 20:59:27.453529 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:27.453434 4484 patch_prober.go:29] interesting pod/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Nov 01 20:59:27.453803 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:27.453755 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m" podUID="7ecec0b0-7e22-4803-891c-0a75173445ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Nov 01 20:59:27.589116 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:27.589073 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:59:27.795492 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:27.795438 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:59:28.166886 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:28.166803 4484 patch_prober.go:29] interesting pod/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Nov 01 20:59:28.166886 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:28.166857 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6" podUID="5caf509e-0540-4ff6-a544-5b9e83bda042" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Nov 01 20:59:28.453392 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:28.453305 4484 patch_prober.go:29] interesting pod/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Nov 01 20:59:28.453392 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:28.453358 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m" podUID="7ecec0b0-7e22-4803-891c-0a75173445ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Nov 01 20:59:29.172528 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:29.172497 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6" Nov 01 20:59:29.173275 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:29.173250 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6" Nov 01 20:59:29.174485 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:29.174464 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6" Nov 01 20:59:29.463375 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:29.463299 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m" Nov 01 20:59:29.463793 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:29.463764 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m" Nov 01 20:59:29.465031 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:29.464944 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m" Nov 01 20:59:31.697353 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:31.697311 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:59:36.581631 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:36.581550 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:59:38.420320 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:38.420281 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:59:38.711235 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:38.711104 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:59:41.341186 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:41.330074 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:59:42.794665 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:42.794628 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:59:45.769071 ip-10-0-141-35 kubenswrapper[4484]: E1101 20:59:45.769032 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 20:59:46.039394 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:46.039299 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:59:50.842782 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:50.842745 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:59:55.678057 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:55.676949 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:59:55.837572 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:55.837513 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 20:59:59.257823 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:59.256203 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv"] Nov 01 20:59:59.257823 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:59.256490 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv" podUID="3ae0b2c8-270b-4a3f-bb16-523874681ade" containerName="istio-proxy" containerID="cri-o://acbed28624ebe42544acb96d055050fed39395495b39d480ce98a0590c69f879" gracePeriod=30 Nov 01 20:59:59.529376 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:59.529173 4484 patch_prober.go:29] interesting pod/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:59:59.529376 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:59.529264 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822" podUID="4048751a-034d-4ab2-937f-6161505e4166" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:59:59.542132 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:59.542096 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf"] Nov 01 20:59:59.542345 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:59.542326 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf" podUID="ef2f968c-433d-49f8-8fc4-7c8551d647fb" containerName="istio-proxy" containerID="cri-o://8eb9816d4eca1ec89f3adadcb08ce4caaa9cc08431c2dbe91e021610b5ae2efb" gracePeriod=30 Nov 01 20:59:59.771946 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:59.771907 4484 scope.go:122] "RemoveContainer" containerID="4de535e1e7b350ec7714fa65a8d9400089829e5350b3a0b91c82dd2462f0b1c7" Nov 01 20:59:59.789872 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:59.789836 4484 scope.go:122] "RemoveContainer" containerID="0b4eb635a8d3d9ca5f23d94d184f613aecef032c401f7e3fb2741f6da7b56af0" Nov 01 20:59:59.816855 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:59.816823 4484 scope.go:122] "RemoveContainer" containerID="83954ada3f526b92f36d0356b7542c9b7e69f7ea4c19c317a98ef0615dc0f662" Nov 01 20:59:59.870347 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:59.870318 4484 scope.go:122] "RemoveContainer" containerID="b690fad52de89a4fbdbec6d5759ff6399cfa995b855663743c7f7791f67a7907" Nov 01 20:59:59.911346 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:59.911322 4484 scope.go:122] "RemoveContainer" containerID="a6f957f8225f7bb299168a28f2fec5525d8ed9e444c8b3ba5dc3fe9c36e692d4" Nov 01 20:59:59.911449 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:59.911347 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8"] Nov 01 20:59:59.911568 ip-10-0-141-35 kubenswrapper[4484]: I1101 20:59:59.911543 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8" podUID="b373faf1-ffb2-4a45-b433-0669635a397c" containerName="istio-proxy" containerID="cri-o://b6541f7eefe7ba24ae375b42001c3d8d5be218cac65c3c82b0736e0ce422fffe" gracePeriod=30 Nov 01 21:00:00.034959 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.034917 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:00.164191 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.164110 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd038b-29892780-vltsq"] Nov 01 21:00:00.190974 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.190935 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892780-xxrwz"] Nov 01 21:00:00.191181 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.191106 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd038b-29892780-vltsq" Nov 01 21:00:00.240532 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.238258 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b19e6e75-0ef8-440d-8617-e5c71bfde0d2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-dd038b-29892780-vltsq\" (UID: \"b19e6e75-0ef8-440d-8617-e5c71bfde0d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd038b-29892780-vltsq" Nov 01 21:00:00.240532 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.238449 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8vtt\" (UniqueName: \"kubernetes.io/projected/b19e6e75-0ef8-440d-8617-e5c71bfde0d2-kube-api-access-l8vtt\") pod \"maas-api-key-cleanup-e2e-worker-w6-dd038b-29892780-vltsq\" (UID: \"b19e6e75-0ef8-440d-8617-e5c71bfde0d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd038b-29892780-vltsq" Nov 01 21:00:00.245615 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.245298 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892780-7pl4v"] Nov 01 21:00:00.251126 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.246721 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892780-xxrwz" Nov 01 21:00:00.251126 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.247431 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:00:00.251126 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.247438 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:00:00.251126 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.247538 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:00:00.278258 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.278228 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892780-jmsct"] Nov 01 21:00:00.281848 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.281818 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892780-7pl4v" Nov 01 21:00:00.344121 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.343650 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4578009-b3ff-4ce6-825a-8f92c1d3d72d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892780-xxrwz\" (UID: \"b4578009-b3ff-4ce6-825a-8f92c1d3d72d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892780-xxrwz" Nov 01 21:00:00.344121 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.343873 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b19e6e75-0ef8-440d-8617-e5c71bfde0d2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-dd038b-29892780-vltsq\" (UID: \"b19e6e75-0ef8-440d-8617-e5c71bfde0d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd038b-29892780-vltsq" Nov 01 21:00:00.344121 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.343962 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da6f250e-6516-4a04-a37a-f30c02ec682b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892780-7pl4v\" (UID: \"da6f250e-6516-4a04-a37a-f30c02ec682b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892780-7pl4v" Nov 01 21:00:00.344121 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.344060 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hdnk\" (UniqueName: \"kubernetes.io/projected/da6f250e-6516-4a04-a37a-f30c02ec682b-kube-api-access-8hdnk\") pod \"maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892780-7pl4v\" (UID: \"da6f250e-6516-4a04-a37a-f30c02ec682b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892780-7pl4v" Nov 01 21:00:00.344457 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.344167 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8vtt\" (UniqueName: \"kubernetes.io/projected/b19e6e75-0ef8-440d-8617-e5c71bfde0d2-kube-api-access-l8vtt\") pod \"maas-api-key-cleanup-e2e-worker-w6-dd038b-29892780-vltsq\" (UID: \"b19e6e75-0ef8-440d-8617-e5c71bfde0d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd038b-29892780-vltsq" Nov 01 21:00:00.344457 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.344218 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l59jl\" (UniqueName: \"kubernetes.io/projected/b4578009-b3ff-4ce6-825a-8f92c1d3d72d-kube-api-access-l59jl\") pod \"maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892780-xxrwz\" (UID: \"b4578009-b3ff-4ce6-825a-8f92c1d3d72d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892780-xxrwz" Nov 01 21:00:00.350066 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.347034 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892780-vdj2s"] Nov 01 21:00:00.350066 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.348371 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b19e6e75-0ef8-440d-8617-e5c71bfde0d2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-dd038b-29892780-vltsq\" (UID: \"b19e6e75-0ef8-440d-8617-e5c71bfde0d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd038b-29892780-vltsq" Nov 01 21:00:00.350066 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.348581 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892780-jmsct" Nov 01 21:00:00.394283 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.394247 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892780-96d26"] Nov 01 21:00:00.399097 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.395278 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892780-vdj2s" Nov 01 21:00:00.406602 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.406571 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8vtt\" (UniqueName: \"kubernetes.io/projected/b19e6e75-0ef8-440d-8617-e5c71bfde0d2-kube-api-access-l8vtt\") pod \"maas-api-key-cleanup-e2e-worker-w6-dd038b-29892780-vltsq\" (UID: \"b19e6e75-0ef8-440d-8617-e5c71bfde0d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd038b-29892780-vltsq" Nov 01 21:00:00.445494 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.445463 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da6f250e-6516-4a04-a37a-f30c02ec682b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892780-7pl4v\" (UID: \"da6f250e-6516-4a04-a37a-f30c02ec682b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892780-7pl4v" Nov 01 21:00:00.445681 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.445527 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hdnk\" (UniqueName: \"kubernetes.io/projected/da6f250e-6516-4a04-a37a-f30c02ec682b-kube-api-access-8hdnk\") pod \"maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892780-7pl4v\" (UID: \"da6f250e-6516-4a04-a37a-f30c02ec682b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892780-7pl4v" Nov 01 21:00:00.445681 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.445572 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5aa9680-52ae-4173-875b-1097a5963fc0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-944263-29892780-jmsct\" (UID: \"e5aa9680-52ae-4173-875b-1097a5963fc0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892780-jmsct" Nov 01 21:00:00.445681 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.445611 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqbqr\" (UniqueName: \"kubernetes.io/projected/ce36bd6e-c550-4cd7-8f34-97b71968e4a2-kube-api-access-nqbqr\") pod \"maas-api-key-cleanup-e2e-worker-w2-cd248f-29892780-vdj2s\" (UID: \"ce36bd6e-c550-4cd7-8f34-97b71968e4a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892780-vdj2s" Nov 01 21:00:00.445867 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.445790 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tgtb\" (UniqueName: \"kubernetes.io/projected/e5aa9680-52ae-4173-875b-1097a5963fc0-kube-api-access-8tgtb\") pod \"maas-api-key-cleanup-e2e-worker-w3-944263-29892780-jmsct\" (UID: \"e5aa9680-52ae-4173-875b-1097a5963fc0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892780-jmsct" Nov 01 21:00:00.445867 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.445837 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce36bd6e-c550-4cd7-8f34-97b71968e4a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-cd248f-29892780-vdj2s\" (UID: \"ce36bd6e-c550-4cd7-8f34-97b71968e4a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892780-vdj2s" Nov 01 21:00:00.446062 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.445923 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l59jl\" (UniqueName: \"kubernetes.io/projected/b4578009-b3ff-4ce6-825a-8f92c1d3d72d-kube-api-access-l59jl\") pod \"maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892780-xxrwz\" (UID: \"b4578009-b3ff-4ce6-825a-8f92c1d3d72d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892780-xxrwz" Nov 01 21:00:00.446062 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.446054 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4578009-b3ff-4ce6-825a-8f92c1d3d72d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892780-xxrwz\" (UID: \"b4578009-b3ff-4ce6-825a-8f92c1d3d72d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892780-xxrwz" Nov 01 21:00:00.446303 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.446282 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da6f250e-6516-4a04-a37a-f30c02ec682b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892780-7pl4v\" (UID: \"da6f250e-6516-4a04-a37a-f30c02ec682b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892780-7pl4v" Nov 01 21:00:00.446715 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.446696 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4578009-b3ff-4ce6-825a-8f92c1d3d72d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892780-xxrwz\" (UID: \"b4578009-b3ff-4ce6-825a-8f92c1d3d72d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892780-xxrwz" Nov 01 21:00:00.464693 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.464660 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b27e66-29892780-hvgx4"] Nov 01 21:00:00.464865 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.464843 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892780-96d26" Nov 01 21:00:00.503374 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.503338 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hdnk\" (UniqueName: \"kubernetes.io/projected/da6f250e-6516-4a04-a37a-f30c02ec682b-kube-api-access-8hdnk\") pod \"maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892780-7pl4v\" (UID: \"da6f250e-6516-4a04-a37a-f30c02ec682b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892780-7pl4v" Nov 01 21:00:00.503572 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.503474 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l59jl\" (UniqueName: \"kubernetes.io/projected/b4578009-b3ff-4ce6-825a-8f92c1d3d72d-kube-api-access-l59jl\") pod \"maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892780-xxrwz\" (UID: \"b4578009-b3ff-4ce6-825a-8f92c1d3d72d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892780-xxrwz" Nov 01 21:00:00.513412 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.513379 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892780-p6gfx"] Nov 01 21:00:00.513537 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.513517 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b27e66-29892780-hvgx4" Nov 01 21:00:00.546581 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.546554 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd038b-29892780-vltsq" Nov 01 21:00:00.547214 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.547187 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfd728c4-7f6b-4197-a147-1f340427f414-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892780-96d26\" (UID: \"cfd728c4-7f6b-4197-a147-1f340427f414\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892780-96d26" Nov 01 21:00:00.547364 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.547344 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr2mg\" (UniqueName: \"kubernetes.io/projected/7369891a-d9d2-42f1-8165-f817e9c00984-kube-api-access-cr2mg\") pod \"maas-api-key-cleanup-e2e-api-a-b27e66-29892780-hvgx4\" (UID: \"7369891a-d9d2-42f1-8165-f817e9c00984\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b27e66-29892780-hvgx4" Nov 01 21:00:00.547433 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.547379 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5aa9680-52ae-4173-875b-1097a5963fc0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-944263-29892780-jmsct\" (UID: \"e5aa9680-52ae-4173-875b-1097a5963fc0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892780-jmsct" Nov 01 21:00:00.547433 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.547414 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7pnm\" (UniqueName: \"kubernetes.io/projected/cfd728c4-7f6b-4197-a147-1f340427f414-kube-api-access-l7pnm\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892780-96d26\" (UID: \"cfd728c4-7f6b-4197-a147-1f340427f414\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892780-96d26" Nov 01 21:00:00.547570 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.547542 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqbqr\" (UniqueName: \"kubernetes.io/projected/ce36bd6e-c550-4cd7-8f34-97b71968e4a2-kube-api-access-nqbqr\") pod \"maas-api-key-cleanup-e2e-worker-w2-cd248f-29892780-vdj2s\" (UID: \"ce36bd6e-c550-4cd7-8f34-97b71968e4a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892780-vdj2s" Nov 01 21:00:00.547653 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.547636 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tgtb\" (UniqueName: \"kubernetes.io/projected/e5aa9680-52ae-4173-875b-1097a5963fc0-kube-api-access-8tgtb\") pod \"maas-api-key-cleanup-e2e-worker-w3-944263-29892780-jmsct\" (UID: \"e5aa9680-52ae-4173-875b-1097a5963fc0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892780-jmsct" Nov 01 21:00:00.547711 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.547676 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce36bd6e-c550-4cd7-8f34-97b71968e4a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-cd248f-29892780-vdj2s\" (UID: \"ce36bd6e-c550-4cd7-8f34-97b71968e4a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892780-vdj2s" Nov 01 21:00:00.547785 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.547768 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7369891a-d9d2-42f1-8165-f817e9c00984-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-b27e66-29892780-hvgx4\" (UID: \"7369891a-d9d2-42f1-8165-f817e9c00984\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b27e66-29892780-hvgx4" Nov 01 21:00:00.548111 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.548091 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5aa9680-52ae-4173-875b-1097a5963fc0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-944263-29892780-jmsct\" (UID: \"e5aa9680-52ae-4173-875b-1097a5963fc0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892780-jmsct" Nov 01 21:00:00.548334 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.548315 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce36bd6e-c550-4cd7-8f34-97b71968e4a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-cd248f-29892780-vdj2s\" (UID: \"ce36bd6e-c550-4cd7-8f34-97b71968e4a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892780-vdj2s" Nov 01 21:00:00.582495 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.582461 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892780-q4ns9"] Nov 01 21:00:00.582665 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.582585 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892780-p6gfx" Nov 01 21:00:00.604918 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.604888 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqbqr\" (UniqueName: \"kubernetes.io/projected/ce36bd6e-c550-4cd7-8f34-97b71968e4a2-kube-api-access-nqbqr\") pod \"maas-api-key-cleanup-e2e-worker-w2-cd248f-29892780-vdj2s\" (UID: \"ce36bd6e-c550-4cd7-8f34-97b71968e4a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892780-vdj2s" Nov 01 21:00:00.604918 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.604915 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tgtb\" (UniqueName: \"kubernetes.io/projected/e5aa9680-52ae-4173-875b-1097a5963fc0-kube-api-access-8tgtb\") pod \"maas-api-key-cleanup-e2e-worker-w3-944263-29892780-jmsct\" (UID: \"e5aa9680-52ae-4173-875b-1097a5963fc0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892780-jmsct" Nov 01 21:00:00.606681 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.606658 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892780-vdj2s" Nov 01 21:00:00.630291 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.630261 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892780-9tm2f"] Nov 01 21:00:00.630496 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.630382 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892780-q4ns9" Nov 01 21:00:00.649446 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.649413 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/474d07b8-cbb2-4beb-a17f-fe445d0c9ad4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892780-q4ns9\" (UID: \"474d07b8-cbb2-4beb-a17f-fe445d0c9ad4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892780-q4ns9" Nov 01 21:00:00.649626 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.649519 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr2mg\" (UniqueName: \"kubernetes.io/projected/7369891a-d9d2-42f1-8165-f817e9c00984-kube-api-access-cr2mg\") pod \"maas-api-key-cleanup-e2e-api-a-b27e66-29892780-hvgx4\" (UID: \"7369891a-d9d2-42f1-8165-f817e9c00984\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b27e66-29892780-hvgx4" Nov 01 21:00:00.649626 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.649565 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38c14ac7-4673-46b7-a721-652a972f3ba5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892780-p6gfx\" (UID: \"38c14ac7-4673-46b7-a721-652a972f3ba5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892780-p6gfx" Nov 01 21:00:00.649626 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.649624 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7pnm\" (UniqueName: \"kubernetes.io/projected/cfd728c4-7f6b-4197-a147-1f340427f414-kube-api-access-l7pnm\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892780-96d26\" (UID: \"cfd728c4-7f6b-4197-a147-1f340427f414\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892780-96d26" Nov 01 21:00:00.649818 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.649772 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrhn2\" (UniqueName: \"kubernetes.io/projected/38c14ac7-4673-46b7-a721-652a972f3ba5-kube-api-access-nrhn2\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892780-p6gfx\" (UID: \"38c14ac7-4673-46b7-a721-652a972f3ba5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892780-p6gfx" Nov 01 21:00:00.649898 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.649879 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7369891a-d9d2-42f1-8165-f817e9c00984-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-b27e66-29892780-hvgx4\" (UID: \"7369891a-d9d2-42f1-8165-f817e9c00984\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b27e66-29892780-hvgx4" Nov 01 21:00:00.650142 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.650123 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqdzk\" (UniqueName: \"kubernetes.io/projected/474d07b8-cbb2-4beb-a17f-fe445d0c9ad4-kube-api-access-pqdzk\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892780-q4ns9\" (UID: \"474d07b8-cbb2-4beb-a17f-fe445d0c9ad4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892780-q4ns9" Nov 01 21:00:00.650205 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.650190 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfd728c4-7f6b-4197-a147-1f340427f414-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892780-96d26\" (UID: \"cfd728c4-7f6b-4197-a147-1f340427f414\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892780-96d26" Nov 01 21:00:00.651218 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.651066 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7369891a-d9d2-42f1-8165-f817e9c00984-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-b27e66-29892780-hvgx4\" (UID: \"7369891a-d9d2-42f1-8165-f817e9c00984\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b27e66-29892780-hvgx4" Nov 01 21:00:00.651218 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.651066 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfd728c4-7f6b-4197-a147-1f340427f414-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892780-96d26\" (UID: \"cfd728c4-7f6b-4197-a147-1f340427f414\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892780-96d26" Nov 01 21:00:00.671624 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.671594 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892780-xxrwz" Nov 01 21:00:00.707420 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.707387 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7pnm\" (UniqueName: \"kubernetes.io/projected/cfd728c4-7f6b-4197-a147-1f340427f414-kube-api-access-l7pnm\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892780-96d26\" (UID: \"cfd728c4-7f6b-4197-a147-1f340427f414\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892780-96d26" Nov 01 21:00:00.707885 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.707831 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr2mg\" (UniqueName: \"kubernetes.io/projected/7369891a-d9d2-42f1-8165-f817e9c00984-kube-api-access-cr2mg\") pod \"maas-api-key-cleanup-e2e-api-a-b27e66-29892780-hvgx4\" (UID: \"7369891a-d9d2-42f1-8165-f817e9c00984\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b27e66-29892780-hvgx4" Nov 01 21:00:00.718909 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.718865 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892780-6xlxn"] Nov 01 21:00:00.719120 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.718955 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892780-9tm2f" Nov 01 21:00:00.746670 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.742846 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892780-7pl4v" Nov 01 21:00:00.758231 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.753782 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqdzk\" (UniqueName: \"kubernetes.io/projected/474d07b8-cbb2-4beb-a17f-fe445d0c9ad4-kube-api-access-pqdzk\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892780-q4ns9\" (UID: \"474d07b8-cbb2-4beb-a17f-fe445d0c9ad4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892780-q4ns9" Nov 01 21:00:00.758231 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.753975 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/474d07b8-cbb2-4beb-a17f-fe445d0c9ad4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892780-q4ns9\" (UID: \"474d07b8-cbb2-4beb-a17f-fe445d0c9ad4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892780-q4ns9" Nov 01 21:00:00.758231 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.754055 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38c14ac7-4673-46b7-a721-652a972f3ba5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892780-p6gfx\" (UID: \"38c14ac7-4673-46b7-a721-652a972f3ba5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892780-p6gfx" Nov 01 21:00:00.758231 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.754171 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrhn2\" (UniqueName: \"kubernetes.io/projected/38c14ac7-4673-46b7-a721-652a972f3ba5-kube-api-access-nrhn2\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892780-p6gfx\" (UID: \"38c14ac7-4673-46b7-a721-652a972f3ba5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892780-p6gfx" Nov 01 21:00:00.758231 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.754253 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0f5aabc-58f2-4166-af2d-39aa67592cb1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892780-9tm2f\" (UID: \"d0f5aabc-58f2-4166-af2d-39aa67592cb1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892780-9tm2f" Nov 01 21:00:00.758231 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.754309 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w65xg\" (UniqueName: \"kubernetes.io/projected/d0f5aabc-58f2-4166-af2d-39aa67592cb1-kube-api-access-w65xg\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892780-9tm2f\" (UID: \"d0f5aabc-58f2-4166-af2d-39aa67592cb1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892780-9tm2f" Nov 01 21:00:00.758231 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.755683 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/474d07b8-cbb2-4beb-a17f-fe445d0c9ad4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892780-q4ns9\" (UID: \"474d07b8-cbb2-4beb-a17f-fe445d0c9ad4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892780-q4ns9" Nov 01 21:00:00.758231 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.756261 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38c14ac7-4673-46b7-a721-652a972f3ba5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892780-p6gfx\" (UID: \"38c14ac7-4673-46b7-a721-652a972f3ba5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892780-p6gfx" Nov 01 21:00:00.770158 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.770084 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:00.789926 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.789871 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892780-jmsct" Nov 01 21:00:00.801268 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.793147 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd038b-29892780-vltsq"] Nov 01 21:00:00.801268 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.793214 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892780-xxrwz"] Nov 01 21:00:00.801268 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.793245 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892780-7pl4v"] Nov 01 21:00:00.801268 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.793278 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b27e66-29892780-hvgx4"] Nov 01 21:00:00.801268 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.793306 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892780-jmsct"] Nov 01 21:00:00.801268 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.793331 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892780-vdj2s"] Nov 01 21:00:00.801268 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.793360 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw"] Nov 01 21:00:00.801268 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.793396 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892780-6xlxn"] Nov 01 21:00:00.801268 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.793422 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892780-p6gfx"] Nov 01 21:00:00.801268 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.793457 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892780-9tm2f"] Nov 01 21:00:00.801268 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.793484 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892780-q4ns9"] Nov 01 21:00:00.801268 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.793513 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892780-96d26"] Nov 01 21:00:00.801268 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.793774 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw" podUID="ad03483e-106e-4de1-b896-d71bff38f81d" containerName="istio-proxy" containerID="cri-o://a6b8ef447c1a49b73cfe47caef054e5e93533fb23b177823a88f4f99cd89e50f" gracePeriod=30 Nov 01 21:00:00.801268 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.794047 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892780-6xlxn" Nov 01 21:00:00.831709 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.831648 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqdzk\" (UniqueName: \"kubernetes.io/projected/474d07b8-cbb2-4beb-a17f-fe445d0c9ad4-kube-api-access-pqdzk\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892780-q4ns9\" (UID: \"474d07b8-cbb2-4beb-a17f-fe445d0c9ad4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892780-q4ns9" Nov 01 21:00:00.833823 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.833757 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrhn2\" (UniqueName: \"kubernetes.io/projected/38c14ac7-4673-46b7-a721-652a972f3ba5-kube-api-access-nrhn2\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892780-p6gfx\" (UID: \"38c14ac7-4673-46b7-a721-652a972f3ba5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892780-p6gfx" Nov 01 21:00:00.837793 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.837732 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892780-96d26" Nov 01 21:00:00.873040 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.867149 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w65xg\" (UniqueName: \"kubernetes.io/projected/d0f5aabc-58f2-4166-af2d-39aa67592cb1-kube-api-access-w65xg\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892780-9tm2f\" (UID: \"d0f5aabc-58f2-4166-af2d-39aa67592cb1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892780-9tm2f" Nov 01 21:00:00.873040 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.869841 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52fb948e-15a6-400e-be44-2e88a1fb1501-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892780-6xlxn\" (UID: \"52fb948e-15a6-400e-be44-2e88a1fb1501\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892780-6xlxn" Nov 01 21:00:00.873040 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.869905 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46v8z\" (UniqueName: \"kubernetes.io/projected/52fb948e-15a6-400e-be44-2e88a1fb1501-kube-api-access-46v8z\") pod \"maas-api-key-cleanup-29892780-6xlxn\" (UID: \"52fb948e-15a6-400e-be44-2e88a1fb1501\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892780-6xlxn" Nov 01 21:00:00.873040 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.870212 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0f5aabc-58f2-4166-af2d-39aa67592cb1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892780-9tm2f\" (UID: \"d0f5aabc-58f2-4166-af2d-39aa67592cb1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892780-9tm2f" Nov 01 21:00:00.873040 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.871038 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0f5aabc-58f2-4166-af2d-39aa67592cb1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892780-9tm2f\" (UID: \"d0f5aabc-58f2-4166-af2d-39aa67592cb1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892780-9tm2f" Nov 01 21:00:00.933939 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.933873 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w65xg\" (UniqueName: \"kubernetes.io/projected/d0f5aabc-58f2-4166-af2d-39aa67592cb1-kube-api-access-w65xg\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892780-9tm2f\" (UID: \"d0f5aabc-58f2-4166-af2d-39aa67592cb1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892780-9tm2f" Nov 01 21:00:00.974238 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.974197 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52fb948e-15a6-400e-be44-2e88a1fb1501-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892780-6xlxn\" (UID: \"52fb948e-15a6-400e-be44-2e88a1fb1501\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892780-6xlxn" Nov 01 21:00:00.974438 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.974269 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46v8z\" (UniqueName: \"kubernetes.io/projected/52fb948e-15a6-400e-be44-2e88a1fb1501-kube-api-access-46v8z\") pod \"maas-api-key-cleanup-29892780-6xlxn\" (UID: \"52fb948e-15a6-400e-be44-2e88a1fb1501\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892780-6xlxn" Nov 01 21:00:00.975795 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.975760 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52fb948e-15a6-400e-be44-2e88a1fb1501-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892780-6xlxn\" (UID: \"52fb948e-15a6-400e-be44-2e88a1fb1501\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892780-6xlxn" Nov 01 21:00:00.976463 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.976439 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b27e66-29892780-hvgx4" Nov 01 21:00:00.977454 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:00.977382 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:01.047929 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:01.047320 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892780-q4ns9" Nov 01 21:00:01.053827 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:01.053731 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46v8z\" (UniqueName: \"kubernetes.io/projected/52fb948e-15a6-400e-be44-2e88a1fb1501-kube-api-access-46v8z\") pod \"maas-api-key-cleanup-29892780-6xlxn\" (UID: \"52fb948e-15a6-400e-be44-2e88a1fb1501\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892780-6xlxn" Nov 01 21:00:01.114345 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:01.110251 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892780-p6gfx" Nov 01 21:00:01.169759 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:01.169729 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892780-9tm2f" Nov 01 21:00:01.221033 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:01.220268 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892780-6xlxn" Nov 01 21:00:01.279271 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:01.276658 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:01.501559 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:01.501462 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892780-xxrwz"] Nov 01 21:00:01.512340 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:01.511211 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd038b-29892780-vltsq"] Nov 01 21:00:01.521417 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:00:01.521384 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb19e6e75_0ef8_440d_8617_e5c71bfde0d2.slice/crio-994947f0e8fa04bb3c0e6620bb9c1cdbba6cc657e0411845a993bd16b5a4b94b WatchSource:0}: Error finding container 994947f0e8fa04bb3c0e6620bb9c1cdbba6cc657e0411845a993bd16b5a4b94b: Status 404 returned error can't find the container with id 994947f0e8fa04bb3c0e6620bb9c1cdbba6cc657e0411845a993bd16b5a4b94b Nov 01 21:00:01.526584 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:01.526560 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892780-vdj2s"] Nov 01 21:00:01.666133 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:00:01.666100 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5aa9680_52ae_4173_875b_1097a5963fc0.slice/crio-65824fed449713ee1a393aa417c114e443691904db47eeba9c603b3a9c3328d1 WatchSource:0}: Error finding container 65824fed449713ee1a393aa417c114e443691904db47eeba9c603b3a9c3328d1: Status 404 returned error can't find the container with id 65824fed449713ee1a393aa417c114e443691904db47eeba9c603b3a9c3328d1 Nov 01 21:00:01.668512 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:01.668445 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892780-jmsct"] Nov 01 21:00:01.714125 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:01.713195 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892780-96d26"] Nov 01 21:00:01.719623 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:01.719576 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892780-7pl4v"] Nov 01 21:00:02.085352 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:02.084843 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892780-q4ns9"] Nov 01 21:00:02.115163 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:02.114916 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892780-jmsct" event={"ID":"e5aa9680-52ae-4173-875b-1097a5963fc0","Type":"ContainerStarted","Data":"65824fed449713ee1a393aa417c114e443691904db47eeba9c603b3a9c3328d1"} Nov 01 21:00:02.149184 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:02.145117 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b27e66-29892780-hvgx4"] Nov 01 21:00:02.149184 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:02.145200 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892780-xxrwz" event={"ID":"b4578009-b3ff-4ce6-825a-8f92c1d3d72d","Type":"ContainerStarted","Data":"c36206bb3bc72cef16a5b9234e92c84ab61800161e90e4ff0a14399a66674b38"} Nov 01 21:00:02.157507 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:02.157471 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892780-vdj2s" event={"ID":"ce36bd6e-c550-4cd7-8f34-97b71968e4a2","Type":"ContainerStarted","Data":"99fb62932bed3a27a7aa019073e66b4152215a56e81414419d909833d88c6203"} Nov 01 21:00:02.164990 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:02.164961 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892780-7pl4v" event={"ID":"da6f250e-6516-4a04-a37a-f30c02ec682b","Type":"ContainerStarted","Data":"9221b19be74cc18e50987c0073c0fd8442013d6701c47e33ae190cc9f3975ead"} Nov 01 21:00:02.169543 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:02.166216 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892780-9tm2f"] Nov 01 21:00:02.169543 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:02.169349 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892780-96d26" event={"ID":"cfd728c4-7f6b-4197-a147-1f340427f414","Type":"ContainerStarted","Data":"fbff743b8f19a7dfeada1d3da59be3eaf945b6f3a127ef8d46d1e918105af4a8"} Nov 01 21:00:02.171353 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:02.171163 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd038b-29892780-vltsq" event={"ID":"b19e6e75-0ef8-440d-8617-e5c71bfde0d2","Type":"ContainerStarted","Data":"994947f0e8fa04bb3c0e6620bb9c1cdbba6cc657e0411845a993bd16b5a4b94b"} Nov 01 21:00:02.180485 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:02.180451 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892780-p6gfx"] Nov 01 21:00:02.440610 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:02.440576 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892780-6xlxn"] Nov 01 21:00:02.499824 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:02.499784 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h"] Nov 01 21:00:02.591637 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:02.591227 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h"] Nov 01 21:00:02.591637 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:02.591361 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h" Nov 01 21:00:02.628979 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:02.628946 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:02.687049 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:02.672750 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2fa50d2-8fa0-4997-9fdf-354318d94561-istio-token\") pod \"e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h\" (UID: \"c2fa50d2-8fa0-4997-9fdf-354318d94561\") " pod="openshift-ingress/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h" Nov 01 21:00:02.687049 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:02.672803 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2fa50d2-8fa0-4997-9fdf-354318d94561-istio-podinfo\") pod \"e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h\" (UID: \"c2fa50d2-8fa0-4997-9fdf-354318d94561\") " pod="openshift-ingress/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h" Nov 01 21:00:02.687049 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:02.672866 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2fa50d2-8fa0-4997-9fdf-354318d94561-workload-certs\") pod \"e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h\" (UID: \"c2fa50d2-8fa0-4997-9fdf-354318d94561\") " pod="openshift-ingress/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h" Nov 01 21:00:02.687049 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:02.672978 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zthvd\" (UniqueName: \"kubernetes.io/projected/c2fa50d2-8fa0-4997-9fdf-354318d94561-kube-api-access-zthvd\") pod \"e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h\" (UID: \"c2fa50d2-8fa0-4997-9fdf-354318d94561\") " pod="openshift-ingress/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h" Nov 01 21:00:02.687049 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:02.673043 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2fa50d2-8fa0-4997-9fdf-354318d94561-istio-data\") pod \"e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h\" (UID: \"c2fa50d2-8fa0-4997-9fdf-354318d94561\") " pod="openshift-ingress/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h" Nov 01 21:00:02.687049 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:02.678507 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2fa50d2-8fa0-4997-9fdf-354318d94561-credential-socket\") pod \"e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h\" (UID: \"c2fa50d2-8fa0-4997-9fdf-354318d94561\") " pod="openshift-ingress/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h" Nov 01 21:00:02.687049 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:02.678578 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2fa50d2-8fa0-4997-9fdf-354318d94561-workload-socket\") pod \"e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h\" (UID: \"c2fa50d2-8fa0-4997-9fdf-354318d94561\") " pod="openshift-ingress/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h" Nov 01 21:00:02.687049 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:02.678682 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2fa50d2-8fa0-4997-9fdf-354318d94561-istiod-ca-cert\") pod \"e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h\" (UID: \"c2fa50d2-8fa0-4997-9fdf-354318d94561\") " pod="openshift-ingress/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h" Nov 01 21:00:02.687049 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:02.678788 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2fa50d2-8fa0-4997-9fdf-354318d94561-istio-envoy\") pod \"e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h\" (UID: \"c2fa50d2-8fa0-4997-9fdf-354318d94561\") " pod="openshift-ingress/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h" Nov 01 21:00:02.687049 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:02.684197 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-29baa8-openshift-default-dockercfg-pn5jd\"" Nov 01 21:00:02.781404 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:02.781355 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2fa50d2-8fa0-4997-9fdf-354318d94561-istio-token\") pod \"e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h\" (UID: \"c2fa50d2-8fa0-4997-9fdf-354318d94561\") " pod="openshift-ingress/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h" Nov 01 21:00:02.781777 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:02.781748 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2fa50d2-8fa0-4997-9fdf-354318d94561-istio-podinfo\") pod \"e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h\" (UID: \"c2fa50d2-8fa0-4997-9fdf-354318d94561\") " pod="openshift-ingress/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h" Nov 01 21:00:02.782651 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:02.782626 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2fa50d2-8fa0-4997-9fdf-354318d94561-workload-certs\") pod \"e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h\" (UID: \"c2fa50d2-8fa0-4997-9fdf-354318d94561\") " pod="openshift-ingress/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h" Nov 01 21:00:02.782949 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:02.782927 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zthvd\" (UniqueName: \"kubernetes.io/projected/c2fa50d2-8fa0-4997-9fdf-354318d94561-kube-api-access-zthvd\") pod \"e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h\" (UID: \"c2fa50d2-8fa0-4997-9fdf-354318d94561\") " pod="openshift-ingress/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h" Nov 01 21:00:02.783193 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:02.783173 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2fa50d2-8fa0-4997-9fdf-354318d94561-istio-data\") pod \"e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h\" (UID: \"c2fa50d2-8fa0-4997-9fdf-354318d94561\") " pod="openshift-ingress/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h" Nov 01 21:00:02.783875 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:02.783841 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2fa50d2-8fa0-4997-9fdf-354318d94561-credential-socket\") pod \"e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h\" (UID: \"c2fa50d2-8fa0-4997-9fdf-354318d94561\") " pod="openshift-ingress/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h" Nov 01 21:00:02.784088 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:02.784053 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2fa50d2-8fa0-4997-9fdf-354318d94561-workload-socket\") pod \"e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h\" (UID: \"c2fa50d2-8fa0-4997-9fdf-354318d94561\") " pod="openshift-ingress/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h" Nov 01 21:00:02.784304 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:02.784285 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2fa50d2-8fa0-4997-9fdf-354318d94561-istiod-ca-cert\") pod \"e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h\" (UID: \"c2fa50d2-8fa0-4997-9fdf-354318d94561\") " pod="openshift-ingress/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h" Nov 01 21:00:02.784555 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:02.784526 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2fa50d2-8fa0-4997-9fdf-354318d94561-istio-envoy\") pod \"e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h\" (UID: \"c2fa50d2-8fa0-4997-9fdf-354318d94561\") " pod="openshift-ingress/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h" Nov 01 21:00:02.855475 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:02.803801 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2fa50d2-8fa0-4997-9fdf-354318d94561-credential-socket\") pod \"e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h\" (UID: \"c2fa50d2-8fa0-4997-9fdf-354318d94561\") " pod="openshift-ingress/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h" Nov 01 21:00:02.855475 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:02.804113 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2fa50d2-8fa0-4997-9fdf-354318d94561-istio-data\") pod \"e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h\" (UID: \"c2fa50d2-8fa0-4997-9fdf-354318d94561\") " pod="openshift-ingress/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h" Nov 01 21:00:02.855475 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:02.804345 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2fa50d2-8fa0-4997-9fdf-354318d94561-workload-certs\") pod \"e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h\" (UID: \"c2fa50d2-8fa0-4997-9fdf-354318d94561\") " pod="openshift-ingress/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h" Nov 01 21:00:02.855475 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:02.804567 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2fa50d2-8fa0-4997-9fdf-354318d94561-workload-socket\") pod \"e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h\" (UID: \"c2fa50d2-8fa0-4997-9fdf-354318d94561\") " pod="openshift-ingress/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h" Nov 01 21:00:02.855475 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:02.805567 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2fa50d2-8fa0-4997-9fdf-354318d94561-istiod-ca-cert\") pod \"e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h\" (UID: \"c2fa50d2-8fa0-4997-9fdf-354318d94561\") " pod="openshift-ingress/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h" Nov 01 21:00:02.855475 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:02.838410 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2fa50d2-8fa0-4997-9fdf-354318d94561-istio-envoy\") pod \"e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h\" (UID: \"c2fa50d2-8fa0-4997-9fdf-354318d94561\") " pod="openshift-ingress/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h" Nov 01 21:00:02.866887 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:02.866809 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2fa50d2-8fa0-4997-9fdf-354318d94561-istio-podinfo\") pod \"e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h\" (UID: \"c2fa50d2-8fa0-4997-9fdf-354318d94561\") " pod="openshift-ingress/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h" Nov 01 21:00:02.892568 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:02.892322 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zthvd\" (UniqueName: \"kubernetes.io/projected/c2fa50d2-8fa0-4997-9fdf-354318d94561-kube-api-access-zthvd\") pod \"e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h\" (UID: \"c2fa50d2-8fa0-4997-9fdf-354318d94561\") " pod="openshift-ingress/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h" Nov 01 21:00:02.896680 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:02.896600 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2fa50d2-8fa0-4997-9fdf-354318d94561-istio-token\") pod \"e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h\" (UID: \"c2fa50d2-8fa0-4997-9fdf-354318d94561\") " pod="openshift-ingress/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h" Nov 01 21:00:02.969019 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:02.964394 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h" Nov 01 21:00:02.969019 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:02.968102 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:03.188292 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:03.187678 4484 generic.go:356] "Generic (PLEG): container finished" podID="b4578009-b3ff-4ce6-825a-8f92c1d3d72d" containerID="64a056b7cd0dc4acbc2f9d31e6f5e563a40d4a6b99a426ffc06825adfe0d0032" exitCode=60 Nov 01 21:00:03.188591 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:03.187869 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892780-xxrwz" event={"ID":"b4578009-b3ff-4ce6-825a-8f92c1d3d72d","Type":"ContainerDied","Data":"64a056b7cd0dc4acbc2f9d31e6f5e563a40d4a6b99a426ffc06825adfe0d0032"} Nov 01 21:00:03.188712 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:03.188254 4484 scope.go:122] "RemoveContainer" containerID="64a056b7cd0dc4acbc2f9d31e6f5e563a40d4a6b99a426ffc06825adfe0d0032" Nov 01 21:00:03.193376 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:03.192882 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892780-9tm2f" event={"ID":"d0f5aabc-58f2-4166-af2d-39aa67592cb1","Type":"ContainerStarted","Data":"67b7695b29080b35d23064ab804827c35a1f9fc02fe23f336c5d043e07498617"} Nov 01 21:00:03.193376 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:03.192960 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892780-9tm2f" event={"ID":"d0f5aabc-58f2-4166-af2d-39aa67592cb1","Type":"ContainerStarted","Data":"1661dc1e34f2174cccb6ecbbf7541d5e0f8ead6921d63bab91b21052597366a7"} Nov 01 21:00:03.195765 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:03.195736 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892780-6xlxn" event={"ID":"52fb948e-15a6-400e-be44-2e88a1fb1501","Type":"ContainerStarted","Data":"e26e9408ee20b56e11e215f7af555968f26c23409d34535f1091d18c8062660a"} Nov 01 21:00:03.198809 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:03.198778 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892780-q4ns9" event={"ID":"474d07b8-cbb2-4beb-a17f-fe445d0c9ad4","Type":"ContainerStarted","Data":"52d1451717da5c8fbaf84e34ee3954b18ded07be6a4653adbe68baedb21848d2"} Nov 01 21:00:03.198918 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:03.198837 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892780-q4ns9" event={"ID":"474d07b8-cbb2-4beb-a17f-fe445d0c9ad4","Type":"ContainerStarted","Data":"e08897e4ef7f7fc1c8c64f184f9e8e279cab445a9e87eacf51371f5e9a910b6d"} Nov 01 21:00:03.201076 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:03.200962 4484 generic.go:356] "Generic (PLEG): container finished" podID="ce36bd6e-c550-4cd7-8f34-97b71968e4a2" containerID="958e5bda1df1d6bca428165b7559663e500a179c74a4950112a79f1fbd041cea" exitCode=60 Nov 01 21:00:03.201206 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:03.201152 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892780-vdj2s" event={"ID":"ce36bd6e-c550-4cd7-8f34-97b71968e4a2","Type":"ContainerDied","Data":"958e5bda1df1d6bca428165b7559663e500a179c74a4950112a79f1fbd041cea"} Nov 01 21:00:03.201931 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:03.201907 4484 scope.go:122] "RemoveContainer" containerID="958e5bda1df1d6bca428165b7559663e500a179c74a4950112a79f1fbd041cea" Nov 01 21:00:03.206140 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:03.204923 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892780-7pl4v" event={"ID":"da6f250e-6516-4a04-a37a-f30c02ec682b","Type":"ContainerStarted","Data":"829dd57c2764e7e1d04b0a9dd0116884ba1f9df0e6a23bd748ef9af3df7efa4d"} Nov 01 21:00:03.207506 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:03.207478 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892780-96d26" event={"ID":"cfd728c4-7f6b-4197-a147-1f340427f414","Type":"ContainerStarted","Data":"13416c43a31bd1cff128b3f3b7bbf17dd9f5dde99022669f311c2180007397e1"} Nov 01 21:00:03.212110 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:03.212076 4484 generic.go:356] "Generic (PLEG): container finished" podID="b19e6e75-0ef8-440d-8617-e5c71bfde0d2" containerID="c0a0694d1b2c91637207732b6a2a5e10c7ce65fe1d9c4ec3b9113e63e478cfb1" exitCode=60 Nov 01 21:00:03.212600 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:03.212577 4484 scope.go:122] "RemoveContainer" containerID="c0a0694d1b2c91637207732b6a2a5e10c7ce65fe1d9c4ec3b9113e63e478cfb1" Nov 01 21:00:03.213163 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:03.213134 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd038b-29892780-vltsq" event={"ID":"b19e6e75-0ef8-440d-8617-e5c71bfde0d2","Type":"ContainerDied","Data":"c0a0694d1b2c91637207732b6a2a5e10c7ce65fe1d9c4ec3b9113e63e478cfb1"} Nov 01 21:00:03.217056 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:03.217025 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892780-p6gfx" event={"ID":"38c14ac7-4673-46b7-a721-652a972f3ba5","Type":"ContainerStarted","Data":"b30496e3a31d655f604ec8e4e435f097d4a96c1e730b19e6cfed76dc9565af42"} Nov 01 21:00:03.217204 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:03.217083 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892780-p6gfx" event={"ID":"38c14ac7-4673-46b7-a721-652a972f3ba5","Type":"ContainerStarted","Data":"44c4cdbec929dc88bccc5c35556f6a7c2d64567b8df9a09308471f51d43149d9"} Nov 01 21:00:03.221172 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:03.221103 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892780-jmsct" event={"ID":"e5aa9680-52ae-4173-875b-1097a5963fc0","Type":"ContainerStarted","Data":"e3a2ba1937596fbcff9b024fcc902f11e8d67631f6ebe75e90ad74c5f22aee2c"} Nov 01 21:00:03.232924 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:03.232891 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b27e66-29892780-hvgx4" event={"ID":"7369891a-d9d2-42f1-8165-f817e9c00984","Type":"ContainerStarted","Data":"37357371e7432de3890d6db5c772d01bd721bfafcc1843ad57ba231db1d9595f"} Nov 01 21:00:03.233142 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:03.233127 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b27e66-29892780-hvgx4" event={"ID":"7369891a-d9d2-42f1-8165-f817e9c00984","Type":"ContainerStarted","Data":"ac161d6640924630e39871b79cf7498dcde038e0511fe40cf9b26ce694131483"} Nov 01 21:00:03.489802 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:03.489746 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892780-q4ns9" podStartSLOduration=3.489726897 podStartE2EDuration="3.489726897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:00:03.429319888 +0000 UTC m=+1754.206960027" watchObservedRunningTime="2026-11-01 21:00:03.489726897 +0000 UTC m=+1754.267367033" Nov 01 21:00:03.496682 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:03.496600 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h"] Nov 01 21:00:03.528935 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:00:03.528895 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2fa50d2_8fa0_4997_9fdf_354318d94561.slice/crio-401ed49514df8e798455704fc65da17495321ec2f7974781f7c83e78fe59db9b WatchSource:0}: Error finding container 401ed49514df8e798455704fc65da17495321ec2f7974781f7c83e78fe59db9b: Status 404 returned error can't find the container with id 401ed49514df8e798455704fc65da17495321ec2f7974781f7c83e78fe59db9b Nov 01 21:00:03.532188 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:03.532151 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:00:03.532330 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:03.532238 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:00:03.532330 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:03.532285 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:00:03.548806 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:03.548710 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892780-9tm2f" podStartSLOduration=3.548687771 podStartE2EDuration="3.548687771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:00:03.545493698 +0000 UTC m=+1754.323133834" watchObservedRunningTime="2026-11-01 21:00:03.548687771 +0000 UTC m=+1754.326327907" Nov 01 21:00:03.774856 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:03.774778 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892780-p6gfx" podStartSLOduration=3.774763048 podStartE2EDuration="3.774763048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:00:03.773850522 +0000 UTC m=+1754.551490658" watchObservedRunningTime="2026-11-01 21:00:03.774763048 +0000 UTC m=+1754.552403183" Nov 01 21:00:03.859914 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:03.859834 4484 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 21:00:03.859914 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:03.859893 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8" podUID="b373faf1-ffb2-4a45-b433-0669635a397c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 21:00:04.119144 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:04.119087 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892780-7pl4v" podStartSLOduration=4.119067261 podStartE2EDuration="4.119067261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:00:04.005358767 +0000 UTC m=+1754.782998904" watchObservedRunningTime="2026-11-01 21:00:04.119067261 +0000 UTC m=+1754.896707398" Nov 01 21:00:04.125245 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:04.124875 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892780-jmsct" podStartSLOduration=4.12485601 podStartE2EDuration="4.12485601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:00:04.118867452 +0000 UTC m=+1754.896507589" watchObservedRunningTime="2026-11-01 21:00:04.12485601 +0000 UTC m=+1754.902496146" Nov 01 21:00:04.238442 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:04.237962 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b27e66-29892780-hvgx4" podStartSLOduration=4.237943291 podStartE2EDuration="4.237943291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:00:04.232767936 +0000 UTC m=+1755.010408072" watchObservedRunningTime="2026-11-01 21:00:04.237943291 +0000 UTC m=+1755.015583421" Nov 01 21:00:04.251149 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:04.251095 4484 generic.go:356] "Generic (PLEG): container finished" podID="cfd728c4-7f6b-4197-a147-1f340427f414" containerID="13416c43a31bd1cff128b3f3b7bbf17dd9f5dde99022669f311c2180007397e1" exitCode=60 Nov 01 21:00:04.251382 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:04.251286 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892780-96d26" event={"ID":"cfd728c4-7f6b-4197-a147-1f340427f414","Type":"ContainerDied","Data":"13416c43a31bd1cff128b3f3b7bbf17dd9f5dde99022669f311c2180007397e1"} Nov 01 21:00:04.251782 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:04.251760 4484 scope.go:122] "RemoveContainer" containerID="13416c43a31bd1cff128b3f3b7bbf17dd9f5dde99022669f311c2180007397e1" Nov 01 21:00:04.255224 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:04.255199 4484 generic.go:356] "Generic (PLEG): container finished" podID="38c14ac7-4673-46b7-a721-652a972f3ba5" containerID="b30496e3a31d655f604ec8e4e435f097d4a96c1e730b19e6cfed76dc9565af42" exitCode=60 Nov 01 21:00:04.255362 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:04.255339 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892780-p6gfx" event={"ID":"38c14ac7-4673-46b7-a721-652a972f3ba5","Type":"ContainerDied","Data":"b30496e3a31d655f604ec8e4e435f097d4a96c1e730b19e6cfed76dc9565af42"} Nov 01 21:00:04.255778 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:04.255754 4484 scope.go:122] "RemoveContainer" containerID="b30496e3a31d655f604ec8e4e435f097d4a96c1e730b19e6cfed76dc9565af42" Nov 01 21:00:04.268965 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:04.268881 4484 generic.go:356] "Generic (PLEG): container finished" podID="e5aa9680-52ae-4173-875b-1097a5963fc0" containerID="e3a2ba1937596fbcff9b024fcc902f11e8d67631f6ebe75e90ad74c5f22aee2c" exitCode=60 Nov 01 21:00:04.269162 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:04.269010 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892780-jmsct" event={"ID":"e5aa9680-52ae-4173-875b-1097a5963fc0","Type":"ContainerDied","Data":"e3a2ba1937596fbcff9b024fcc902f11e8d67631f6ebe75e90ad74c5f22aee2c"} Nov 01 21:00:04.269707 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:04.269677 4484 scope.go:122] "RemoveContainer" containerID="e3a2ba1937596fbcff9b024fcc902f11e8d67631f6ebe75e90ad74c5f22aee2c" Nov 01 21:00:04.280380 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:04.279583 4484 generic.go:356] "Generic (PLEG): container finished" podID="7369891a-d9d2-42f1-8165-f817e9c00984" containerID="37357371e7432de3890d6db5c772d01bd721bfafcc1843ad57ba231db1d9595f" exitCode=60 Nov 01 21:00:04.280380 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:04.279723 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b27e66-29892780-hvgx4" event={"ID":"7369891a-d9d2-42f1-8165-f817e9c00984","Type":"ContainerDied","Data":"37357371e7432de3890d6db5c772d01bd721bfafcc1843ad57ba231db1d9595f"} Nov 01 21:00:04.280380 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:04.280367 4484 scope.go:122] "RemoveContainer" containerID="37357371e7432de3890d6db5c772d01bd721bfafcc1843ad57ba231db1d9595f" Nov 01 21:00:04.284797 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:04.284759 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892780-xxrwz" event={"ID":"b4578009-b3ff-4ce6-825a-8f92c1d3d72d","Type":"ContainerStarted","Data":"9d0b372aec9fcdefa4b06701f0f453d152bba5f0ba272bf6dbcc18271befddcc"} Nov 01 21:00:04.287368 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:04.287340 4484 generic.go:356] "Generic (PLEG): container finished" podID="d0f5aabc-58f2-4166-af2d-39aa67592cb1" containerID="67b7695b29080b35d23064ab804827c35a1f9fc02fe23f336c5d043e07498617" exitCode=60 Nov 01 21:00:04.287528 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:04.287459 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892780-9tm2f" event={"ID":"d0f5aabc-58f2-4166-af2d-39aa67592cb1","Type":"ContainerDied","Data":"67b7695b29080b35d23064ab804827c35a1f9fc02fe23f336c5d043e07498617"} Nov 01 21:00:04.287980 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:04.287710 4484 scope.go:122] "RemoveContainer" containerID="67b7695b29080b35d23064ab804827c35a1f9fc02fe23f336c5d043e07498617" Nov 01 21:00:04.290058 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:04.290031 4484 generic.go:356] "Generic (PLEG): container finished" podID="52fb948e-15a6-400e-be44-2e88a1fb1501" containerID="64fb35033e756f653fa8e0ea2202f417fc2b32a45a437ca053b5ca5219060891" exitCode=60 Nov 01 21:00:04.290141 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:04.290113 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892780-6xlxn" event={"ID":"52fb948e-15a6-400e-be44-2e88a1fb1501","Type":"ContainerDied","Data":"64fb35033e756f653fa8e0ea2202f417fc2b32a45a437ca053b5ca5219060891"} Nov 01 21:00:04.290591 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:04.290504 4484 scope.go:122] "RemoveContainer" containerID="64fb35033e756f653fa8e0ea2202f417fc2b32a45a437ca053b5ca5219060891" Nov 01 21:00:04.291889 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:04.291868 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h" event={"ID":"c2fa50d2-8fa0-4997-9fdf-354318d94561","Type":"ContainerStarted","Data":"401ed49514df8e798455704fc65da17495321ec2f7974781f7c83e78fe59db9b"} Nov 01 21:00:04.293624 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:04.293603 4484 generic.go:356] "Generic (PLEG): container finished" podID="474d07b8-cbb2-4beb-a17f-fe445d0c9ad4" containerID="52d1451717da5c8fbaf84e34ee3954b18ded07be6a4653adbe68baedb21848d2" exitCode=7 Nov 01 21:00:04.293722 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:04.293692 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892780-q4ns9" event={"ID":"474d07b8-cbb2-4beb-a17f-fe445d0c9ad4","Type":"ContainerDied","Data":"52d1451717da5c8fbaf84e34ee3954b18ded07be6a4653adbe68baedb21848d2"} Nov 01 21:00:04.293977 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:04.293958 4484 scope.go:122] "RemoveContainer" containerID="52d1451717da5c8fbaf84e34ee3954b18ded07be6a4653adbe68baedb21848d2" Nov 01 21:00:04.295480 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:04.295456 4484 generic.go:356] "Generic (PLEG): container finished" podID="da6f250e-6516-4a04-a37a-f30c02ec682b" containerID="829dd57c2764e7e1d04b0a9dd0116884ba1f9df0e6a23bd748ef9af3df7efa4d" exitCode=60 Nov 01 21:00:04.295588 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:04.295510 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892780-7pl4v" event={"ID":"da6f250e-6516-4a04-a37a-f30c02ec682b","Type":"ContainerDied","Data":"829dd57c2764e7e1d04b0a9dd0116884ba1f9df0e6a23bd748ef9af3df7efa4d"} Nov 01 21:00:04.295793 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:04.295777 4484 scope.go:122] "RemoveContainer" containerID="829dd57c2764e7e1d04b0a9dd0116884ba1f9df0e6a23bd748ef9af3df7efa4d" Nov 01 21:00:04.895484 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:04.722946 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892780-xxrwz" podStartSLOduration=4.722926154 podStartE2EDuration="4.722926154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:00:04.720376247 +0000 UTC m=+1755.498016382" watchObservedRunningTime="2026-11-01 21:00:04.722926154 +0000 UTC m=+1755.500566290" Nov 01 21:00:04.895484 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:04.832520 4484 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 21:00:04.895484 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:04.832571 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw" podUID="ad03483e-106e-4de1-b896-d71bff38f81d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 21:00:05.060209 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.060183 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf" Nov 01 21:00:05.166695 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.166663 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef2f968c-433d-49f8-8fc4-7c8551d647fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef2f968c-433d-49f8-8fc4-7c8551d647fb-workload-socket\") pod \"ef2f968c-433d-49f8-8fc4-7c8551d647fb\" (UID: \"ef2f968c-433d-49f8-8fc4-7c8551d647fb\") " Nov 01 21:00:05.166846 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.166722 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef2f968c-433d-49f8-8fc4-7c8551d647fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef2f968c-433d-49f8-8fc4-7c8551d647fb-istio-data\") pod \"ef2f968c-433d-49f8-8fc4-7c8551d647fb\" (UID: \"ef2f968c-433d-49f8-8fc4-7c8551d647fb\") " Nov 01 21:00:05.166846 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.166764 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef2f968c-433d-49f8-8fc4-7c8551d647fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef2f968c-433d-49f8-8fc4-7c8551d647fb-workload-certs\") pod \"ef2f968c-433d-49f8-8fc4-7c8551d647fb\" (UID: \"ef2f968c-433d-49f8-8fc4-7c8551d647fb\") " Nov 01 21:00:05.166846 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.166806 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef2f968c-433d-49f8-8fc4-7c8551d647fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef2f968c-433d-49f8-8fc4-7c8551d647fb-istiod-ca-cert\") pod \"ef2f968c-433d-49f8-8fc4-7c8551d647fb\" (UID: \"ef2f968c-433d-49f8-8fc4-7c8551d647fb\") " Nov 01 21:00:05.167066 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.166871 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef2f968c-433d-49f8-8fc4-7c8551d647fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef2f968c-433d-49f8-8fc4-7c8551d647fb-istio-envoy\") pod \"ef2f968c-433d-49f8-8fc4-7c8551d647fb\" (UID: \"ef2f968c-433d-49f8-8fc4-7c8551d647fb\") " Nov 01 21:00:05.167577 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.167387 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef2f968c-433d-49f8-8fc4-7c8551d647fb-istio-data" pod "ef2f968c-433d-49f8-8fc4-7c8551d647fb" (UID: "ef2f968c-433d-49f8-8fc4-7c8551d647fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:00:05.168080 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.167689 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef2f968c-433d-49f8-8fc4-7c8551d647fb-workload-socket" pod "ef2f968c-433d-49f8-8fc4-7c8551d647fb" (UID: "ef2f968c-433d-49f8-8fc4-7c8551d647fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:00:05.168931 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.168523 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef2f968c-433d-49f8-8fc4-7c8551d647fb-istiod-ca-cert" pod "ef2f968c-433d-49f8-8fc4-7c8551d647fb" (UID: "ef2f968c-433d-49f8-8fc4-7c8551d647fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:00:05.168931 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.168902 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef2f968c-433d-49f8-8fc4-7c8551d647fb-workload-certs" pod "ef2f968c-433d-49f8-8fc4-7c8551d647fb" (UID: "ef2f968c-433d-49f8-8fc4-7c8551d647fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:00:05.170861 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.169230 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef2f968c-433d-49f8-8fc4-7c8551d647fb-kube-api-access-jhvhs\" (UniqueName: \"kubernetes.io/projected/ef2f968c-433d-49f8-8fc4-7c8551d647fb-kube-api-access-jhvhs\") pod \"ef2f968c-433d-49f8-8fc4-7c8551d647fb\" (UID: \"ef2f968c-433d-49f8-8fc4-7c8551d647fb\") " Nov 01 21:00:05.170861 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.169268 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef2f968c-433d-49f8-8fc4-7c8551d647fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef2f968c-433d-49f8-8fc4-7c8551d647fb-credential-socket\") pod \"ef2f968c-433d-49f8-8fc4-7c8551d647fb\" (UID: \"ef2f968c-433d-49f8-8fc4-7c8551d647fb\") " Nov 01 21:00:05.170861 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.169346 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ef2f968c-433d-49f8-8fc4-7c8551d647fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef2f968c-433d-49f8-8fc4-7c8551d647fb-istio-podinfo\") pod \"ef2f968c-433d-49f8-8fc4-7c8551d647fb\" (UID: \"ef2f968c-433d-49f8-8fc4-7c8551d647fb\") " Nov 01 21:00:05.170861 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.169475 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef2f968c-433d-49f8-8fc4-7c8551d647fb-istio-token\" (UniqueName: \"kubernetes.io/projected/ef2f968c-433d-49f8-8fc4-7c8551d647fb-istio-token\") pod \"ef2f968c-433d-49f8-8fc4-7c8551d647fb\" (UID: \"ef2f968c-433d-49f8-8fc4-7c8551d647fb\") " Nov 01 21:00:05.170861 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.170315 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef2f968c-433d-49f8-8fc4-7c8551d647fb-credential-socket" pod "ef2f968c-433d-49f8-8fc4-7c8551d647fb" (UID: "ef2f968c-433d-49f8-8fc4-7c8551d647fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:00:05.171464 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.171442 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef2f968c-433d-49f8-8fc4-7c8551d647fb-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:05.171558 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.171469 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef2f968c-433d-49f8-8fc4-7c8551d647fb-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:05.171558 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.171483 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef2f968c-433d-49f8-8fc4-7c8551d647fb-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:05.171558 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.171498 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef2f968c-433d-49f8-8fc4-7c8551d647fb-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:05.171558 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.171512 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef2f968c-433d-49f8-8fc4-7c8551d647fb-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:05.307776 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.306280 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef2f968c-433d-49f8-8fc4-7c8551d647fb-istio-podinfo" pod "ef2f968c-433d-49f8-8fc4-7c8551d647fb" (UID: "ef2f968c-433d-49f8-8fc4-7c8551d647fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:00:05.307776 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.306520 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef2f968c-433d-49f8-8fc4-7c8551d647fb-kube-api-access-jhvhs" pod "ef2f968c-433d-49f8-8fc4-7c8551d647fb" (UID: "ef2f968c-433d-49f8-8fc4-7c8551d647fb"). InnerVolumeSpecName "kube-api-access-jhvhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:00:05.307776 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.306667 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef2f968c-433d-49f8-8fc4-7c8551d647fb-istio-token" pod "ef2f968c-433d-49f8-8fc4-7c8551d647fb" (UID: "ef2f968c-433d-49f8-8fc4-7c8551d647fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:00:05.307776 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.307079 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef2f968c-433d-49f8-8fc4-7c8551d647fb-istio-envoy" pod "ef2f968c-433d-49f8-8fc4-7c8551d647fb" (UID: "ef2f968c-433d-49f8-8fc4-7c8551d647fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:00:05.320354 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.320057 4484 generic.go:356] "Generic (PLEG): container finished" podID="b4578009-b3ff-4ce6-825a-8f92c1d3d72d" containerID="9d0b372aec9fcdefa4b06701f0f453d152bba5f0ba272bf6dbcc18271befddcc" exitCode=60 Nov 01 21:00:05.320354 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.320190 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892780-xxrwz" event={"ID":"b4578009-b3ff-4ce6-825a-8f92c1d3d72d","Type":"ContainerDied","Data":"9d0b372aec9fcdefa4b06701f0f453d152bba5f0ba272bf6dbcc18271befddcc"} Nov 01 21:00:05.320354 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.320246 4484 scope.go:122] "RemoveContainer" containerID="64a056b7cd0dc4acbc2f9d31e6f5e563a40d4a6b99a426ffc06825adfe0d0032" Nov 01 21:00:05.320573 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.320507 4484 scope.go:122] "RemoveContainer" containerID="9d0b372aec9fcdefa4b06701f0f453d152bba5f0ba272bf6dbcc18271befddcc" Nov 01 21:00:05.320985 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:00:05.320953 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892780-xxrwz_odh-ai-gateway-infra(b4578009-b3ff-4ce6-825a-8f92c1d3d72d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892780-xxrwz" podUID="b4578009-b3ff-4ce6-825a-8f92c1d3d72d" Nov 01 21:00:05.323966 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.323929 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892780-9tm2f" event={"ID":"d0f5aabc-58f2-4166-af2d-39aa67592cb1","Type":"ContainerStarted","Data":"764a3b1800e86a3495f68a14ecfda5256427b0ad2f3dae0f7028d7a07359b00d"} Nov 01 21:00:05.326296 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.326272 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h" event={"ID":"c2fa50d2-8fa0-4997-9fdf-354318d94561","Type":"ContainerStarted","Data":"208dd4fa036e63c78be6383d20c5d27a5acb11fe9023605beafda5d7a153a014"} Nov 01 21:00:05.328608 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.328587 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892780-q4ns9" event={"ID":"474d07b8-cbb2-4beb-a17f-fe445d0c9ad4","Type":"ContainerStarted","Data":"f5fcc30aa2fbf88fedd5d1850cfb015c3f61a92090dddf5ba5b52f0ae14a8d06"} Nov 01 21:00:05.331008 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.330971 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892780-vdj2s" event={"ID":"ce36bd6e-c550-4cd7-8f34-97b71968e4a2","Type":"ContainerStarted","Data":"cc5207f27244b099a2a26e328bbbe6c463f52f9fe5dd7491a9f72322f648daa1"} Nov 01 21:00:05.331390 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.331374 4484 scope.go:122] "RemoveContainer" containerID="cc5207f27244b099a2a26e328bbbe6c463f52f9fe5dd7491a9f72322f648daa1" Nov 01 21:00:05.331772 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:00:05.331746 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-cd248f-29892780-vdj2s_odh-ai-gateway-infra(ce36bd6e-c550-4cd7-8f34-97b71968e4a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892780-vdj2s" podUID="ce36bd6e-c550-4cd7-8f34-97b71968e4a2" Nov 01 21:00:05.334054 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.333760 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892780-7pl4v" event={"ID":"da6f250e-6516-4a04-a37a-f30c02ec682b","Type":"ContainerStarted","Data":"8b4816f8698560a1753b6a3691d097b27016c4ac7012b7ecd68d5948de464d2d"} Nov 01 21:00:05.335705 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.335683 4484 generic.go:356] "Generic (PLEG): container finished" podID="b373faf1-ffb2-4a45-b433-0669635a397c" containerID="b6541f7eefe7ba24ae375b42001c3d8d5be218cac65c3c82b0736e0ce422fffe" exitCode=0 Nov 01 21:00:05.335840 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.335764 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8" event={"ID":"b373faf1-ffb2-4a45-b433-0669635a397c","Type":"ContainerDied","Data":"b6541f7eefe7ba24ae375b42001c3d8d5be218cac65c3c82b0736e0ce422fffe"} Nov 01 21:00:05.339438 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.338600 4484 generic.go:356] "Generic (PLEG): container finished" podID="3ae0b2c8-270b-4a3f-bb16-523874681ade" containerID="acbed28624ebe42544acb96d055050fed39395495b39d480ce98a0590c69f879" exitCode=0 Nov 01 21:00:05.339438 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.338710 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv" event={"ID":"3ae0b2c8-270b-4a3f-bb16-523874681ade","Type":"ContainerDied","Data":"acbed28624ebe42544acb96d055050fed39395495b39d480ce98a0590c69f879"} Nov 01 21:00:05.342415 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.341551 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd038b-29892780-vltsq" event={"ID":"b19e6e75-0ef8-440d-8617-e5c71bfde0d2","Type":"ContainerStarted","Data":"664a1e35abd176532a1bebe3892ba5deea20919b1b062d25762c87dab4d27c8c"} Nov 01 21:00:05.342415 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.341630 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv" Nov 01 21:00:05.342415 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.341782 4484 scope.go:122] "RemoveContainer" containerID="664a1e35abd176532a1bebe3892ba5deea20919b1b062d25762c87dab4d27c8c" Nov 01 21:00:05.342415 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:00:05.342254 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-dd038b-29892780-vltsq_odh-ai-gateway-infra(b19e6e75-0ef8-440d-8617-e5c71bfde0d2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd038b-29892780-vltsq" podUID="b19e6e75-0ef8-440d-8617-e5c71bfde0d2" Nov 01 21:00:05.343930 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.343911 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892780-p6gfx" event={"ID":"38c14ac7-4673-46b7-a721-652a972f3ba5","Type":"ContainerStarted","Data":"680415e314c9d523bb13494243c4ac1bf4d90a45770b71f5ffb8c4c2b6eb7735"} Nov 01 21:00:05.346268 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.346246 4484 generic.go:356] "Generic (PLEG): container finished" podID="ef2f968c-433d-49f8-8fc4-7c8551d647fb" containerID="8eb9816d4eca1ec89f3adadcb08ce4caaa9cc08431c2dbe91e021610b5ae2efb" exitCode=0 Nov 01 21:00:05.346355 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.346288 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf" Nov 01 21:00:05.346355 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.346302 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf" event={"ID":"ef2f968c-433d-49f8-8fc4-7c8551d647fb","Type":"ContainerDied","Data":"8eb9816d4eca1ec89f3adadcb08ce4caaa9cc08431c2dbe91e021610b5ae2efb"} Nov 01 21:00:05.346355 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.346347 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf" event={"ID":"ef2f968c-433d-49f8-8fc4-7c8551d647fb","Type":"ContainerDied","Data":"05027bfb9625e224cf26f0a39f8b19ede4a13ceb3e6b012f6d53346a6a82c2e3"} Nov 01 21:00:05.348534 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.348500 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892780-jmsct" event={"ID":"e5aa9680-52ae-4173-875b-1097a5963fc0","Type":"ContainerStarted","Data":"78c31ce9df621c4320497c8e8b45517255aac51955e9116324f4c0d02644c553"} Nov 01 21:00:05.350665 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.350642 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b27e66-29892780-hvgx4" event={"ID":"7369891a-d9d2-42f1-8165-f817e9c00984","Type":"ContainerStarted","Data":"7e63bb45c74fef2b816cb0175b4280435c3110e0ad03786267ff37c6dcb73fe4"} Nov 01 21:00:05.353205 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.353184 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8" Nov 01 21:00:05.378191 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.376305 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef2f968c-433d-49f8-8fc4-7c8551d647fb-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:05.378191 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.376333 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef2f968c-433d-49f8-8fc4-7c8551d647fb-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:05.378191 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.376345 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhvhs\" (UniqueName: \"kubernetes.io/projected/ef2f968c-433d-49f8-8fc4-7c8551d647fb-kube-api-access-jhvhs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:05.378191 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.376355 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef2f968c-433d-49f8-8fc4-7c8551d647fb-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:05.399074 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.399044 4484 scope.go:122] "RemoveContainer" containerID="5f3068fefdcfccb633c8492083fc1e3420cd25dc941ed3797c287eeaadc6303d" Nov 01 21:00:05.476696 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.476674 4484 scope.go:122] "RemoveContainer" containerID="8eb9816d4eca1ec89f3adadcb08ce4caaa9cc08431c2dbe91e021610b5ae2efb" Nov 01 21:00:05.479246 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.479195 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ae0b2c8-270b-4a3f-bb16-523874681ade-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ae0b2c8-270b-4a3f-bb16-523874681ade-credential-socket\") pod \"3ae0b2c8-270b-4a3f-bb16-523874681ade\" (UID: \"3ae0b2c8-270b-4a3f-bb16-523874681ade\") " Nov 01 21:00:05.479810 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.479720 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ae0b2c8-270b-4a3f-bb16-523874681ade-credential-socket" pod "3ae0b2c8-270b-4a3f-bb16-523874681ade" (UID: "3ae0b2c8-270b-4a3f-bb16-523874681ade"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:00:05.479987 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.479970 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b373faf1-ffb2-4a45-b433-0669635a397c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b373faf1-ffb2-4a45-b433-0669635a397c-credential-socket\") pod \"b373faf1-ffb2-4a45-b433-0669635a397c\" (UID: \"b373faf1-ffb2-4a45-b433-0669635a397c\") " Nov 01 21:00:05.480274 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.480126 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b373faf1-ffb2-4a45-b433-0669635a397c-credential-socket" pod "b373faf1-ffb2-4a45-b433-0669635a397c" (UID: "b373faf1-ffb2-4a45-b433-0669635a397c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:00:05.480401 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.480253 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b373faf1-ffb2-4a45-b433-0669635a397c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b373faf1-ffb2-4a45-b433-0669635a397c-istio-podinfo\") pod \"b373faf1-ffb2-4a45-b433-0669635a397c\" (UID: \"b373faf1-ffb2-4a45-b433-0669635a397c\") " Nov 01 21:00:05.480401 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.480325 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3ae0b2c8-270b-4a3f-bb16-523874681ade-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ae0b2c8-270b-4a3f-bb16-523874681ade-istio-podinfo\") pod \"3ae0b2c8-270b-4a3f-bb16-523874681ade\" (UID: \"3ae0b2c8-270b-4a3f-bb16-523874681ade\") " Nov 01 21:00:05.480524 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.480408 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b373faf1-ffb2-4a45-b433-0669635a397c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b373faf1-ffb2-4a45-b433-0669635a397c-workload-socket\") pod \"b373faf1-ffb2-4a45-b433-0669635a397c\" (UID: \"b373faf1-ffb2-4a45-b433-0669635a397c\") " Nov 01 21:00:05.480585 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.480543 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ae0b2c8-270b-4a3f-bb16-523874681ade-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ae0b2c8-270b-4a3f-bb16-523874681ade-workload-socket\") pod \"3ae0b2c8-270b-4a3f-bb16-523874681ade\" (UID: \"3ae0b2c8-270b-4a3f-bb16-523874681ade\") " Nov 01 21:00:05.480644 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.480613 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ae0b2c8-270b-4a3f-bb16-523874681ade-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ae0b2c8-270b-4a3f-bb16-523874681ade-istio-envoy\") pod \"3ae0b2c8-270b-4a3f-bb16-523874681ade\" (UID: \"3ae0b2c8-270b-4a3f-bb16-523874681ade\") " Nov 01 21:00:05.481337 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.480743 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b373faf1-ffb2-4a45-b433-0669635a397c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b373faf1-ffb2-4a45-b433-0669635a397c-istiod-ca-cert\") pod \"b373faf1-ffb2-4a45-b433-0669635a397c\" (UID: \"b373faf1-ffb2-4a45-b433-0669635a397c\") " Nov 01 21:00:05.481337 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.480785 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b373faf1-ffb2-4a45-b433-0669635a397c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b373faf1-ffb2-4a45-b433-0669635a397c-istio-data\") pod \"b373faf1-ffb2-4a45-b433-0669635a397c\" (UID: \"b373faf1-ffb2-4a45-b433-0669635a397c\") " Nov 01 21:00:05.481337 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.480873 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ae0b2c8-270b-4a3f-bb16-523874681ade-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ae0b2c8-270b-4a3f-bb16-523874681ade-workload-certs\") pod \"3ae0b2c8-270b-4a3f-bb16-523874681ade\" (UID: \"3ae0b2c8-270b-4a3f-bb16-523874681ade\") " Nov 01 21:00:05.481337 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.481034 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ae0b2c8-270b-4a3f-bb16-523874681ade-kube-api-access-c5dj8\" (UniqueName: \"kubernetes.io/projected/3ae0b2c8-270b-4a3f-bb16-523874681ade-kube-api-access-c5dj8\") pod \"3ae0b2c8-270b-4a3f-bb16-523874681ade\" (UID: \"3ae0b2c8-270b-4a3f-bb16-523874681ade\") " Nov 01 21:00:05.481337 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.481097 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b373faf1-ffb2-4a45-b433-0669635a397c-istio-token\" (UniqueName: \"kubernetes.io/projected/b373faf1-ffb2-4a45-b433-0669635a397c-istio-token\") pod \"b373faf1-ffb2-4a45-b433-0669635a397c\" (UID: \"b373faf1-ffb2-4a45-b433-0669635a397c\") " Nov 01 21:00:05.481337 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.481133 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ae0b2c8-270b-4a3f-bb16-523874681ade-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ae0b2c8-270b-4a3f-bb16-523874681ade-istio-data\") pod \"3ae0b2c8-270b-4a3f-bb16-523874681ade\" (UID: \"3ae0b2c8-270b-4a3f-bb16-523874681ade\") " Nov 01 21:00:05.481337 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.481222 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b373faf1-ffb2-4a45-b433-0669635a397c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b373faf1-ffb2-4a45-b433-0669635a397c-workload-certs\") pod \"b373faf1-ffb2-4a45-b433-0669635a397c\" (UID: \"b373faf1-ffb2-4a45-b433-0669635a397c\") " Nov 01 21:00:05.481337 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.481273 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b373faf1-ffb2-4a45-b433-0669635a397c-kube-api-access-w9q4n\" (UniqueName: \"kubernetes.io/projected/b373faf1-ffb2-4a45-b433-0669635a397c-kube-api-access-w9q4n\") pod \"b373faf1-ffb2-4a45-b433-0669635a397c\" (UID: \"b373faf1-ffb2-4a45-b433-0669635a397c\") " Nov 01 21:00:05.482280 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.481366 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ae0b2c8-270b-4a3f-bb16-523874681ade-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ae0b2c8-270b-4a3f-bb16-523874681ade-istiod-ca-cert\") pod \"3ae0b2c8-270b-4a3f-bb16-523874681ade\" (UID: \"3ae0b2c8-270b-4a3f-bb16-523874681ade\") " Nov 01 21:00:05.482280 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.481424 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b373faf1-ffb2-4a45-b433-0669635a397c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b373faf1-ffb2-4a45-b433-0669635a397c-istio-envoy\") pod \"b373faf1-ffb2-4a45-b433-0669635a397c\" (UID: \"b373faf1-ffb2-4a45-b433-0669635a397c\") " Nov 01 21:00:05.482280 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.481523 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ae0b2c8-270b-4a3f-bb16-523874681ade-istio-token\" (UniqueName: \"kubernetes.io/projected/3ae0b2c8-270b-4a3f-bb16-523874681ade-istio-token\") pod \"3ae0b2c8-270b-4a3f-bb16-523874681ade\" (UID: \"3ae0b2c8-270b-4a3f-bb16-523874681ade\") " Nov 01 21:00:05.483947 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.483919 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b373faf1-ffb2-4a45-b433-0669635a397c-workload-certs" pod "b373faf1-ffb2-4a45-b433-0669635a397c" (UID: "b373faf1-ffb2-4a45-b433-0669635a397c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:00:05.484873 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.484531 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ae0b2c8-270b-4a3f-bb16-523874681ade-istio-data" pod "3ae0b2c8-270b-4a3f-bb16-523874681ade" (UID: "3ae0b2c8-270b-4a3f-bb16-523874681ade"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:00:05.484873 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.484824 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ae0b2c8-270b-4a3f-bb16-523874681ade-istiod-ca-cert" pod "3ae0b2c8-270b-4a3f-bb16-523874681ade" (UID: "3ae0b2c8-270b-4a3f-bb16-523874681ade"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:00:05.487255 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.487238 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ae0b2c8-270b-4a3f-bb16-523874681ade-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:05.487362 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.487352 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b373faf1-ffb2-4a45-b433-0669635a397c-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:05.487424 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.487416 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ae0b2c8-270b-4a3f-bb16-523874681ade-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:05.487481 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.487472 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ae0b2c8-270b-4a3f-bb16-523874681ade-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:05.487614 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.487603 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b373faf1-ffb2-4a45-b433-0669635a397c-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:05.490552 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.490534 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b373faf1-ffb2-4a45-b433-0669635a397c-istio-data" pod "b373faf1-ffb2-4a45-b433-0669635a397c" (UID: "b373faf1-ffb2-4a45-b433-0669635a397c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:00:05.490861 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.490847 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b373faf1-ffb2-4a45-b433-0669635a397c-istiod-ca-cert" pod "b373faf1-ffb2-4a45-b433-0669635a397c" (UID: "b373faf1-ffb2-4a45-b433-0669635a397c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:00:05.491137 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.491122 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ae0b2c8-270b-4a3f-bb16-523874681ade-workload-certs" pod "3ae0b2c8-270b-4a3f-bb16-523874681ade" (UID: "3ae0b2c8-270b-4a3f-bb16-523874681ade"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:00:05.491348 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.491336 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b373faf1-ffb2-4a45-b433-0669635a397c-workload-socket" pod "b373faf1-ffb2-4a45-b433-0669635a397c" (UID: "b373faf1-ffb2-4a45-b433-0669635a397c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:00:05.491565 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.491553 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ae0b2c8-270b-4a3f-bb16-523874681ade-workload-socket" pod "3ae0b2c8-270b-4a3f-bb16-523874681ade" (UID: "3ae0b2c8-270b-4a3f-bb16-523874681ade"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:00:05.500591 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.500269 4484 scope.go:122] "RemoveContainer" containerID="8eb9816d4eca1ec89f3adadcb08ce4caaa9cc08431c2dbe91e021610b5ae2efb" Nov 01 21:00:05.500849 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:00:05.500817 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eb9816d4eca1ec89f3adadcb08ce4caaa9cc08431c2dbe91e021610b5ae2efb\": container with ID starting with 8eb9816d4eca1ec89f3adadcb08ce4caaa9cc08431c2dbe91e021610b5ae2efb not found: ID does not exist" containerID="8eb9816d4eca1ec89f3adadcb08ce4caaa9cc08431c2dbe91e021610b5ae2efb" Nov 01 21:00:05.500966 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.500857 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eb9816d4eca1ec89f3adadcb08ce4caaa9cc08431c2dbe91e021610b5ae2efb"} err="failed to get container status \"8eb9816d4eca1ec89f3adadcb08ce4caaa9cc08431c2dbe91e021610b5ae2efb\": rpc error: code = NotFound desc = could not find container \"8eb9816d4eca1ec89f3adadcb08ce4caaa9cc08431c2dbe91e021610b5ae2efb\": container with ID starting with 8eb9816d4eca1ec89f3adadcb08ce4caaa9cc08431c2dbe91e021610b5ae2efb not found: ID does not exist" Nov 01 21:00:05.507614 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.507216 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b373faf1-ffb2-4a45-b433-0669635a397c-istio-envoy" pod "b373faf1-ffb2-4a45-b433-0669635a397c" (UID: "b373faf1-ffb2-4a45-b433-0669635a397c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:00:05.507614 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.507277 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ae0b2c8-270b-4a3f-bb16-523874681ade-istio-token" pod "3ae0b2c8-270b-4a3f-bb16-523874681ade" (UID: "3ae0b2c8-270b-4a3f-bb16-523874681ade"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:00:05.507614 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.507304 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b373faf1-ffb2-4a45-b433-0669635a397c-istio-podinfo" pod "b373faf1-ffb2-4a45-b433-0669635a397c" (UID: "b373faf1-ffb2-4a45-b433-0669635a397c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:00:05.507614 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.507359 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ae0b2c8-270b-4a3f-bb16-523874681ade-istio-podinfo" pod "3ae0b2c8-270b-4a3f-bb16-523874681ade" (UID: "3ae0b2c8-270b-4a3f-bb16-523874681ade"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:00:05.507614 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.507398 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b373faf1-ffb2-4a45-b433-0669635a397c-kube-api-access-w9q4n" pod "b373faf1-ffb2-4a45-b433-0669635a397c" (UID: "b373faf1-ffb2-4a45-b433-0669635a397c"). InnerVolumeSpecName "kube-api-access-w9q4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:00:05.507614 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.507431 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b373faf1-ffb2-4a45-b433-0669635a397c-istio-token" pod "b373faf1-ffb2-4a45-b433-0669635a397c" (UID: "b373faf1-ffb2-4a45-b433-0669635a397c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:00:05.508151 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.508109 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ae0b2c8-270b-4a3f-bb16-523874681ade-kube-api-access-c5dj8" pod "3ae0b2c8-270b-4a3f-bb16-523874681ade" (UID: "3ae0b2c8-270b-4a3f-bb16-523874681ade"). InnerVolumeSpecName "kube-api-access-c5dj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:00:05.508450 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.508270 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ae0b2c8-270b-4a3f-bb16-523874681ade-istio-envoy" pod "3ae0b2c8-270b-4a3f-bb16-523874681ade" (UID: "3ae0b2c8-270b-4a3f-bb16-523874681ade"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:00:05.589798 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.589756 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5dj8\" (UniqueName: \"kubernetes.io/projected/3ae0b2c8-270b-4a3f-bb16-523874681ade-kube-api-access-c5dj8\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:05.589798 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.589795 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b373faf1-ffb2-4a45-b433-0669635a397c-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:05.589798 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.589809 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9q4n\" (UniqueName: \"kubernetes.io/projected/b373faf1-ffb2-4a45-b433-0669635a397c-kube-api-access-w9q4n\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:05.590226 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.589821 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b373faf1-ffb2-4a45-b433-0669635a397c-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:05.590226 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.589834 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ae0b2c8-270b-4a3f-bb16-523874681ade-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:05.590226 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.589867 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b373faf1-ffb2-4a45-b433-0669635a397c-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:05.590226 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.589879 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ae0b2c8-270b-4a3f-bb16-523874681ade-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:05.590226 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.589893 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b373faf1-ffb2-4a45-b433-0669635a397c-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:05.590226 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.589906 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ae0b2c8-270b-4a3f-bb16-523874681ade-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:05.590226 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.589918 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ae0b2c8-270b-4a3f-bb16-523874681ade-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:05.590226 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.589930 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b373faf1-ffb2-4a45-b433-0669635a397c-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:05.590226 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.589943 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b373faf1-ffb2-4a45-b433-0669635a397c-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:05.590226 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.589957 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ae0b2c8-270b-4a3f-bb16-523874681ade-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:05.966704 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.966612 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h" Nov 01 21:00:05.973796 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:05.973766 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h" Nov 01 21:00:06.233161 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.233082 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw" Nov 01 21:00:06.257764 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.257702 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:06.392839 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.392799 4484 generic.go:356] "Generic (PLEG): container finished" podID="836acdb1-4e39-472f-ba72-a07d61f977bf" containerID="2bd1c28a8eb5bb55790b360ae048808b234637ab065e345fb7be2188b90906b1" exitCode=0 Nov 01 21:00:06.393086 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.392884 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl" event={"ID":"836acdb1-4e39-472f-ba72-a07d61f977bf","Type":"ContainerDied","Data":"2bd1c28a8eb5bb55790b360ae048808b234637ab065e345fb7be2188b90906b1"} Nov 01 21:00:06.393496 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.393478 4484 scope.go:122] "RemoveContainer" containerID="2bd1c28a8eb5bb55790b360ae048808b234637ab065e345fb7be2188b90906b1" Nov 01 21:00:06.395281 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.395260 4484 generic.go:356] "Generic (PLEG): container finished" podID="ce36bd6e-c550-4cd7-8f34-97b71968e4a2" containerID="cc5207f27244b099a2a26e328bbbe6c463f52f9fe5dd7491a9f72322f648daa1" exitCode=60 Nov 01 21:00:06.395392 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.395362 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892780-vdj2s" event={"ID":"ce36bd6e-c550-4cd7-8f34-97b71968e4a2","Type":"ContainerDied","Data":"cc5207f27244b099a2a26e328bbbe6c463f52f9fe5dd7491a9f72322f648daa1"} Nov 01 21:00:06.395462 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.395416 4484 scope.go:122] "RemoveContainer" containerID="958e5bda1df1d6bca428165b7559663e500a179c74a4950112a79f1fbd041cea" Nov 01 21:00:06.395577 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.395543 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:00:06.395672 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.395620 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:00:06.395672 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.395652 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:00:06.395861 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.395844 4484 scope.go:122] "RemoveContainer" containerID="cc5207f27244b099a2a26e328bbbe6c463f52f9fe5dd7491a9f72322f648daa1" Nov 01 21:00:06.396343 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:00:06.396309 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-cd248f-29892780-vdj2s_odh-ai-gateway-infra(ce36bd6e-c550-4cd7-8f34-97b71968e4a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892780-vdj2s" podUID="ce36bd6e-c550-4cd7-8f34-97b71968e4a2" Nov 01 21:00:06.399879 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.399859 4484 generic.go:356] "Generic (PLEG): container finished" podID="da6f250e-6516-4a04-a37a-f30c02ec682b" containerID="8b4816f8698560a1753b6a3691d097b27016c4ac7012b7ecd68d5948de464d2d" exitCode=60 Nov 01 21:00:06.399986 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.399969 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892780-7pl4v" event={"ID":"da6f250e-6516-4a04-a37a-f30c02ec682b","Type":"ContainerDied","Data":"8b4816f8698560a1753b6a3691d097b27016c4ac7012b7ecd68d5948de464d2d"} Nov 01 21:00:06.400281 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.400266 4484 scope.go:122] "RemoveContainer" containerID="8b4816f8698560a1753b6a3691d097b27016c4ac7012b7ecd68d5948de464d2d" Nov 01 21:00:06.400591 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:00:06.400570 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892780-7pl4v_odh-ai-gateway-infra(da6f250e-6516-4a04-a37a-f30c02ec682b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892780-7pl4v" podUID="da6f250e-6516-4a04-a37a-f30c02ec682b" Nov 01 21:00:06.402213 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.402183 4484 generic.go:356] "Generic (PLEG): container finished" podID="cfd728c4-7f6b-4197-a147-1f340427f414" containerID="86c922ecb57d854e00f28025a38f999523a4bbda0672f1e73b4eec919d35792a" exitCode=60 Nov 01 21:00:06.402315 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.402265 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892780-96d26" event={"ID":"cfd728c4-7f6b-4197-a147-1f340427f414","Type":"ContainerDied","Data":"86c922ecb57d854e00f28025a38f999523a4bbda0672f1e73b4eec919d35792a"} Nov 01 21:00:06.402549 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.402531 4484 scope.go:122] "RemoveContainer" containerID="86c922ecb57d854e00f28025a38f999523a4bbda0672f1e73b4eec919d35792a" Nov 01 21:00:06.402851 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:00:06.402829 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892780-96d26_odh-ai-gateway-infra(cfd728c4-7f6b-4197-a147-1f340427f414)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892780-96d26" podUID="cfd728c4-7f6b-4197-a147-1f340427f414" Nov 01 21:00:06.403043 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.403017 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad03483e-106e-4de1-b896-d71bff38f81d-istio-token\" (UniqueName: \"kubernetes.io/projected/ad03483e-106e-4de1-b896-d71bff38f81d-istio-token\") pod \"ad03483e-106e-4de1-b896-d71bff38f81d\" (UID: \"ad03483e-106e-4de1-b896-d71bff38f81d\") " Nov 01 21:00:06.403115 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.403073 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad03483e-106e-4de1-b896-d71bff38f81d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad03483e-106e-4de1-b896-d71bff38f81d-credential-socket\") pod \"ad03483e-106e-4de1-b896-d71bff38f81d\" (UID: \"ad03483e-106e-4de1-b896-d71bff38f81d\") " Nov 01 21:00:06.403558 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.403184 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad03483e-106e-4de1-b896-d71bff38f81d-kube-api-access-8fgvd\" (UniqueName: \"kubernetes.io/projected/ad03483e-106e-4de1-b896-d71bff38f81d-kube-api-access-8fgvd\") pod \"ad03483e-106e-4de1-b896-d71bff38f81d\" (UID: \"ad03483e-106e-4de1-b896-d71bff38f81d\") " Nov 01 21:00:06.403558 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.403220 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad03483e-106e-4de1-b896-d71bff38f81d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad03483e-106e-4de1-b896-d71bff38f81d-workload-certs\") pod \"ad03483e-106e-4de1-b896-d71bff38f81d\" (UID: \"ad03483e-106e-4de1-b896-d71bff38f81d\") " Nov 01 21:00:06.403558 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.403274 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad03483e-106e-4de1-b896-d71bff38f81d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad03483e-106e-4de1-b896-d71bff38f81d-istiod-ca-cert\") pod \"ad03483e-106e-4de1-b896-d71bff38f81d\" (UID: \"ad03483e-106e-4de1-b896-d71bff38f81d\") " Nov 01 21:00:06.403558 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.403410 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad03483e-106e-4de1-b896-d71bff38f81d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad03483e-106e-4de1-b896-d71bff38f81d-istio-envoy\") pod \"ad03483e-106e-4de1-b896-d71bff38f81d\" (UID: \"ad03483e-106e-4de1-b896-d71bff38f81d\") " Nov 01 21:00:06.403558 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.403520 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad03483e-106e-4de1-b896-d71bff38f81d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad03483e-106e-4de1-b896-d71bff38f81d-istio-data\") pod \"ad03483e-106e-4de1-b896-d71bff38f81d\" (UID: \"ad03483e-106e-4de1-b896-d71bff38f81d\") " Nov 01 21:00:06.403558 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.403533 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad03483e-106e-4de1-b896-d71bff38f81d-workload-certs" pod "ad03483e-106e-4de1-b896-d71bff38f81d" (UID: "ad03483e-106e-4de1-b896-d71bff38f81d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:00:06.403958 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.403644 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad03483e-106e-4de1-b896-d71bff38f81d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad03483e-106e-4de1-b896-d71bff38f81d-workload-socket\") pod \"ad03483e-106e-4de1-b896-d71bff38f81d\" (UID: \"ad03483e-106e-4de1-b896-d71bff38f81d\") " Nov 01 21:00:06.403958 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.403672 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad03483e-106e-4de1-b896-d71bff38f81d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad03483e-106e-4de1-b896-d71bff38f81d-istio-podinfo\") pod \"ad03483e-106e-4de1-b896-d71bff38f81d\" (UID: \"ad03483e-106e-4de1-b896-d71bff38f81d\") " Nov 01 21:00:06.403958 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.403919 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad03483e-106e-4de1-b896-d71bff38f81d-istio-data" pod "ad03483e-106e-4de1-b896-d71bff38f81d" (UID: "ad03483e-106e-4de1-b896-d71bff38f81d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:00:06.404200 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.403710 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad03483e-106e-4de1-b896-d71bff38f81d-istiod-ca-cert" pod "ad03483e-106e-4de1-b896-d71bff38f81d" (UID: "ad03483e-106e-4de1-b896-d71bff38f81d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:00:06.404320 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.404295 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8" event={"ID":"b373faf1-ffb2-4a45-b433-0669635a397c","Type":"ContainerDied","Data":"df29c92cb8534650ae52a18d1ba0d842a9cb9aad1a175c332b06c54ae1f83ab1"} Nov 01 21:00:06.408232 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.404341 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad03483e-106e-4de1-b896-d71bff38f81d-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:06.408232 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.404364 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad03483e-106e-4de1-b896-d71bff38f81d-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:06.408232 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.404376 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad03483e-106e-4de1-b896-d71bff38f81d-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:06.408232 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.404404 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8" Nov 01 21:00:06.408232 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.404435 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad03483e-106e-4de1-b896-d71bff38f81d-credential-socket" pod "ad03483e-106e-4de1-b896-d71bff38f81d" (UID: "ad03483e-106e-4de1-b896-d71bff38f81d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:00:06.408232 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.405611 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad03483e-106e-4de1-b896-d71bff38f81d-kube-api-access-8fgvd" pod "ad03483e-106e-4de1-b896-d71bff38f81d" (UID: "ad03483e-106e-4de1-b896-d71bff38f81d"). InnerVolumeSpecName "kube-api-access-8fgvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:00:06.408232 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.406228 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad03483e-106e-4de1-b896-d71bff38f81d-istio-envoy" pod "ad03483e-106e-4de1-b896-d71bff38f81d" (UID: "ad03483e-106e-4de1-b896-d71bff38f81d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:00:06.408232 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.406597 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv" Nov 01 21:00:06.408232 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.406648 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv" event={"ID":"3ae0b2c8-270b-4a3f-bb16-523874681ade","Type":"ContainerDied","Data":"a38be239ccd76dd09887dc7e0567072b542401cbcfdbc0c73539e5b2d8dc76b3"} Nov 01 21:00:06.408232 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.406598 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad03483e-106e-4de1-b896-d71bff38f81d-istio-token" pod "ad03483e-106e-4de1-b896-d71bff38f81d" (UID: "ad03483e-106e-4de1-b896-d71bff38f81d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:00:06.408232 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.407288 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad03483e-106e-4de1-b896-d71bff38f81d-istio-podinfo" pod "ad03483e-106e-4de1-b896-d71bff38f81d" (UID: "ad03483e-106e-4de1-b896-d71bff38f81d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:00:06.409290 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.409264 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad03483e-106e-4de1-b896-d71bff38f81d-workload-socket" pod "ad03483e-106e-4de1-b896-d71bff38f81d" (UID: "ad03483e-106e-4de1-b896-d71bff38f81d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:00:06.409697 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.409675 4484 generic.go:356] "Generic (PLEG): container finished" podID="b19e6e75-0ef8-440d-8617-e5c71bfde0d2" containerID="664a1e35abd176532a1bebe3892ba5deea20919b1b062d25762c87dab4d27c8c" exitCode=60 Nov 01 21:00:06.409784 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.409740 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd038b-29892780-vltsq" event={"ID":"b19e6e75-0ef8-440d-8617-e5c71bfde0d2","Type":"ContainerDied","Data":"664a1e35abd176532a1bebe3892ba5deea20919b1b062d25762c87dab4d27c8c"} Nov 01 21:00:06.411034 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.410157 4484 scope.go:122] "RemoveContainer" containerID="664a1e35abd176532a1bebe3892ba5deea20919b1b062d25762c87dab4d27c8c" Nov 01 21:00:06.411034 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:00:06.410517 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-dd038b-29892780-vltsq_odh-ai-gateway-infra(b19e6e75-0ef8-440d-8617-e5c71bfde0d2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd038b-29892780-vltsq" podUID="b19e6e75-0ef8-440d-8617-e5c71bfde0d2" Nov 01 21:00:06.415184 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.414339 4484 generic.go:356] "Generic (PLEG): container finished" podID="38c14ac7-4673-46b7-a721-652a972f3ba5" containerID="680415e314c9d523bb13494243c4ac1bf4d90a45770b71f5ffb8c4c2b6eb7735" exitCode=60 Nov 01 21:00:06.415184 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.414460 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892780-p6gfx" event={"ID":"38c14ac7-4673-46b7-a721-652a972f3ba5","Type":"ContainerDied","Data":"680415e314c9d523bb13494243c4ac1bf4d90a45770b71f5ffb8c4c2b6eb7735"} Nov 01 21:00:06.415184 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.414740 4484 scope.go:122] "RemoveContainer" containerID="680415e314c9d523bb13494243c4ac1bf4d90a45770b71f5ffb8c4c2b6eb7735" Nov 01 21:00:06.415184 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:00:06.415118 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892780-p6gfx_odh-ai-gateway-infra(38c14ac7-4673-46b7-a721-652a972f3ba5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892780-p6gfx" podUID="38c14ac7-4673-46b7-a721-652a972f3ba5" Nov 01 21:00:06.416307 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.416238 4484 generic.go:356] "Generic (PLEG): container finished" podID="ad03483e-106e-4de1-b896-d71bff38f81d" containerID="a6b8ef447c1a49b73cfe47caef054e5e93533fb23b177823a88f4f99cd89e50f" exitCode=0 Nov 01 21:00:06.416414 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.416340 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw" event={"ID":"ad03483e-106e-4de1-b896-d71bff38f81d","Type":"ContainerDied","Data":"a6b8ef447c1a49b73cfe47caef054e5e93533fb23b177823a88f4f99cd89e50f"} Nov 01 21:00:06.416414 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.416391 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw" event={"ID":"ad03483e-106e-4de1-b896-d71bff38f81d","Type":"ContainerDied","Data":"c7c067e5a73d17a5d51be8430355c02f9c2d546d577d745074a9e97a836492c9"} Nov 01 21:00:06.416545 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.416468 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw" Nov 01 21:00:06.419366 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.419282 4484 scope.go:122] "RemoveContainer" containerID="829dd57c2764e7e1d04b0a9dd0116884ba1f9df0e6a23bd748ef9af3df7efa4d" Nov 01 21:00:06.426783 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.426748 4484 generic.go:356] "Generic (PLEG): container finished" podID="e5aa9680-52ae-4173-875b-1097a5963fc0" containerID="78c31ce9df621c4320497c8e8b45517255aac51955e9116324f4c0d02644c553" exitCode=60 Nov 01 21:00:06.427056 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.426829 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892780-jmsct" event={"ID":"e5aa9680-52ae-4173-875b-1097a5963fc0","Type":"ContainerDied","Data":"78c31ce9df621c4320497c8e8b45517255aac51955e9116324f4c0d02644c553"} Nov 01 21:00:06.427428 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.427407 4484 scope.go:122] "RemoveContainer" containerID="78c31ce9df621c4320497c8e8b45517255aac51955e9116324f4c0d02644c553" Nov 01 21:00:06.427925 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:00:06.427905 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-944263-29892780-jmsct_odh-ai-gateway-infra(e5aa9680-52ae-4173-875b-1097a5963fc0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892780-jmsct" podUID="e5aa9680-52ae-4173-875b-1097a5963fc0" Nov 01 21:00:06.430365 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.430337 4484 generic.go:356] "Generic (PLEG): container finished" podID="7369891a-d9d2-42f1-8165-f817e9c00984" containerID="7e63bb45c74fef2b816cb0175b4280435c3110e0ad03786267ff37c6dcb73fe4" exitCode=60 Nov 01 21:00:06.430474 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.430428 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b27e66-29892780-hvgx4" event={"ID":"7369891a-d9d2-42f1-8165-f817e9c00984","Type":"ContainerDied","Data":"7e63bb45c74fef2b816cb0175b4280435c3110e0ad03786267ff37c6dcb73fe4"} Nov 01 21:00:06.430798 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.430777 4484 scope.go:122] "RemoveContainer" containerID="7e63bb45c74fef2b816cb0175b4280435c3110e0ad03786267ff37c6dcb73fe4" Nov 01 21:00:06.431241 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:00:06.431217 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-b27e66-29892780-hvgx4_odh-ai-gateway-infra(7369891a-d9d2-42f1-8165-f817e9c00984)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b27e66-29892780-hvgx4" podUID="7369891a-d9d2-42f1-8165-f817e9c00984" Nov 01 21:00:06.432852 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.432832 4484 scope.go:122] "RemoveContainer" containerID="9d0b372aec9fcdefa4b06701f0f453d152bba5f0ba272bf6dbcc18271befddcc" Nov 01 21:00:06.433258 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:00:06.433237 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892780-xxrwz_odh-ai-gateway-infra(b4578009-b3ff-4ce6-825a-8f92c1d3d72d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892780-xxrwz" podUID="b4578009-b3ff-4ce6-825a-8f92c1d3d72d" Nov 01 21:00:06.434344 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.434324 4484 generic.go:356] "Generic (PLEG): container finished" podID="4048751a-034d-4ab2-937f-6161505e4166" containerID="20c6f9f9c84adcb86aad2f2a31f2cfad160d309e9a25f92974e5c257bf5bfc04" exitCode=0 Nov 01 21:00:06.434437 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.434392 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822" event={"ID":"4048751a-034d-4ab2-937f-6161505e4166","Type":"ContainerDied","Data":"20c6f9f9c84adcb86aad2f2a31f2cfad160d309e9a25f92974e5c257bf5bfc04"} Nov 01 21:00:06.434776 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.434754 4484 scope.go:122] "RemoveContainer" containerID="20c6f9f9c84adcb86aad2f2a31f2cfad160d309e9a25f92974e5c257bf5bfc04" Nov 01 21:00:06.436265 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.436242 4484 generic.go:356] "Generic (PLEG): container finished" podID="d0f5aabc-58f2-4166-af2d-39aa67592cb1" containerID="764a3b1800e86a3495f68a14ecfda5256427b0ad2f3dae0f7028d7a07359b00d" exitCode=60 Nov 01 21:00:06.436367 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.436331 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892780-9tm2f" event={"ID":"d0f5aabc-58f2-4166-af2d-39aa67592cb1","Type":"ContainerDied","Data":"764a3b1800e86a3495f68a14ecfda5256427b0ad2f3dae0f7028d7a07359b00d"} Nov 01 21:00:06.436578 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.436561 4484 scope.go:122] "RemoveContainer" containerID="764a3b1800e86a3495f68a14ecfda5256427b0ad2f3dae0f7028d7a07359b00d" Nov 01 21:00:06.436657 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.436625 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:00:06.436741 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.436722 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:00:06.436800 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.436780 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:00:06.436912 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:00:06.436885 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892780-9tm2f_odh-ai-gateway-infra(d0f5aabc-58f2-4166-af2d-39aa67592cb1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892780-9tm2f" podUID="d0f5aabc-58f2-4166-af2d-39aa67592cb1" Nov 01 21:00:06.439695 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.439669 4484 generic.go:356] "Generic (PLEG): container finished" podID="52fb948e-15a6-400e-be44-2e88a1fb1501" containerID="edb13e2895e67dcae2d30e6e57e3d00990b2a5d7c8aefda510835bd73ffae129" exitCode=60 Nov 01 21:00:06.439807 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.439762 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892780-6xlxn" event={"ID":"52fb948e-15a6-400e-be44-2e88a1fb1501","Type":"ContainerDied","Data":"edb13e2895e67dcae2d30e6e57e3d00990b2a5d7c8aefda510835bd73ffae129"} Nov 01 21:00:06.440166 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.440150 4484 scope.go:122] "RemoveContainer" containerID="edb13e2895e67dcae2d30e6e57e3d00990b2a5d7c8aefda510835bd73ffae129" Nov 01 21:00:06.440444 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:00:06.440373 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892780-6xlxn_odh-ai-gateway-infra(52fb948e-15a6-400e-be44-2e88a1fb1501)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892780-6xlxn" podUID="52fb948e-15a6-400e-be44-2e88a1fb1501" Nov 01 21:00:06.442598 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.442570 4484 generic.go:356] "Generic (PLEG): container finished" podID="474d07b8-cbb2-4beb-a17f-fe445d0c9ad4" containerID="f5fcc30aa2fbf88fedd5d1850cfb015c3f61a92090dddf5ba5b52f0ae14a8d06" exitCode=60 Nov 01 21:00:06.442720 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.442663 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892780-q4ns9" event={"ID":"474d07b8-cbb2-4beb-a17f-fe445d0c9ad4","Type":"ContainerDied","Data":"f5fcc30aa2fbf88fedd5d1850cfb015c3f61a92090dddf5ba5b52f0ae14a8d06"} Nov 01 21:00:06.442894 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.442878 4484 scope.go:122] "RemoveContainer" containerID="f5fcc30aa2fbf88fedd5d1850cfb015c3f61a92090dddf5ba5b52f0ae14a8d06" Nov 01 21:00:06.442974 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.442930 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h" Nov 01 21:00:06.443225 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:00:06.443184 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892780-q4ns9_odh-ai-gateway-infra(474d07b8-cbb2-4beb-a17f-fe445d0c9ad4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892780-q4ns9" podUID="474d07b8-cbb2-4beb-a17f-fe445d0c9ad4" Nov 01 21:00:06.444120 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.444103 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h" Nov 01 21:00:06.484245 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.484118 4484 scope.go:122] "RemoveContainer" containerID="13416c43a31bd1cff128b3f3b7bbf17dd9f5dde99022669f311c2180007397e1" Nov 01 21:00:06.507338 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.507313 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fgvd\" (UniqueName: \"kubernetes.io/projected/ad03483e-106e-4de1-b896-d71bff38f81d-kube-api-access-8fgvd\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:06.507484 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.507340 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad03483e-106e-4de1-b896-d71bff38f81d-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:06.507484 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.507357 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad03483e-106e-4de1-b896-d71bff38f81d-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:06.507484 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.507366 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad03483e-106e-4de1-b896-d71bff38f81d-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:06.507484 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.507381 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad03483e-106e-4de1-b896-d71bff38f81d-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:06.507484 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.507393 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad03483e-106e-4de1-b896-d71bff38f81d-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:06.511931 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.511909 4484 scope.go:122] "RemoveContainer" containerID="b6541f7eefe7ba24ae375b42001c3d8d5be218cac65c3c82b0736e0ce422fffe" Nov 01 21:00:06.524616 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.524590 4484 scope.go:122] "RemoveContainer" containerID="acbed28624ebe42544acb96d055050fed39395495b39d480ce98a0590c69f879" Nov 01 21:00:06.545356 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.544365 4484 scope.go:122] "RemoveContainer" containerID="c0a0694d1b2c91637207732b6a2a5e10c7ce65fe1d9c4ec3b9113e63e478cfb1" Nov 01 21:00:06.569911 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.569884 4484 scope.go:122] "RemoveContainer" containerID="b30496e3a31d655f604ec8e4e435f097d4a96c1e730b19e6cfed76dc9565af42" Nov 01 21:00:06.592445 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.592421 4484 scope.go:122] "RemoveContainer" containerID="a6b8ef447c1a49b73cfe47caef054e5e93533fb23b177823a88f4f99cd89e50f" Nov 01 21:00:06.624531 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.624506 4484 scope.go:122] "RemoveContainer" containerID="a6b8ef447c1a49b73cfe47caef054e5e93533fb23b177823a88f4f99cd89e50f" Nov 01 21:00:06.625319 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:00:06.625290 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6b8ef447c1a49b73cfe47caef054e5e93533fb23b177823a88f4f99cd89e50f\": container with ID starting with a6b8ef447c1a49b73cfe47caef054e5e93533fb23b177823a88f4f99cd89e50f not found: ID does not exist" containerID="a6b8ef447c1a49b73cfe47caef054e5e93533fb23b177823a88f4f99cd89e50f" Nov 01 21:00:06.625442 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.625328 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6b8ef447c1a49b73cfe47caef054e5e93533fb23b177823a88f4f99cd89e50f"} err="failed to get container status \"a6b8ef447c1a49b73cfe47caef054e5e93533fb23b177823a88f4f99cd89e50f\": rpc error: code = NotFound desc = could not find container \"a6b8ef447c1a49b73cfe47caef054e5e93533fb23b177823a88f4f99cd89e50f\": container with ID starting with a6b8ef447c1a49b73cfe47caef054e5e93533fb23b177823a88f4f99cd89e50f not found: ID does not exist" Nov 01 21:00:06.625442 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.625345 4484 scope.go:122] "RemoveContainer" containerID="e3a2ba1937596fbcff9b024fcc902f11e8d67631f6ebe75e90ad74c5f22aee2c" Nov 01 21:00:06.642560 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.642537 4484 scope.go:122] "RemoveContainer" containerID="37357371e7432de3890d6db5c772d01bd721bfafcc1843ad57ba231db1d9595f" Nov 01 21:00:06.661285 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.661148 4484 scope.go:122] "RemoveContainer" containerID="67b7695b29080b35d23064ab804827c35a1f9fc02fe23f336c5d043e07498617" Nov 01 21:00:06.681197 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.681174 4484 scope.go:122] "RemoveContainer" containerID="64fb35033e756f653fa8e0ea2202f417fc2b32a45a437ca053b5ca5219060891" Nov 01 21:00:06.698729 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.698704 4484 scope.go:122] "RemoveContainer" containerID="52d1451717da5c8fbaf84e34ee3954b18ded07be6a4653adbe68baedb21848d2" Nov 01 21:00:06.790478 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.790444 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf"] Nov 01 21:00:06.798225 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:06.798187 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-lbcsf"] Nov 01 21:00:07.117076 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:07.116693 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:07.189021 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:07.172846 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h" podStartSLOduration=5.1728212 podStartE2EDuration="5.1728212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:00:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:00:07.041574153 +0000 UTC m=+1757.819214322" watchObservedRunningTime="2026-11-01 21:00:07.1728212 +0000 UTC m=+1757.950461337" Nov 01 21:00:07.383677 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:07.383601 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef2f968c-433d-49f8-8fc4-7c8551d647fb" path="/var/lib/kubelet/pods/ef2f968c-433d-49f8-8fc4-7c8551d647fb/volumes" Nov 01 21:00:07.472311 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:07.472280 4484 scope.go:122] "RemoveContainer" containerID="664a1e35abd176532a1bebe3892ba5deea20919b1b062d25762c87dab4d27c8c" Nov 01 21:00:07.472824 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:00:07.472796 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-dd038b-29892780-vltsq_odh-ai-gateway-infra(b19e6e75-0ef8-440d-8617-e5c71bfde0d2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd038b-29892780-vltsq" podUID="b19e6e75-0ef8-440d-8617-e5c71bfde0d2" Nov 01 21:00:07.473714 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:07.473685 4484 scope.go:122] "RemoveContainer" containerID="680415e314c9d523bb13494243c4ac1bf4d90a45770b71f5ffb8c4c2b6eb7735" Nov 01 21:00:07.474053 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:00:07.474031 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892780-p6gfx_odh-ai-gateway-infra(38c14ac7-4673-46b7-a721-652a972f3ba5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892780-p6gfx" podUID="38c14ac7-4673-46b7-a721-652a972f3ba5" Nov 01 21:00:07.476299 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:07.476276 4484 scope.go:122] "RemoveContainer" containerID="78c31ce9df621c4320497c8e8b45517255aac51955e9116324f4c0d02644c553" Nov 01 21:00:07.476664 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:00:07.476640 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-944263-29892780-jmsct_odh-ai-gateway-infra(e5aa9680-52ae-4173-875b-1097a5963fc0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892780-jmsct" podUID="e5aa9680-52ae-4173-875b-1097a5963fc0" Nov 01 21:00:07.477936 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:07.477915 4484 scope.go:122] "RemoveContainer" containerID="7e63bb45c74fef2b816cb0175b4280435c3110e0ad03786267ff37c6dcb73fe4" Nov 01 21:00:07.478230 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:00:07.478208 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-b27e66-29892780-hvgx4_odh-ai-gateway-infra(7369891a-d9d2-42f1-8165-f817e9c00984)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b27e66-29892780-hvgx4" podUID="7369891a-d9d2-42f1-8165-f817e9c00984" Nov 01 21:00:07.479599 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:07.479576 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822" event={"ID":"4048751a-034d-4ab2-937f-6161505e4166","Type":"ContainerStarted","Data":"429a4ca14aa787dd0f7b09db887feee8c5d69c7e08a2f527007420308f3cd47e"} Nov 01 21:00:07.481789 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:07.481772 4484 scope.go:122] "RemoveContainer" containerID="764a3b1800e86a3495f68a14ecfda5256427b0ad2f3dae0f7028d7a07359b00d" Nov 01 21:00:07.482049 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:00:07.482029 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892780-9tm2f_odh-ai-gateway-infra(d0f5aabc-58f2-4166-af2d-39aa67592cb1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892780-9tm2f" podUID="d0f5aabc-58f2-4166-af2d-39aa67592cb1" Nov 01 21:00:07.483770 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:07.483747 4484 scope.go:122] "RemoveContainer" containerID="edb13e2895e67dcae2d30e6e57e3d00990b2a5d7c8aefda510835bd73ffae129" Nov 01 21:00:07.484146 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:00:07.484119 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892780-6xlxn_odh-ai-gateway-infra(52fb948e-15a6-400e-be44-2e88a1fb1501)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892780-6xlxn" podUID="52fb948e-15a6-400e-be44-2e88a1fb1501" Nov 01 21:00:07.486265 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:07.485907 4484 scope.go:122] "RemoveContainer" containerID="f5fcc30aa2fbf88fedd5d1850cfb015c3f61a92090dddf5ba5b52f0ae14a8d06" Nov 01 21:00:07.486265 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:00:07.486208 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892780-q4ns9_odh-ai-gateway-infra(474d07b8-cbb2-4beb-a17f-fe445d0c9ad4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892780-q4ns9" podUID="474d07b8-cbb2-4beb-a17f-fe445d0c9ad4" Nov 01 21:00:07.488396 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:07.488372 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl" event={"ID":"836acdb1-4e39-472f-ba72-a07d61f977bf","Type":"ContainerStarted","Data":"d48d997a5bf4c9ad97bff1333c606b77d823d57bc7069382150aa38f099e22d7"} Nov 01 21:00:07.490326 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:07.490307 4484 scope.go:122] "RemoveContainer" containerID="cc5207f27244b099a2a26e328bbbe6c463f52f9fe5dd7491a9f72322f648daa1" Nov 01 21:00:07.490539 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:00:07.490513 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-cd248f-29892780-vdj2s_odh-ai-gateway-infra(ce36bd6e-c550-4cd7-8f34-97b71968e4a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892780-vdj2s" podUID="ce36bd6e-c550-4cd7-8f34-97b71968e4a2" Nov 01 21:00:07.491781 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:07.491762 4484 scope.go:122] "RemoveContainer" containerID="8b4816f8698560a1753b6a3691d097b27016c4ac7012b7ecd68d5948de464d2d" Nov 01 21:00:07.491962 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:00:07.491945 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892780-7pl4v_odh-ai-gateway-infra(da6f250e-6516-4a04-a37a-f30c02ec682b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892780-7pl4v" podUID="da6f250e-6516-4a04-a37a-f30c02ec682b" Nov 01 21:00:07.494405 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:07.494387 4484 scope.go:122] "RemoveContainer" containerID="86c922ecb57d854e00f28025a38f999523a4bbda0672f1e73b4eec919d35792a" Nov 01 21:00:07.494592 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:00:07.494574 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892780-96d26_odh-ai-gateway-infra(cfd728c4-7f6b-4197-a147-1f340427f414)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892780-96d26" podUID="cfd728c4-7f6b-4197-a147-1f340427f414" Nov 01 21:00:07.533280 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:07.530866 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822" Nov 01 21:00:07.537019 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:07.535134 4484 patch_prober.go:29] interesting pod/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Nov 01 21:00:07.537019 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:07.535203 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822" podUID="4048751a-034d-4ab2-937f-6161505e4166" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Nov 01 21:00:07.666629 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:07.666524 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv"] Nov 01 21:00:07.673587 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:07.673478 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-tsxlv"] Nov 01 21:00:08.052590 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:08.052543 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl" Nov 01 21:00:08.053713 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:08.053691 4484 patch_prober.go:29] interesting pod/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Nov 01 21:00:08.053918 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:08.053895 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl" podUID="836acdb1-4e39-472f-ba72-a07d61f977bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Nov 01 21:00:08.542195 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:08.542144 4484 patch_prober.go:29] interesting pod/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Nov 01 21:00:08.542867 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:08.542828 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822" podUID="4048751a-034d-4ab2-937f-6161505e4166" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Nov 01 21:00:08.560041 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:08.559777 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8"] Nov 01 21:00:08.564211 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:08.564176 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-cznb8"] Nov 01 21:00:09.037493 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:09.037454 4484 patch_prober.go:29] interesting pod/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Nov 01 21:00:09.037818 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:09.037784 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl" podUID="836acdb1-4e39-472f-ba72-a07d61f977bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Nov 01 21:00:09.383725 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:09.383631 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ae0b2c8-270b-4a3f-bb16-523874681ade" path="/var/lib/kubelet/pods/3ae0b2c8-270b-4a3f-bb16-523874681ade/volumes" Nov 01 21:00:09.384339 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:09.384316 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b373faf1-ffb2-4a45-b433-0669635a397c" path="/var/lib/kubelet/pods/b373faf1-ffb2-4a45-b433-0669635a397c/volumes" Nov 01 21:00:09.386436 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:09.384804 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw"] Nov 01 21:00:09.386436 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:09.384844 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-tb6nw"] Nov 01 21:00:09.554191 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:09.551375 4484 patch_prober.go:29] interesting pod/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Nov 01 21:00:09.554191 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:09.551439 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822" podUID="4048751a-034d-4ab2-937f-6161505e4166" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Nov 01 21:00:09.934021 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:09.932599 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:10.038023 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:10.037395 4484 patch_prober.go:29] interesting pod/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Nov 01 21:00:10.038023 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:10.037454 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl" podUID="836acdb1-4e39-472f-ba72-a07d61f977bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Nov 01 21:00:10.162041 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:10.160586 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:10.376391 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:10.376344 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:10.539021 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:10.526241 4484 patch_prober.go:29] interesting pod/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Nov 01 21:00:10.539021 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:10.526316 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822" podUID="4048751a-034d-4ab2-937f-6161505e4166" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Nov 01 21:00:11.045508 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:11.045032 4484 patch_prober.go:29] interesting pod/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Nov 01 21:00:11.045508 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:11.045101 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl" podUID="836acdb1-4e39-472f-ba72-a07d61f977bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Nov 01 21:00:11.378246 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:11.378055 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad03483e-106e-4de1-b896-d71bff38f81d" path="/var/lib/kubelet/pods/ad03483e-106e-4de1-b896-d71bff38f81d/volumes" Nov 01 21:00:11.466494 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:11.466446 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl" Nov 01 21:00:11.523021 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:11.519415 4484 patch_prober.go:29] interesting pod/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Nov 01 21:00:11.523021 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:11.519498 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822" podUID="4048751a-034d-4ab2-937f-6161505e4166" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Nov 01 21:00:11.714377 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:11.714033 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:12.054025 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:12.051511 4484 patch_prober.go:29] interesting pod/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Nov 01 21:00:12.054025 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:12.051583 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl" podUID="836acdb1-4e39-472f-ba72-a07d61f977bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Nov 01 21:00:12.524020 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:12.523392 4484 patch_prober.go:29] interesting pod/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Nov 01 21:00:12.524020 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:12.523454 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822" podUID="4048751a-034d-4ab2-937f-6161505e4166" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Nov 01 21:00:13.038070 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:13.038034 4484 patch_prober.go:29] interesting pod/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Nov 01 21:00:13.038298 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:13.038087 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl" podUID="836acdb1-4e39-472f-ba72-a07d61f977bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Nov 01 21:00:13.488742 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:13.488706 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822" Nov 01 21:00:13.521744 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:13.521715 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822" Nov 01 21:00:13.605678 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:13.605648 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822" Nov 01 21:00:14.042839 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:14.042808 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl" Nov 01 21:00:14.044097 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:14.044075 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl" Nov 01 21:00:15.374246 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:15.374221 4484 scope.go:122] "RemoveContainer" containerID="9d0b372aec9fcdefa4b06701f0f453d152bba5f0ba272bf6dbcc18271befddcc" Nov 01 21:00:15.805173 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:00:15.805134 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:00:15.946403 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:15.946355 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:16.118338 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:16.118221 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:16.364941 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:16.364914 4484 scope.go:122] "RemoveContainer" containerID="cc5207f27244b099a2a26e328bbbe6c463f52f9fe5dd7491a9f72322f648daa1" Nov 01 21:00:16.631541 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:16.631504 4484 generic.go:356] "Generic (PLEG): container finished" podID="b4578009-b3ff-4ce6-825a-8f92c1d3d72d" containerID="f5581bdf4ccd0c61f1a9431d34650bdd54266a28db48175fc391f451c306736c" exitCode=60 Nov 01 21:00:16.632091 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:16.631577 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892780-xxrwz" event={"ID":"b4578009-b3ff-4ce6-825a-8f92c1d3d72d","Type":"ContainerDied","Data":"f5581bdf4ccd0c61f1a9431d34650bdd54266a28db48175fc391f451c306736c"} Nov 01 21:00:16.632091 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:16.631659 4484 scope.go:122] "RemoveContainer" containerID="9d0b372aec9fcdefa4b06701f0f453d152bba5f0ba272bf6dbcc18271befddcc" Nov 01 21:00:16.632225 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:16.632204 4484 scope.go:122] "RemoveContainer" containerID="f5581bdf4ccd0c61f1a9431d34650bdd54266a28db48175fc391f451c306736c" Nov 01 21:00:16.632713 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:00:16.632605 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892780-xxrwz_odh-ai-gateway-infra(b4578009-b3ff-4ce6-825a-8f92c1d3d72d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892780-xxrwz" podUID="b4578009-b3ff-4ce6-825a-8f92c1d3d72d" Nov 01 21:00:16.974106 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:16.974018 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:17.147019 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:17.139755 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:17.471041 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:17.469696 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:17.474254 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:17.474229 4484 scope.go:122] "RemoveContainer" containerID="664a1e35abd176532a1bebe3892ba5deea20919b1b062d25762c87dab4d27c8c" Nov 01 21:00:17.487500 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:17.478094 4484 scope.go:122] "RemoveContainer" containerID="680415e314c9d523bb13494243c4ac1bf4d90a45770b71f5ffb8c4c2b6eb7735" Nov 01 21:00:17.487500 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:17.478394 4484 scope.go:122] "RemoveContainer" containerID="86c922ecb57d854e00f28025a38f999523a4bbda0672f1e73b4eec919d35792a" Nov 01 21:00:17.665049 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:17.658818 4484 generic.go:356] "Generic (PLEG): container finished" podID="ce36bd6e-c550-4cd7-8f34-97b71968e4a2" containerID="3b01f7630192c53ae6ec52ef39065272f71f4374eefd958a93e255415c987a66" exitCode=60 Nov 01 21:00:17.665049 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:17.658867 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892780-vdj2s" event={"ID":"ce36bd6e-c550-4cd7-8f34-97b71968e4a2","Type":"ContainerDied","Data":"3b01f7630192c53ae6ec52ef39065272f71f4374eefd958a93e255415c987a66"} Nov 01 21:00:17.665049 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:17.658921 4484 scope.go:122] "RemoveContainer" containerID="cc5207f27244b099a2a26e328bbbe6c463f52f9fe5dd7491a9f72322f648daa1" Nov 01 21:00:17.665049 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:17.659445 4484 scope.go:122] "RemoveContainer" containerID="3b01f7630192c53ae6ec52ef39065272f71f4374eefd958a93e255415c987a66" Nov 01 21:00:17.665049 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:00:17.659868 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-cd248f-29892780-vdj2s_odh-ai-gateway-infra(ce36bd6e-c550-4cd7-8f34-97b71968e4a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892780-vdj2s" podUID="ce36bd6e-c550-4cd7-8f34-97b71968e4a2" Nov 01 21:00:17.768462 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:17.768428 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892780-xxrwz"] Nov 01 21:00:18.212015 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.210873 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq"] Nov 01 21:00:18.215848 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.212927 4484 cpu_manager.go:414] "containerMap: removing container" podUID="b373faf1-ffb2-4a45-b433-0669635a397c" containerName="istio-proxy" Nov 01 21:00:18.215848 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.212954 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b373faf1-ffb2-4a45-b433-0669635a397c" containerName="istio-proxy" Nov 01 21:00:18.215848 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.212980 4484 cpu_manager.go:414] "containerMap: removing container" podUID="3ae0b2c8-270b-4a3f-bb16-523874681ade" containerName="istio-proxy" Nov 01 21:00:18.215848 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.212988 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ae0b2c8-270b-4a3f-bb16-523874681ade" containerName="istio-proxy" Nov 01 21:00:18.215848 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.213059 4484 cpu_manager.go:414] "containerMap: removing container" podUID="ef2f968c-433d-49f8-8fc4-7c8551d647fb" containerName="istio-proxy" Nov 01 21:00:18.215848 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.213068 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef2f968c-433d-49f8-8fc4-7c8551d647fb" containerName="istio-proxy" Nov 01 21:00:18.215848 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.213114 4484 cpu_manager.go:414] "containerMap: removing container" podUID="ad03483e-106e-4de1-b896-d71bff38f81d" containerName="istio-proxy" Nov 01 21:00:18.215848 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.213123 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad03483e-106e-4de1-b896-d71bff38f81d" containerName="istio-proxy" Nov 01 21:00:18.215848 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.213179 4484 cpu_manager.go:414] "containerMap: removing container" podUID="3ae0b2c8-270b-4a3f-bb16-523874681ade" containerName="istio-proxy" Nov 01 21:00:18.215848 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.213185 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ae0b2c8-270b-4a3f-bb16-523874681ade" containerName="istio-proxy" Nov 01 21:00:18.215848 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.213583 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ae0b2c8-270b-4a3f-bb16-523874681ade" containerName="istio-proxy" Nov 01 21:00:18.215848 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.213616 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="b373faf1-ffb2-4a45-b433-0669635a397c" containerName="istio-proxy" Nov 01 21:00:18.215848 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.213641 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef2f968c-433d-49f8-8fc4-7c8551d647fb" containerName="istio-proxy" Nov 01 21:00:18.215848 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.213670 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad03483e-106e-4de1-b896-d71bff38f81d" containerName="istio-proxy" Nov 01 21:00:18.229023 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.227308 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq" Nov 01 21:00:18.231444 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.230822 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq"] Nov 01 21:00:18.233815 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.232962 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:18.284130 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.284060 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-dab736-openshift-default-dockercfg-c8cbv\"" Nov 01 21:00:18.362488 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.362387 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-istio-envoy\") pod \"e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq\" (UID: \"6c3c6fec-0841-4a0c-b761-626d8c53a1cb\") " pod="openshift-ingress/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq" Nov 01 21:00:18.366457 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.362856 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-istio-data\") pod \"e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq\" (UID: \"6c3c6fec-0841-4a0c-b761-626d8c53a1cb\") " pod="openshift-ingress/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq" Nov 01 21:00:18.370433 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.370404 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-istio-podinfo\") pod \"e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq\" (UID: \"6c3c6fec-0841-4a0c-b761-626d8c53a1cb\") " pod="openshift-ingress/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq" Nov 01 21:00:18.370732 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.370712 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-workload-certs\") pod \"e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq\" (UID: \"6c3c6fec-0841-4a0c-b761-626d8c53a1cb\") " pod="openshift-ingress/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq" Nov 01 21:00:18.370973 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.370956 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-workload-socket\") pod \"e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq\" (UID: \"6c3c6fec-0841-4a0c-b761-626d8c53a1cb\") " pod="openshift-ingress/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq" Nov 01 21:00:18.371165 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.371149 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-istiod-ca-cert\") pod \"e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq\" (UID: \"6c3c6fec-0841-4a0c-b761-626d8c53a1cb\") " pod="openshift-ingress/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq" Nov 01 21:00:18.371423 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.371309 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-istio-token\") pod \"e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq\" (UID: \"6c3c6fec-0841-4a0c-b761-626d8c53a1cb\") " pod="openshift-ingress/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq" Nov 01 21:00:18.371751 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.371701 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb8bm\" (UniqueName: \"kubernetes.io/projected/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-kube-api-access-xb8bm\") pod \"e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq\" (UID: \"6c3c6fec-0841-4a0c-b761-626d8c53a1cb\") " pod="openshift-ingress/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq" Nov 01 21:00:18.372170 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.372127 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-credential-socket\") pod \"e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq\" (UID: \"6c3c6fec-0841-4a0c-b761-626d8c53a1cb\") " pod="openshift-ingress/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq" Nov 01 21:00:18.372549 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.368299 4484 scope.go:122] "RemoveContainer" containerID="f5fcc30aa2fbf88fedd5d1850cfb015c3f61a92090dddf5ba5b52f0ae14a8d06" Nov 01 21:00:18.373525 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.373509 4484 scope.go:122] "RemoveContainer" containerID="edb13e2895e67dcae2d30e6e57e3d00990b2a5d7c8aefda510835bd73ffae129" Nov 01 21:00:18.374320 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.374305 4484 scope.go:122] "RemoveContainer" containerID="78c31ce9df621c4320497c8e8b45517255aac51955e9116324f4c0d02644c553" Nov 01 21:00:18.406200 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.400211 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892780-xxrwz" Nov 01 21:00:18.473770 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.473740 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4578009-b3ff-4ce6-825a-8f92c1d3d72d-kube-api-access-l59jl\" (UniqueName: \"kubernetes.io/projected/b4578009-b3ff-4ce6-825a-8f92c1d3d72d-kube-api-access-l59jl\") pod \"b4578009-b3ff-4ce6-825a-8f92c1d3d72d\" (UID: \"b4578009-b3ff-4ce6-825a-8f92c1d3d72d\") " Nov 01 21:00:18.473941 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.473898 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4578009-b3ff-4ce6-825a-8f92c1d3d72d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4578009-b3ff-4ce6-825a-8f92c1d3d72d-maas-api-service-ca\") pod \"b4578009-b3ff-4ce6-825a-8f92c1d3d72d\" (UID: \"b4578009-b3ff-4ce6-825a-8f92c1d3d72d\") " Nov 01 21:00:18.474288 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.474260 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4578009-b3ff-4ce6-825a-8f92c1d3d72d-maas-api-service-ca" pod "b4578009-b3ff-4ce6-825a-8f92c1d3d72d" (UID: "b4578009-b3ff-4ce6-825a-8f92c1d3d72d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:00:18.474288 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.474277 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-credential-socket\") pod \"e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq\" (UID: \"6c3c6fec-0841-4a0c-b761-626d8c53a1cb\") " pod="openshift-ingress/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq" Nov 01 21:00:18.474506 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.474336 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-istio-envoy\") pod \"e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq\" (UID: \"6c3c6fec-0841-4a0c-b761-626d8c53a1cb\") " pod="openshift-ingress/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq" Nov 01 21:00:18.474506 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.474468 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-istio-data\") pod \"e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq\" (UID: \"6c3c6fec-0841-4a0c-b761-626d8c53a1cb\") " pod="openshift-ingress/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq" Nov 01 21:00:18.474506 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.474503 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-istio-podinfo\") pod \"e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq\" (UID: \"6c3c6fec-0841-4a0c-b761-626d8c53a1cb\") " pod="openshift-ingress/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq" Nov 01 21:00:18.474672 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.474559 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-workload-certs\") pod \"e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq\" (UID: \"6c3c6fec-0841-4a0c-b761-626d8c53a1cb\") " pod="openshift-ingress/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq" Nov 01 21:00:18.474672 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.474588 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-credential-socket\") pod \"e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq\" (UID: \"6c3c6fec-0841-4a0c-b761-626d8c53a1cb\") " pod="openshift-ingress/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq" Nov 01 21:00:18.474795 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.474761 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-workload-socket\") pod \"e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq\" (UID: \"6c3c6fec-0841-4a0c-b761-626d8c53a1cb\") " pod="openshift-ingress/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq" Nov 01 21:00:18.474857 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.474798 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-istio-data\") pod \"e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq\" (UID: \"6c3c6fec-0841-4a0c-b761-626d8c53a1cb\") " pod="openshift-ingress/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq" Nov 01 21:00:18.474857 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.474827 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-istiod-ca-cert\") pod \"e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq\" (UID: \"6c3c6fec-0841-4a0c-b761-626d8c53a1cb\") " pod="openshift-ingress/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq" Nov 01 21:00:18.474974 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.474894 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-istio-token\") pod \"e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq\" (UID: \"6c3c6fec-0841-4a0c-b761-626d8c53a1cb\") " pod="openshift-ingress/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq" Nov 01 21:00:18.474974 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.474949 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-workload-certs\") pod \"e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq\" (UID: \"6c3c6fec-0841-4a0c-b761-626d8c53a1cb\") " pod="openshift-ingress/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq" Nov 01 21:00:18.475097 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.475035 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb8bm\" (UniqueName: \"kubernetes.io/projected/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-kube-api-access-xb8bm\") pod \"e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq\" (UID: \"6c3c6fec-0841-4a0c-b761-626d8c53a1cb\") " pod="openshift-ingress/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq" Nov 01 21:00:18.475183 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.475155 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-workload-socket\") pod \"e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq\" (UID: \"6c3c6fec-0841-4a0c-b761-626d8c53a1cb\") " pod="openshift-ingress/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq" Nov 01 21:00:18.475508 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.475484 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4578009-b3ff-4ce6-825a-8f92c1d3d72d-maas-api-service-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:18.475587 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.475524 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-istiod-ca-cert\") pod \"e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq\" (UID: \"6c3c6fec-0841-4a0c-b761-626d8c53a1cb\") " pod="openshift-ingress/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq" Nov 01 21:00:18.476421 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.476397 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4578009-b3ff-4ce6-825a-8f92c1d3d72d-kube-api-access-l59jl" pod "b4578009-b3ff-4ce6-825a-8f92c1d3d72d" (UID: "b4578009-b3ff-4ce6-825a-8f92c1d3d72d"). InnerVolumeSpecName "kube-api-access-l59jl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:00:18.477157 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.477137 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-istio-podinfo\") pod \"e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq\" (UID: \"6c3c6fec-0841-4a0c-b761-626d8c53a1cb\") " pod="openshift-ingress/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq" Nov 01 21:00:18.477279 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.477259 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-istio-envoy\") pod \"e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq\" (UID: \"6c3c6fec-0841-4a0c-b761-626d8c53a1cb\") " pod="openshift-ingress/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq" Nov 01 21:00:18.577042 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.576955 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l59jl\" (UniqueName: \"kubernetes.io/projected/b4578009-b3ff-4ce6-825a-8f92c1d3d72d-kube-api-access-l59jl\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:18.644959 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.644815 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb8bm\" (UniqueName: \"kubernetes.io/projected/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-kube-api-access-xb8bm\") pod \"e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq\" (UID: \"6c3c6fec-0841-4a0c-b761-626d8c53a1cb\") " pod="openshift-ingress/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq" Nov 01 21:00:18.671881 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.671846 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-istio-token\") pod \"e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq\" (UID: \"6c3c6fec-0841-4a0c-b761-626d8c53a1cb\") " pod="openshift-ingress/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq" Nov 01 21:00:18.699065 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.697784 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:18.715778 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.714468 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892780-96d26" event={"ID":"cfd728c4-7f6b-4197-a147-1f340427f414","Type":"ContainerStarted","Data":"106c40c76afdcd5459e2551a44fd14ea46114e7829c0a611c823f798446a9bed"} Nov 01 21:00:18.720357 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.720303 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd038b-29892780-vltsq" event={"ID":"b19e6e75-0ef8-440d-8617-e5c71bfde0d2","Type":"ContainerStarted","Data":"addbc52446ddbfa8465e17a232e1cba685b5197bc2e9b13cfc2d2fab8f7c95a9"} Nov 01 21:00:18.723942 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.723912 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892780-p6gfx" event={"ID":"38c14ac7-4673-46b7-a721-652a972f3ba5","Type":"ContainerStarted","Data":"d22b765898c651cb3947da46372fca09760dcfee60152d468a23a3a620687a37"} Nov 01 21:00:18.725746 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.725723 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892780-xxrwz" event={"ID":"b4578009-b3ff-4ce6-825a-8f92c1d3d72d","Type":"ContainerDied","Data":"c36206bb3bc72cef16a5b9234e92c84ab61800161e90e4ff0a14399a66674b38"} Nov 01 21:00:18.725746 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.725738 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892780-xxrwz" Nov 01 21:00:18.725910 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.725776 4484 scope.go:122] "RemoveContainer" containerID="f5581bdf4ccd0c61f1a9431d34650bdd54266a28db48175fc391f451c306736c" Nov 01 21:00:18.788284 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.788255 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892780-vdj2s"] Nov 01 21:00:18.837312 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.837257 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892780-96d26" podStartSLOduration=18.837241868 podStartE2EDuration="18.837241868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:00:18.835991301 +0000 UTC m=+1769.613631436" watchObservedRunningTime="2026-11-01 21:00:18.837241868 +0000 UTC m=+1769.614882003" Nov 01 21:00:18.945874 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.945843 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq" Nov 01 21:00:18.955158 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:18.955106 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd038b-29892780-vltsq" podStartSLOduration=18.955090875 podStartE2EDuration="18.955090875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:00:18.951796366 +0000 UTC m=+1769.729436502" watchObservedRunningTime="2026-11-01 21:00:18.955090875 +0000 UTC m=+1769.732731004" Nov 01 21:00:19.080185 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:19.080111 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892780-xxrwz"] Nov 01 21:00:19.082059 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:19.082031 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892780-vdj2s" Nov 01 21:00:19.087062 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:19.087034 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9ae21-29892780-xxrwz"] Nov 01 21:00:19.192767 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:19.192687 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce36bd6e-c550-4cd7-8f34-97b71968e4a2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce36bd6e-c550-4cd7-8f34-97b71968e4a2-maas-api-service-ca\") pod \"ce36bd6e-c550-4cd7-8f34-97b71968e4a2\" (UID: \"ce36bd6e-c550-4cd7-8f34-97b71968e4a2\") " Nov 01 21:00:19.192922 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:19.192890 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce36bd6e-c550-4cd7-8f34-97b71968e4a2-kube-api-access-nqbqr\" (UniqueName: \"kubernetes.io/projected/ce36bd6e-c550-4cd7-8f34-97b71968e4a2-kube-api-access-nqbqr\") pod \"ce36bd6e-c550-4cd7-8f34-97b71968e4a2\" (UID: \"ce36bd6e-c550-4cd7-8f34-97b71968e4a2\") " Nov 01 21:00:19.193060 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:19.193037 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce36bd6e-c550-4cd7-8f34-97b71968e4a2-maas-api-service-ca" pod "ce36bd6e-c550-4cd7-8f34-97b71968e4a2" (UID: "ce36bd6e-c550-4cd7-8f34-97b71968e4a2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:00:19.193493 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:19.193474 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce36bd6e-c550-4cd7-8f34-97b71968e4a2-maas-api-service-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:19.195377 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:19.195342 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce36bd6e-c550-4cd7-8f34-97b71968e4a2-kube-api-access-nqbqr" pod "ce36bd6e-c550-4cd7-8f34-97b71968e4a2" (UID: "ce36bd6e-c550-4cd7-8f34-97b71968e4a2"). InnerVolumeSpecName "kube-api-access-nqbqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:00:19.295556 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:19.295521 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqbqr\" (UniqueName: \"kubernetes.io/projected/ce36bd6e-c550-4cd7-8f34-97b71968e4a2-kube-api-access-nqbqr\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:19.364949 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:19.364918 4484 scope.go:122] "RemoveContainer" containerID="8b4816f8698560a1753b6a3691d097b27016c4ac7012b7ecd68d5948de464d2d" Nov 01 21:00:19.375242 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:19.375213 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4578009-b3ff-4ce6-825a-8f92c1d3d72d" path="/var/lib/kubelet/pods/b4578009-b3ff-4ce6-825a-8f92c1d3d72d/volumes" Nov 01 21:00:19.427726 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:19.427650 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq"] Nov 01 21:00:19.435461 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:00:19.435430 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c3c6fec_0841_4a0c_b761_626d8c53a1cb.slice/crio-d6f5356e786c5ba83b5215c9fcb613b3ff089d2b2912b975bb5a078a47ff6361 WatchSource:0}: Error finding container d6f5356e786c5ba83b5215c9fcb613b3ff089d2b2912b975bb5a078a47ff6361: Status 404 returned error can't find the container with id d6f5356e786c5ba83b5215c9fcb613b3ff089d2b2912b975bb5a078a47ff6361 Nov 01 21:00:19.438950 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:19.438913 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:00:19.439071 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:19.438987 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:00:19.439071 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:19.439045 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:00:19.472466 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:19.472435 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:19.693065 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:19.692101 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:19.915032 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:19.912237 4484 generic.go:356] "Generic (PLEG): container finished" podID="52fb948e-15a6-400e-be44-2e88a1fb1501" containerID="478f1708aa08fb0a88a6764b8d876364f76f5427d962cd5c773761703be593cf" exitCode=60 Nov 01 21:00:19.915032 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:19.912375 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892780-6xlxn" event={"ID":"52fb948e-15a6-400e-be44-2e88a1fb1501","Type":"ContainerDied","Data":"478f1708aa08fb0a88a6764b8d876364f76f5427d962cd5c773761703be593cf"} Nov 01 21:00:19.915032 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:19.912451 4484 scope.go:122] "RemoveContainer" containerID="edb13e2895e67dcae2d30e6e57e3d00990b2a5d7c8aefda510835bd73ffae129" Nov 01 21:00:19.915032 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:19.913205 4484 scope.go:122] "RemoveContainer" containerID="478f1708aa08fb0a88a6764b8d876364f76f5427d962cd5c773761703be593cf" Nov 01 21:00:19.915032 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:00:19.913786 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29892780-6xlxn_odh-ai-gateway-infra(52fb948e-15a6-400e-be44-2e88a1fb1501)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892780-6xlxn" podUID="52fb948e-15a6-400e-be44-2e88a1fb1501" Nov 01 21:00:19.941022 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:19.923133 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892780-96d26"] Nov 01 21:00:19.941022 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:19.923216 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-29baa8-6c565db695-5m42b"] Nov 01 21:00:19.941022 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:19.924762 4484 cpu_manager.go:414] "containerMap: removing container" podUID="ce36bd6e-c550-4cd7-8f34-97b71968e4a2" containerName="cleanup" Nov 01 21:00:19.941022 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:19.924786 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce36bd6e-c550-4cd7-8f34-97b71968e4a2" containerName="cleanup" Nov 01 21:00:19.941022 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:19.924814 4484 cpu_manager.go:414] "containerMap: removing container" podUID="ce36bd6e-c550-4cd7-8f34-97b71968e4a2" containerName="cleanup" Nov 01 21:00:19.941022 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:19.924822 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce36bd6e-c550-4cd7-8f34-97b71968e4a2" containerName="cleanup" Nov 01 21:00:19.941022 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:19.924883 4484 cpu_manager.go:414] "containerMap: removing container" podUID="b4578009-b3ff-4ce6-825a-8f92c1d3d72d" containerName="cleanup" Nov 01 21:00:19.941022 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:19.924890 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4578009-b3ff-4ce6-825a-8f92c1d3d72d" containerName="cleanup" Nov 01 21:00:19.941022 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:19.924902 4484 cpu_manager.go:414] "containerMap: removing container" podUID="b4578009-b3ff-4ce6-825a-8f92c1d3d72d" containerName="cleanup" Nov 01 21:00:19.941022 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:19.924908 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4578009-b3ff-4ce6-825a-8f92c1d3d72d" containerName="cleanup" Nov 01 21:00:19.941022 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:19.924930 4484 cpu_manager.go:414] "containerMap: removing container" podUID="b4578009-b3ff-4ce6-825a-8f92c1d3d72d" containerName="cleanup" Nov 01 21:00:19.941022 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:19.924938 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4578009-b3ff-4ce6-825a-8f92c1d3d72d" containerName="cleanup" Nov 01 21:00:19.976197 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:19.947175 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4578009-b3ff-4ce6-825a-8f92c1d3d72d" containerName="cleanup" Nov 01 21:00:20.103544 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.102100 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4578009-b3ff-4ce6-825a-8f92c1d3d72d" containerName="cleanup" Nov 01 21:00:20.103544 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.102154 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce36bd6e-c550-4cd7-8f34-97b71968e4a2" containerName="cleanup" Nov 01 21:00:20.103544 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.102169 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4578009-b3ff-4ce6-825a-8f92c1d3d72d" containerName="cleanup" Nov 01 21:00:20.103544 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.102222 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ae0b2c8-270b-4a3f-bb16-523874681ade" containerName="istio-proxy" Nov 01 21:00:20.103544 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.102248 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce36bd6e-c550-4cd7-8f34-97b71968e4a2" containerName="cleanup" Nov 01 21:00:20.103544 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.102275 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce36bd6e-c550-4cd7-8f34-97b71968e4a2" containerName="cleanup" Nov 01 21:00:20.140660 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.116117 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-29baa8-6c565db695-5m42b" Nov 01 21:00:20.140660 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.125754 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64a2f0cb-f518-45ff-9f7c-69cd38eacdeb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-29baa8-6c565db695-5m42b\" (UID: \"64a2f0cb-f518-45ff-9f7c-69cd38eacdeb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-29baa8-6c565db695-5m42b" Nov 01 21:00:20.140660 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.125897 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh8bf\" (UniqueName: \"kubernetes.io/projected/64a2f0cb-f518-45ff-9f7c-69cd38eacdeb-kube-api-access-hh8bf\") pod \"payload-pre-processing-e2e-shared-b-w0-29baa8-6c565db695-5m42b\" (UID: \"64a2f0cb-f518-45ff-9f7c-69cd38eacdeb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-29baa8-6c565db695-5m42b" Nov 01 21:00:20.218059 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.212731 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-29baa8-dockercfg-np5p6\"" Nov 01 21:00:20.218059 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.212822 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-29baa8\"" Nov 01 21:00:20.237576 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.237541 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64a2f0cb-f518-45ff-9f7c-69cd38eacdeb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-29baa8-6c565db695-5m42b\" (UID: \"64a2f0cb-f518-45ff-9f7c-69cd38eacdeb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-29baa8-6c565db695-5m42b" Nov 01 21:00:20.237727 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.237703 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh8bf\" (UniqueName: \"kubernetes.io/projected/64a2f0cb-f518-45ff-9f7c-69cd38eacdeb-kube-api-access-hh8bf\") pod \"payload-pre-processing-e2e-shared-b-w0-29baa8-6c565db695-5m42b\" (UID: \"64a2f0cb-f518-45ff-9f7c-69cd38eacdeb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-29baa8-6c565db695-5m42b" Nov 01 21:00:20.244198 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.243966 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892780-q4ns9" event={"ID":"474d07b8-cbb2-4beb-a17f-fe445d0c9ad4","Type":"ContainerStarted","Data":"68d3804261d6c49100133c75df03c677b32acad7fc1ab236dc12f0cb08c39fe7"} Nov 01 21:00:20.259868 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.258805 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq" event={"ID":"6c3c6fec-0841-4a0c-b761-626d8c53a1cb","Type":"ContainerStarted","Data":"a5d91c1d8490cb6b27129606cf5d3c39f95dedafea0ed794a5cd959283a6e608"} Nov 01 21:00:20.259868 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.258884 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq" event={"ID":"6c3c6fec-0841-4a0c-b761-626d8c53a1cb","Type":"ContainerStarted","Data":"d6f5356e786c5ba83b5215c9fcb613b3ff089d2b2912b975bb5a078a47ff6361"} Nov 01 21:00:20.261356 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.261332 4484 scope.go:122] "RemoveContainer" containerID="68d3804261d6c49100133c75df03c677b32acad7fc1ab236dc12f0cb08c39fe7" Nov 01 21:00:20.310799 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.310760 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64a2f0cb-f518-45ff-9f7c-69cd38eacdeb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-29baa8-6c565db695-5m42b\" (UID: \"64a2f0cb-f518-45ff-9f7c-69cd38eacdeb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-29baa8-6c565db695-5m42b" Nov 01 21:00:20.311282 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.310864 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-29baa8-6c565db695-5m42b"] Nov 01 21:00:20.313150 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.313117 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892780-vdj2s" event={"ID":"ce36bd6e-c550-4cd7-8f34-97b71968e4a2","Type":"ContainerDied","Data":"99fb62932bed3a27a7aa019073e66b4152215a56e81414419d909833d88c6203"} Nov 01 21:00:20.313302 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.313286 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892780-vdj2s" Nov 01 21:00:20.345244 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:00:20.319740 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892780-q4ns9_odh-ai-gateway-infra(474d07b8-cbb2-4beb-a17f-fe445d0c9ad4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892780-q4ns9" podUID="474d07b8-cbb2-4beb-a17f-fe445d0c9ad4" Nov 01 21:00:20.345244 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.320980 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh8bf\" (UniqueName: \"kubernetes.io/projected/64a2f0cb-f518-45ff-9f7c-69cd38eacdeb-kube-api-access-hh8bf\") pod \"payload-pre-processing-e2e-shared-b-w0-29baa8-6c565db695-5m42b\" (UID: \"64a2f0cb-f518-45ff-9f7c-69cd38eacdeb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-29baa8-6c565db695-5m42b" Nov 01 21:00:20.345244 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.323375 4484 generic.go:356] "Generic (PLEG): container finished" podID="cfd728c4-7f6b-4197-a147-1f340427f414" containerID="106c40c76afdcd5459e2551a44fd14ea46114e7829c0a611c823f798446a9bed" exitCode=60 Nov 01 21:00:20.345244 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.323643 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892780-96d26" event={"ID":"cfd728c4-7f6b-4197-a147-1f340427f414","Type":"ContainerDied","Data":"106c40c76afdcd5459e2551a44fd14ea46114e7829c0a611c823f798446a9bed"} Nov 01 21:00:20.345244 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.328808 4484 generic.go:356] "Generic (PLEG): container finished" podID="b19e6e75-0ef8-440d-8617-e5c71bfde0d2" containerID="addbc52446ddbfa8465e17a232e1cba685b5197bc2e9b13cfc2d2fab8f7c95a9" exitCode=60 Nov 01 21:00:20.345244 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.329304 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd038b-29892780-vltsq" event={"ID":"b19e6e75-0ef8-440d-8617-e5c71bfde0d2","Type":"ContainerDied","Data":"addbc52446ddbfa8465e17a232e1cba685b5197bc2e9b13cfc2d2fab8f7c95a9"} Nov 01 21:00:20.345244 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.331380 4484 generic.go:356] "Generic (PLEG): container finished" podID="38c14ac7-4673-46b7-a721-652a972f3ba5" containerID="d22b765898c651cb3947da46372fca09760dcfee60152d468a23a3a620687a37" exitCode=60 Nov 01 21:00:20.345244 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.331526 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892780-p6gfx" event={"ID":"38c14ac7-4673-46b7-a721-652a972f3ba5","Type":"ContainerDied","Data":"d22b765898c651cb3947da46372fca09760dcfee60152d468a23a3a620687a37"} Nov 01 21:00:20.345244 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.331691 4484 scope.go:122] "RemoveContainer" containerID="addbc52446ddbfa8465e17a232e1cba685b5197bc2e9b13cfc2d2fab8f7c95a9" Nov 01 21:00:20.345244 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.332026 4484 scope.go:122] "RemoveContainer" containerID="d22b765898c651cb3947da46372fca09760dcfee60152d468a23a3a620687a37" Nov 01 21:00:20.345244 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:00:20.332469 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892780-p6gfx_odh-ai-gateway-infra(38c14ac7-4673-46b7-a721-652a972f3ba5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892780-p6gfx" podUID="38c14ac7-4673-46b7-a721-652a972f3ba5" Nov 01 21:00:20.345244 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.335148 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892780-jmsct" event={"ID":"e5aa9680-52ae-4173-875b-1097a5963fc0","Type":"ContainerStarted","Data":"7d59d6fcb5cedea42bafbe7bf13c77c68bb03b3ffb23c84062387cc3a1171e75"} Nov 01 21:00:20.345244 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.335722 4484 scope.go:122] "RemoveContainer" containerID="7d59d6fcb5cedea42bafbe7bf13c77c68bb03b3ffb23c84062387cc3a1171e75" Nov 01 21:00:20.345244 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:00:20.336270 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-944263-29892780-jmsct_odh-ai-gateway-infra(e5aa9680-52ae-4173-875b-1097a5963fc0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892780-jmsct" podUID="e5aa9680-52ae-4173-875b-1097a5963fc0" Nov 01 21:00:20.345244 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:00:20.340333 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-dd038b-29892780-vltsq_odh-ai-gateway-infra(b19e6e75-0ef8-440d-8617-e5c71bfde0d2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd038b-29892780-vltsq" podUID="b19e6e75-0ef8-440d-8617-e5c71bfde0d2" Nov 01 21:00:20.368421 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.368380 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd038b-29892780-vltsq"] Nov 01 21:00:20.372362 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.372327 4484 scope.go:122] "RemoveContainer" containerID="764a3b1800e86a3495f68a14ecfda5256427b0ad2f3dae0f7028d7a07359b00d" Nov 01 21:00:20.372581 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.372553 4484 scope.go:122] "RemoveContainer" containerID="7e63bb45c74fef2b816cb0175b4280435c3110e0ad03786267ff37c6dcb73fe4" Nov 01 21:00:20.374785 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.374763 4484 scope.go:122] "RemoveContainer" containerID="3b01f7630192c53ae6ec52ef39065272f71f4374eefd958a93e255415c987a66" Nov 01 21:00:20.377469 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.375401 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-29baa8-6bbff57b65-6djp8"] Nov 01 21:00:20.378229 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.377720 4484 cpu_manager.go:414] "containerMap: removing container" podUID="ce36bd6e-c550-4cd7-8f34-97b71968e4a2" containerName="cleanup" Nov 01 21:00:20.378229 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.377742 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce36bd6e-c550-4cd7-8f34-97b71968e4a2" containerName="cleanup" Nov 01 21:00:20.387323 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.387299 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-29baa8-6bbff57b65-6djp8" Nov 01 21:00:20.395124 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.395030 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-29baa8-6bbff57b65-6djp8"] Nov 01 21:00:20.408604 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.408573 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892780-p6gfx"] Nov 01 21:00:20.417343 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.417238 4484 scope.go:122] "RemoveContainer" containerID="86c922ecb57d854e00f28025a38f999523a4bbda0672f1e73b4eec919d35792a" Nov 01 21:00:20.468807 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.468632 4484 scope.go:122] "RemoveContainer" containerID="664a1e35abd176532a1bebe3892ba5deea20919b1b062d25762c87dab4d27c8c" Nov 01 21:00:20.485175 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.483173 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnp8q\" (UniqueName: \"kubernetes.io/projected/4048d711-2455-4907-b85f-0baa9602e224-kube-api-access-tnp8q\") pod \"payload-processing-e2e-shared-b-w0-29baa8-6bbff57b65-6djp8\" (UID: \"4048d711-2455-4907-b85f-0baa9602e224\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-29baa8-6bbff57b65-6djp8" Nov 01 21:00:20.485175 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.483455 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4048d711-2455-4907-b85f-0baa9602e224-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-29baa8-6bbff57b65-6djp8\" (UID: \"4048d711-2455-4907-b85f-0baa9602e224\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-29baa8-6bbff57b65-6djp8" Nov 01 21:00:20.490370 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.490078 4484 scope.go:122] "RemoveContainer" containerID="680415e314c9d523bb13494243c4ac1bf4d90a45770b71f5ffb8c4c2b6eb7735" Nov 01 21:00:20.553711 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.553653 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq" podStartSLOduration=3.553638413 podStartE2EDuration="3.553638413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:00:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:00:20.549219802 +0000 UTC m=+1771.326859950" watchObservedRunningTime="2026-11-01 21:00:20.553638413 +0000 UTC m=+1771.331278548" Nov 01 21:00:20.585159 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.585120 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnp8q\" (UniqueName: \"kubernetes.io/projected/4048d711-2455-4907-b85f-0baa9602e224-kube-api-access-tnp8q\") pod \"payload-processing-e2e-shared-b-w0-29baa8-6bbff57b65-6djp8\" (UID: \"4048d711-2455-4907-b85f-0baa9602e224\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-29baa8-6bbff57b65-6djp8" Nov 01 21:00:20.585366 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.585237 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4048d711-2455-4907-b85f-0baa9602e224-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-29baa8-6bbff57b65-6djp8\" (UID: \"4048d711-2455-4907-b85f-0baa9602e224\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-29baa8-6bbff57b65-6djp8" Nov 01 21:00:20.586051 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.586021 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4048d711-2455-4907-b85f-0baa9602e224-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-29baa8-6bbff57b65-6djp8\" (UID: \"4048d711-2455-4907-b85f-0baa9602e224\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-29baa8-6bbff57b65-6djp8" Nov 01 21:00:20.604292 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.604247 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-29baa8-6c565db695-5m42b" Nov 01 21:00:20.643749 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.643710 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnp8q\" (UniqueName: \"kubernetes.io/projected/4048d711-2455-4907-b85f-0baa9602e224-kube-api-access-tnp8q\") pod \"payload-processing-e2e-shared-b-w0-29baa8-6bbff57b65-6djp8\" (UID: \"4048d711-2455-4907-b85f-0baa9602e224\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-29baa8-6bbff57b65-6djp8" Nov 01 21:00:20.717019 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.712207 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-29baa8-6bbff57b65-6djp8" Nov 01 21:00:20.717019 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.713720 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892780-96d26" Nov 01 21:00:20.792824 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.792790 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfd728c4-7f6b-4197-a147-1f340427f414-kube-api-access-l7pnm\" (UniqueName: \"kubernetes.io/projected/cfd728c4-7f6b-4197-a147-1f340427f414-kube-api-access-l7pnm\") pod \"cfd728c4-7f6b-4197-a147-1f340427f414\" (UID: \"cfd728c4-7f6b-4197-a147-1f340427f414\") " Nov 01 21:00:20.793017 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.792853 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfd728c4-7f6b-4197-a147-1f340427f414-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfd728c4-7f6b-4197-a147-1f340427f414-maas-api-service-ca\") pod \"cfd728c4-7f6b-4197-a147-1f340427f414\" (UID: \"cfd728c4-7f6b-4197-a147-1f340427f414\") " Nov 01 21:00:20.793370 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.793342 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfd728c4-7f6b-4197-a147-1f340427f414-maas-api-service-ca" pod "cfd728c4-7f6b-4197-a147-1f340427f414" (UID: "cfd728c4-7f6b-4197-a147-1f340427f414"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:00:20.793650 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.793631 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfd728c4-7f6b-4197-a147-1f340427f414-maas-api-service-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:20.799389 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.798368 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfd728c4-7f6b-4197-a147-1f340427f414-kube-api-access-l7pnm" pod "cfd728c4-7f6b-4197-a147-1f340427f414" (UID: "cfd728c4-7f6b-4197-a147-1f340427f414"). InnerVolumeSpecName "kube-api-access-l7pnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:00:20.835106 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.834390 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892780-vdj2s"] Nov 01 21:00:20.839602 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.839568 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892780-vdj2s"] Nov 01 21:00:20.895075 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.895043 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7pnm\" (UniqueName: \"kubernetes.io/projected/cfd728c4-7f6b-4197-a147-1f340427f414-kube-api-access-l7pnm\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:20.946737 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.946490 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq" Nov 01 21:00:20.965325 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:20.965294 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq" Nov 01 21:00:21.060962 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:21.060927 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892780-6xlxn"] Nov 01 21:00:21.110767 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:00:21.110736 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64a2f0cb_f518_45ff_9f7c_69cd38eacdeb.slice/crio-a1bed4d63ab788808d647befe9ba773d22fd3b875f744ca6e2d732966fcfb6c9 WatchSource:0}: Error finding container a1bed4d63ab788808d647befe9ba773d22fd3b875f744ca6e2d732966fcfb6c9: Status 404 returned error can't find the container with id a1bed4d63ab788808d647befe9ba773d22fd3b875f744ca6e2d732966fcfb6c9 Nov 01 21:00:21.112845 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:21.112822 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-29baa8-6c565db695-5m42b"] Nov 01 21:00:21.270496 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:00:21.270466 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4048d711_2455_4907_b85f_0baa9602e224.slice/crio-4ad6c742592f9892764831f6382bbe393aa43fa2622966149cad423b9875ed40 WatchSource:0}: Error finding container 4ad6c742592f9892764831f6382bbe393aa43fa2622966149cad423b9875ed40: Status 404 returned error can't find the container with id 4ad6c742592f9892764831f6382bbe393aa43fa2622966149cad423b9875ed40 Nov 01 21:00:21.271108 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:21.271080 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-29baa8-6bbff57b65-6djp8"] Nov 01 21:00:21.347771 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:21.347741 4484 generic.go:356] "Generic (PLEG): container finished" podID="da6f250e-6516-4a04-a37a-f30c02ec682b" containerID="e4d6d7f222eebd847c2ef7e294a6f5640223540cd4189cf4c4b03c92c6dd6fe3" exitCode=60 Nov 01 21:00:21.347915 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:21.347822 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892780-7pl4v" event={"ID":"da6f250e-6516-4a04-a37a-f30c02ec682b","Type":"ContainerDied","Data":"e4d6d7f222eebd847c2ef7e294a6f5640223540cd4189cf4c4b03c92c6dd6fe3"} Nov 01 21:00:21.347915 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:21.347890 4484 scope.go:122] "RemoveContainer" containerID="8b4816f8698560a1753b6a3691d097b27016c4ac7012b7ecd68d5948de464d2d" Nov 01 21:00:21.348396 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:21.348379 4484 scope.go:122] "RemoveContainer" containerID="e4d6d7f222eebd847c2ef7e294a6f5640223540cd4189cf4c4b03c92c6dd6fe3" Nov 01 21:00:21.348846 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:00:21.348825 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892780-7pl4v_odh-ai-gateway-infra(da6f250e-6516-4a04-a37a-f30c02ec682b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892780-7pl4v" podUID="da6f250e-6516-4a04-a37a-f30c02ec682b" Nov 01 21:00:21.350328 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:21.350044 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892780-96d26" Nov 01 21:00:21.350328 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:21.350081 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892780-96d26" event={"ID":"cfd728c4-7f6b-4197-a147-1f340427f414","Type":"ContainerDied","Data":"fbff743b8f19a7dfeada1d3da59be3eaf945b6f3a127ef8d46d1e918105af4a8"} Nov 01 21:00:21.359651 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:21.359619 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-29baa8-6c565db695-5m42b" event={"ID":"64a2f0cb-f518-45ff-9f7c-69cd38eacdeb","Type":"ContainerStarted","Data":"3a748d11adb52251c619a14166fc56369c6912b6ef19b0adc16c9885f56e1adc"} Nov 01 21:00:21.359775 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:21.359685 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-29baa8-6c565db695-5m42b" event={"ID":"64a2f0cb-f518-45ff-9f7c-69cd38eacdeb","Type":"ContainerStarted","Data":"a1bed4d63ab788808d647befe9ba773d22fd3b875f744ca6e2d732966fcfb6c9"} Nov 01 21:00:21.359958 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:21.359939 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-29baa8-6c565db695-5m42b" Nov 01 21:00:21.364615 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:21.364597 4484 scope.go:122] "RemoveContainer" containerID="106c40c76afdcd5459e2551a44fd14ea46114e7829c0a611c823f798446a9bed" Nov 01 21:00:21.371429 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:21.371385 4484 generic.go:356] "Generic (PLEG): container finished" podID="e5aa9680-52ae-4173-875b-1097a5963fc0" containerID="7d59d6fcb5cedea42bafbe7bf13c77c68bb03b3ffb23c84062387cc3a1171e75" exitCode=60 Nov 01 21:00:21.379545 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:21.379517 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce36bd6e-c550-4cd7-8f34-97b71968e4a2" path="/var/lib/kubelet/pods/ce36bd6e-c550-4cd7-8f34-97b71968e4a2/volumes" Nov 01 21:00:21.379952 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:21.379931 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892780-jmsct" event={"ID":"e5aa9680-52ae-4173-875b-1097a5963fc0","Type":"ContainerDied","Data":"7d59d6fcb5cedea42bafbe7bf13c77c68bb03b3ffb23c84062387cc3a1171e75"} Nov 01 21:00:21.380053 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:21.379989 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-29baa8-6bbff57b65-6djp8" Nov 01 21:00:21.380053 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:21.380041 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b27e66-29892780-hvgx4" event={"ID":"7369891a-d9d2-42f1-8165-f817e9c00984","Type":"ContainerStarted","Data":"e7c802121a4e393414de6bc15052257c2221bab158bccd11d8c6357063ee278a"} Nov 01 21:00:21.380158 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:21.380066 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-29baa8-6bbff57b65-6djp8" event={"ID":"4048d711-2455-4907-b85f-0baa9602e224","Type":"ContainerStarted","Data":"4ad6c742592f9892764831f6382bbe393aa43fa2622966149cad423b9875ed40"} Nov 01 21:00:21.380357 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:21.380341 4484 scope.go:122] "RemoveContainer" containerID="78c31ce9df621c4320497c8e8b45517255aac51955e9116324f4c0d02644c553" Nov 01 21:00:21.380649 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:21.380633 4484 scope.go:122] "RemoveContainer" containerID="7d59d6fcb5cedea42bafbe7bf13c77c68bb03b3ffb23c84062387cc3a1171e75" Nov 01 21:00:21.380882 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:00:21.380865 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-944263-29892780-jmsct_odh-ai-gateway-infra(e5aa9680-52ae-4173-875b-1097a5963fc0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892780-jmsct" podUID="e5aa9680-52ae-4173-875b-1097a5963fc0" Nov 01 21:00:21.382319 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:21.382298 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892780-9tm2f" event={"ID":"d0f5aabc-58f2-4166-af2d-39aa67592cb1","Type":"ContainerStarted","Data":"ebd29d2f86acfcbbbeb047888e6e5f6f91f1c0a2da89f7b9cb0579f26b166341"} Nov 01 21:00:21.387122 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:21.386293 4484 generic.go:356] "Generic (PLEG): container finished" podID="474d07b8-cbb2-4beb-a17f-fe445d0c9ad4" containerID="68d3804261d6c49100133c75df03c677b32acad7fc1ab236dc12f0cb08c39fe7" exitCode=60 Nov 01 21:00:21.387122 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:21.386781 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892780-q4ns9" event={"ID":"474d07b8-cbb2-4beb-a17f-fe445d0c9ad4","Type":"ContainerDied","Data":"68d3804261d6c49100133c75df03c677b32acad7fc1ab236dc12f0cb08c39fe7"} Nov 01 21:00:21.387122 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:21.387047 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq" Nov 01 21:00:21.387739 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:21.387424 4484 scope.go:122] "RemoveContainer" containerID="68d3804261d6c49100133c75df03c677b32acad7fc1ab236dc12f0cb08c39fe7" Nov 01 21:00:21.387739 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:00:21.387703 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892780-q4ns9_odh-ai-gateway-infra(474d07b8-cbb2-4beb-a17f-fe445d0c9ad4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892780-q4ns9" podUID="474d07b8-cbb2-4beb-a17f-fe445d0c9ad4" Nov 01 21:00:21.388517 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:21.388482 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq" Nov 01 21:00:21.405290 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:21.405260 4484 scope.go:122] "RemoveContainer" containerID="f5fcc30aa2fbf88fedd5d1850cfb015c3f61a92090dddf5ba5b52f0ae14a8d06" Nov 01 21:00:21.445205 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:21.445167 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892780-q4ns9"] Nov 01 21:00:21.647495 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:21.647460 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd038b-29892780-vltsq" Nov 01 21:00:21.652919 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:21.652892 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892780-p6gfx" Nov 01 21:00:21.689493 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:21.689469 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892780-6xlxn" Nov 01 21:00:21.717127 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:21.717088 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52fb948e-15a6-400e-be44-2e88a1fb1501-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52fb948e-15a6-400e-be44-2e88a1fb1501-maas-api-service-ca\") pod \"52fb948e-15a6-400e-be44-2e88a1fb1501\" (UID: \"52fb948e-15a6-400e-be44-2e88a1fb1501\") " Nov 01 21:00:21.717585 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:21.717192 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38c14ac7-4673-46b7-a721-652a972f3ba5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38c14ac7-4673-46b7-a721-652a972f3ba5-maas-api-service-ca\") pod \"38c14ac7-4673-46b7-a721-652a972f3ba5\" (UID: \"38c14ac7-4673-46b7-a721-652a972f3ba5\") " Nov 01 21:00:21.717585 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:21.717462 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52fb948e-15a6-400e-be44-2e88a1fb1501-kube-api-access-46v8z\" (UniqueName: \"kubernetes.io/projected/52fb948e-15a6-400e-be44-2e88a1fb1501-kube-api-access-46v8z\") pod \"52fb948e-15a6-400e-be44-2e88a1fb1501\" (UID: \"52fb948e-15a6-400e-be44-2e88a1fb1501\") " Nov 01 21:00:21.717585 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:21.717516 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38c14ac7-4673-46b7-a721-652a972f3ba5-maas-api-service-ca" pod "38c14ac7-4673-46b7-a721-652a972f3ba5" (UID: "38c14ac7-4673-46b7-a721-652a972f3ba5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:00:21.717585 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:21.717517 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52fb948e-15a6-400e-be44-2e88a1fb1501-maas-api-service-ca" pod "52fb948e-15a6-400e-be44-2e88a1fb1501" (UID: "52fb948e-15a6-400e-be44-2e88a1fb1501"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:00:21.717585 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:21.717580 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b19e6e75-0ef8-440d-8617-e5c71bfde0d2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b19e6e75-0ef8-440d-8617-e5c71bfde0d2-maas-api-service-ca\") pod \"b19e6e75-0ef8-440d-8617-e5c71bfde0d2\" (UID: \"b19e6e75-0ef8-440d-8617-e5c71bfde0d2\") " Nov 01 21:00:21.717847 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:21.717621 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b19e6e75-0ef8-440d-8617-e5c71bfde0d2-kube-api-access-l8vtt\" (UniqueName: \"kubernetes.io/projected/b19e6e75-0ef8-440d-8617-e5c71bfde0d2-kube-api-access-l8vtt\") pod \"b19e6e75-0ef8-440d-8617-e5c71bfde0d2\" (UID: \"b19e6e75-0ef8-440d-8617-e5c71bfde0d2\") " Nov 01 21:00:21.717847 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:21.717681 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38c14ac7-4673-46b7-a721-652a972f3ba5-kube-api-access-nrhn2\" (UniqueName: \"kubernetes.io/projected/38c14ac7-4673-46b7-a721-652a972f3ba5-kube-api-access-nrhn2\") pod \"38c14ac7-4673-46b7-a721-652a972f3ba5\" (UID: \"38c14ac7-4673-46b7-a721-652a972f3ba5\") " Nov 01 21:00:21.718014 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:21.717976 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b19e6e75-0ef8-440d-8617-e5c71bfde0d2-maas-api-service-ca" pod "b19e6e75-0ef8-440d-8617-e5c71bfde0d2" (UID: "b19e6e75-0ef8-440d-8617-e5c71bfde0d2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:00:21.718366 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:21.718341 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b19e6e75-0ef8-440d-8617-e5c71bfde0d2-maas-api-service-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:21.718484 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:21.718369 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52fb948e-15a6-400e-be44-2e88a1fb1501-maas-api-service-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:21.718484 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:21.718381 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38c14ac7-4673-46b7-a721-652a972f3ba5-maas-api-service-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:21.719828 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:21.719800 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38c14ac7-4673-46b7-a721-652a972f3ba5-kube-api-access-nrhn2" pod "38c14ac7-4673-46b7-a721-652a972f3ba5" (UID: "38c14ac7-4673-46b7-a721-652a972f3ba5"). InnerVolumeSpecName "kube-api-access-nrhn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:00:21.719917 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:21.719862 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b19e6e75-0ef8-440d-8617-e5c71bfde0d2-kube-api-access-l8vtt" pod "b19e6e75-0ef8-440d-8617-e5c71bfde0d2" (UID: "b19e6e75-0ef8-440d-8617-e5c71bfde0d2"). InnerVolumeSpecName "kube-api-access-l8vtt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:00:21.719917 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:21.719897 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52fb948e-15a6-400e-be44-2e88a1fb1501-kube-api-access-46v8z" pod "52fb948e-15a6-400e-be44-2e88a1fb1501" (UID: "52fb948e-15a6-400e-be44-2e88a1fb1501"). InnerVolumeSpecName "kube-api-access-46v8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:00:21.819947 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:21.819860 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrhn2\" (UniqueName: \"kubernetes.io/projected/38c14ac7-4673-46b7-a721-652a972f3ba5-kube-api-access-nrhn2\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:21.819947 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:21.819891 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46v8z\" (UniqueName: \"kubernetes.io/projected/52fb948e-15a6-400e-be44-2e88a1fb1501-kube-api-access-46v8z\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:21.819947 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:21.819902 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8vtt\" (UniqueName: \"kubernetes.io/projected/b19e6e75-0ef8-440d-8617-e5c71bfde0d2-kube-api-access-l8vtt\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:21.995325 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:21.995270 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-29baa8-6bbff57b65-6djp8" podStartSLOduration=2.995254733 podStartE2EDuration="2.995254733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:00:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:00:21.990887253 +0000 UTC m=+1772.768527388" watchObservedRunningTime="2026-11-01 21:00:21.995254733 +0000 UTC m=+1772.772894867" Nov 01 21:00:22.228564 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:22.228464 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892780-jmsct"] Nov 01 21:00:22.329869 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:22.329826 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:22.410307 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:22.410273 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892780-96d26"] Nov 01 21:00:22.416603 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:22.416556 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24f923-29892780-96d26"] Nov 01 21:00:22.416866 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:22.416848 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd038b-29892780-vltsq" Nov 01 21:00:22.416987 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:22.416960 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd038b-29892780-vltsq" event={"ID":"b19e6e75-0ef8-440d-8617-e5c71bfde0d2","Type":"ContainerDied","Data":"994947f0e8fa04bb3c0e6620bb9c1cdbba6cc657e0411845a993bd16b5a4b94b"} Nov 01 21:00:22.417183 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:22.417160 4484 scope.go:122] "RemoveContainer" containerID="addbc52446ddbfa8465e17a232e1cba685b5197bc2e9b13cfc2d2fab8f7c95a9" Nov 01 21:00:22.419408 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:22.419150 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892780-p6gfx" event={"ID":"38c14ac7-4673-46b7-a721-652a972f3ba5","Type":"ContainerDied","Data":"44c4cdbec929dc88bccc5c35556f6a7c2d64567b8df9a09308471f51d43149d9"} Nov 01 21:00:22.419408 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:22.419248 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892780-p6gfx" Nov 01 21:00:22.426213 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:22.426181 4484 generic.go:356] "Generic (PLEG): container finished" podID="7369891a-d9d2-42f1-8165-f817e9c00984" containerID="e7c802121a4e393414de6bc15052257c2221bab158bccd11d8c6357063ee278a" exitCode=60 Nov 01 21:00:22.426337 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:22.426303 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b27e66-29892780-hvgx4" event={"ID":"7369891a-d9d2-42f1-8165-f817e9c00984","Type":"ContainerDied","Data":"e7c802121a4e393414de6bc15052257c2221bab158bccd11d8c6357063ee278a"} Nov 01 21:00:22.426871 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:22.426852 4484 scope.go:122] "RemoveContainer" containerID="e7c802121a4e393414de6bc15052257c2221bab158bccd11d8c6357063ee278a" Nov 01 21:00:22.427333 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:00:22.427296 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-b27e66-29892780-hvgx4_odh-ai-gateway-infra(7369891a-d9d2-42f1-8165-f817e9c00984)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b27e66-29892780-hvgx4" podUID="7369891a-d9d2-42f1-8165-f817e9c00984" Nov 01 21:00:22.433774 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:22.433684 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-29baa8-6bbff57b65-6djp8" event={"ID":"4048d711-2455-4907-b85f-0baa9602e224","Type":"ContainerStarted","Data":"c7141cbaa17e41d34f189140fef8f80314dc87f7456324ee34bfe162d9632ce3"} Nov 01 21:00:22.436259 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:22.436236 4484 generic.go:356] "Generic (PLEG): container finished" podID="d0f5aabc-58f2-4166-af2d-39aa67592cb1" containerID="ebd29d2f86acfcbbbeb047888e6e5f6f91f1c0a2da89f7b9cb0579f26b166341" exitCode=60 Nov 01 21:00:22.436364 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:22.436310 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892780-9tm2f" event={"ID":"d0f5aabc-58f2-4166-af2d-39aa67592cb1","Type":"ContainerDied","Data":"ebd29d2f86acfcbbbeb047888e6e5f6f91f1c0a2da89f7b9cb0579f26b166341"} Nov 01 21:00:22.436947 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:22.436597 4484 scope.go:122] "RemoveContainer" containerID="ebd29d2f86acfcbbbeb047888e6e5f6f91f1c0a2da89f7b9cb0579f26b166341" Nov 01 21:00:22.436947 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:00:22.436870 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892780-9tm2f_odh-ai-gateway-infra(d0f5aabc-58f2-4166-af2d-39aa67592cb1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892780-9tm2f" podUID="d0f5aabc-58f2-4166-af2d-39aa67592cb1" Nov 01 21:00:22.439884 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:22.439857 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892780-6xlxn" event={"ID":"52fb948e-15a6-400e-be44-2e88a1fb1501","Type":"ContainerDied","Data":"e26e9408ee20b56e11e215f7af555968f26c23409d34535f1091d18c8062660a"} Nov 01 21:00:22.439976 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:22.439860 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892780-6xlxn" Nov 01 21:00:22.446165 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:22.446142 4484 scope.go:122] "RemoveContainer" containerID="d22b765898c651cb3947da46372fca09760dcfee60152d468a23a3a620687a37" Nov 01 21:00:22.468395 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:22.467160 4484 scope.go:122] "RemoveContainer" containerID="7e63bb45c74fef2b816cb0175b4280435c3110e0ad03786267ff37c6dcb73fe4" Nov 01 21:00:22.488233 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:22.488115 4484 scope.go:122] "RemoveContainer" containerID="764a3b1800e86a3495f68a14ecfda5256427b0ad2f3dae0f7028d7a07359b00d" Nov 01 21:00:22.502598 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:22.502171 4484 scope.go:122] "RemoveContainer" containerID="478f1708aa08fb0a88a6764b8d876364f76f5427d962cd5c773761703be593cf" Nov 01 21:00:22.693676 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:22.693643 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892780-jmsct" Nov 01 21:00:22.721169 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:22.721141 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892780-q4ns9" Nov 01 21:00:22.761434 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:22.761339 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5aa9680-52ae-4173-875b-1097a5963fc0-kube-api-access-8tgtb\" (UniqueName: \"kubernetes.io/projected/e5aa9680-52ae-4173-875b-1097a5963fc0-kube-api-access-8tgtb\") pod \"e5aa9680-52ae-4173-875b-1097a5963fc0\" (UID: \"e5aa9680-52ae-4173-875b-1097a5963fc0\") " Nov 01 21:00:22.761434 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:22.761393 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/474d07b8-cbb2-4beb-a17f-fe445d0c9ad4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/474d07b8-cbb2-4beb-a17f-fe445d0c9ad4-maas-api-service-ca\") pod \"474d07b8-cbb2-4beb-a17f-fe445d0c9ad4\" (UID: \"474d07b8-cbb2-4beb-a17f-fe445d0c9ad4\") " Nov 01 21:00:22.761728 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:22.761698 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5aa9680-52ae-4173-875b-1097a5963fc0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5aa9680-52ae-4173-875b-1097a5963fc0-maas-api-service-ca\") pod \"e5aa9680-52ae-4173-875b-1097a5963fc0\" (UID: \"e5aa9680-52ae-4173-875b-1097a5963fc0\") " Nov 01 21:00:22.761728 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:22.761723 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/474d07b8-cbb2-4beb-a17f-fe445d0c9ad4-kube-api-access-pqdzk\" (UniqueName: \"kubernetes.io/projected/474d07b8-cbb2-4beb-a17f-fe445d0c9ad4-kube-api-access-pqdzk\") pod \"474d07b8-cbb2-4beb-a17f-fe445d0c9ad4\" (UID: \"474d07b8-cbb2-4beb-a17f-fe445d0c9ad4\") " Nov 01 21:00:22.761917 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:22.761786 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/474d07b8-cbb2-4beb-a17f-fe445d0c9ad4-maas-api-service-ca" pod "474d07b8-cbb2-4beb-a17f-fe445d0c9ad4" (UID: "474d07b8-cbb2-4beb-a17f-fe445d0c9ad4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:00:22.762102 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:22.762076 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5aa9680-52ae-4173-875b-1097a5963fc0-maas-api-service-ca" pod "e5aa9680-52ae-4173-875b-1097a5963fc0" (UID: "e5aa9680-52ae-4173-875b-1097a5963fc0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:00:22.762260 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:22.762244 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5aa9680-52ae-4173-875b-1097a5963fc0-maas-api-service-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:22.762397 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:22.762262 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/474d07b8-cbb2-4beb-a17f-fe445d0c9ad4-maas-api-service-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:22.763569 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:22.763544 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5aa9680-52ae-4173-875b-1097a5963fc0-kube-api-access-8tgtb" pod "e5aa9680-52ae-4173-875b-1097a5963fc0" (UID: "e5aa9680-52ae-4173-875b-1097a5963fc0"). InnerVolumeSpecName "kube-api-access-8tgtb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:00:22.763864 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:22.763845 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/474d07b8-cbb2-4beb-a17f-fe445d0c9ad4-kube-api-access-pqdzk" pod "474d07b8-cbb2-4beb-a17f-fe445d0c9ad4" (UID: "474d07b8-cbb2-4beb-a17f-fe445d0c9ad4"). InnerVolumeSpecName "kube-api-access-pqdzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:00:22.863790 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:22.863753 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tgtb\" (UniqueName: \"kubernetes.io/projected/e5aa9680-52ae-4173-875b-1097a5963fc0-kube-api-access-8tgtb\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:22.863790 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:22.863787 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqdzk\" (UniqueName: \"kubernetes.io/projected/474d07b8-cbb2-4beb-a17f-fe445d0c9ad4-kube-api-access-pqdzk\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:22.887073 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:22.887039 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b27e66-29892780-hvgx4"] Nov 01 21:00:23.008924 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:23.008856 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-29baa8-6c565db695-5m42b" podStartSLOduration=4.008835057 podStartE2EDuration="4.008835057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:00:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:00:23.004608063 +0000 UTC m=+1773.782248197" watchObservedRunningTime="2026-11-01 21:00:23.008835057 +0000 UTC m=+1773.786475193" Nov 01 21:00:23.250120 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:23.245420 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd038b-29892780-vltsq"] Nov 01 21:00:23.254100 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:23.254029 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd038b-29892780-vltsq"] Nov 01 21:00:23.261909 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:23.261844 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892780-9tm2f"] Nov 01 21:00:23.374496 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:23.374465 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b19e6e75-0ef8-440d-8617-e5c71bfde0d2" path="/var/lib/kubelet/pods/b19e6e75-0ef8-440d-8617-e5c71bfde0d2/volumes" Nov 01 21:00:23.374976 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:23.374958 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfd728c4-7f6b-4197-a147-1f340427f414" path="/var/lib/kubelet/pods/cfd728c4-7f6b-4197-a147-1f340427f414/volumes" Nov 01 21:00:23.418314 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:23.418278 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892780-6xlxn"] Nov 01 21:00:23.447983 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:23.447947 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892780-6xlxn"] Nov 01 21:00:23.456951 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:23.456926 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892780-jmsct" Nov 01 21:00:23.457133 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:23.456922 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892780-jmsct" event={"ID":"e5aa9680-52ae-4173-875b-1097a5963fc0","Type":"ContainerDied","Data":"65824fed449713ee1a393aa417c114e443691904db47eeba9c603b3a9c3328d1"} Nov 01 21:00:23.457133 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:23.457067 4484 scope.go:122] "RemoveContainer" containerID="7d59d6fcb5cedea42bafbe7bf13c77c68bb03b3ffb23c84062387cc3a1171e75" Nov 01 21:00:23.466302 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:23.466234 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892780-q4ns9" event={"ID":"474d07b8-cbb2-4beb-a17f-fe445d0c9ad4","Type":"ContainerDied","Data":"e08897e4ef7f7fc1c8c64f184f9e8e279cab445a9e87eacf51371f5e9a910b6d"} Nov 01 21:00:23.466302 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:23.466278 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892780-q4ns9" Nov 01 21:00:23.471940 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:23.471917 4484 scope.go:122] "RemoveContainer" containerID="68d3804261d6c49100133c75df03c677b32acad7fc1ab236dc12f0cb08c39fe7" Nov 01 21:00:23.683086 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:23.683050 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b27e66-29892780-hvgx4" Nov 01 21:00:23.747440 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:23.747408 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892780-9tm2f" Nov 01 21:00:23.807488 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:23.807399 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7369891a-d9d2-42f1-8165-f817e9c00984-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7369891a-d9d2-42f1-8165-f817e9c00984-maas-api-service-ca\") pod \"7369891a-d9d2-42f1-8165-f817e9c00984\" (UID: \"7369891a-d9d2-42f1-8165-f817e9c00984\") " Nov 01 21:00:23.807654 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:23.807529 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7369891a-d9d2-42f1-8165-f817e9c00984-kube-api-access-cr2mg\" (UniqueName: \"kubernetes.io/projected/7369891a-d9d2-42f1-8165-f817e9c00984-kube-api-access-cr2mg\") pod \"7369891a-d9d2-42f1-8165-f817e9c00984\" (UID: \"7369891a-d9d2-42f1-8165-f817e9c00984\") " Nov 01 21:00:23.807822 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:23.807797 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7369891a-d9d2-42f1-8165-f817e9c00984-maas-api-service-ca" pod "7369891a-d9d2-42f1-8165-f817e9c00984" (UID: "7369891a-d9d2-42f1-8165-f817e9c00984"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:00:23.808101 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:23.808083 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7369891a-d9d2-42f1-8165-f817e9c00984-maas-api-service-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:23.809666 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:23.809642 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7369891a-d9d2-42f1-8165-f817e9c00984-kube-api-access-cr2mg" pod "7369891a-d9d2-42f1-8165-f817e9c00984" (UID: "7369891a-d9d2-42f1-8165-f817e9c00984"). InnerVolumeSpecName "kube-api-access-cr2mg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:00:23.905415 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:23.905381 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892780-7pl4v"] Nov 01 21:00:23.909779 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:23.909751 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0f5aabc-58f2-4166-af2d-39aa67592cb1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0f5aabc-58f2-4166-af2d-39aa67592cb1-maas-api-service-ca\") pod \"d0f5aabc-58f2-4166-af2d-39aa67592cb1\" (UID: \"d0f5aabc-58f2-4166-af2d-39aa67592cb1\") " Nov 01 21:00:23.909988 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:23.909970 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0f5aabc-58f2-4166-af2d-39aa67592cb1-kube-api-access-w65xg\" (UniqueName: \"kubernetes.io/projected/d0f5aabc-58f2-4166-af2d-39aa67592cb1-kube-api-access-w65xg\") pod \"d0f5aabc-58f2-4166-af2d-39aa67592cb1\" (UID: \"d0f5aabc-58f2-4166-af2d-39aa67592cb1\") " Nov 01 21:00:23.910210 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:23.910184 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0f5aabc-58f2-4166-af2d-39aa67592cb1-maas-api-service-ca" pod "d0f5aabc-58f2-4166-af2d-39aa67592cb1" (UID: "d0f5aabc-58f2-4166-af2d-39aa67592cb1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:00:23.910547 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:23.910524 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cr2mg\" (UniqueName: \"kubernetes.io/projected/7369891a-d9d2-42f1-8165-f817e9c00984-kube-api-access-cr2mg\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:23.910547 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:23.910549 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0f5aabc-58f2-4166-af2d-39aa67592cb1-maas-api-service-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:23.912147 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:23.912127 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0f5aabc-58f2-4166-af2d-39aa67592cb1-kube-api-access-w65xg" pod "d0f5aabc-58f2-4166-af2d-39aa67592cb1" (UID: "d0f5aabc-58f2-4166-af2d-39aa67592cb1"). InnerVolumeSpecName "kube-api-access-w65xg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:00:24.012396 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:24.012361 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w65xg\" (UniqueName: \"kubernetes.io/projected/d0f5aabc-58f2-4166-af2d-39aa67592cb1-kube-api-access-w65xg\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:24.051145 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:24.051115 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892780-7pl4v" Nov 01 21:00:24.117316 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:24.117229 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892780-p6gfx"] Nov 01 21:00:24.124695 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:24.124660 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892780-p6gfx"] Nov 01 21:00:24.215091 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:24.215055 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da6f250e-6516-4a04-a37a-f30c02ec682b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da6f250e-6516-4a04-a37a-f30c02ec682b-maas-api-service-ca\") pod \"da6f250e-6516-4a04-a37a-f30c02ec682b\" (UID: \"da6f250e-6516-4a04-a37a-f30c02ec682b\") " Nov 01 21:00:24.215379 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:24.215362 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da6f250e-6516-4a04-a37a-f30c02ec682b-kube-api-access-8hdnk\" (UniqueName: \"kubernetes.io/projected/da6f250e-6516-4a04-a37a-f30c02ec682b-kube-api-access-8hdnk\") pod \"da6f250e-6516-4a04-a37a-f30c02ec682b\" (UID: \"da6f250e-6516-4a04-a37a-f30c02ec682b\") " Nov 01 21:00:24.215552 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:24.215521 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da6f250e-6516-4a04-a37a-f30c02ec682b-maas-api-service-ca" pod "da6f250e-6516-4a04-a37a-f30c02ec682b" (UID: "da6f250e-6516-4a04-a37a-f30c02ec682b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:00:24.216045 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:24.215986 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da6f250e-6516-4a04-a37a-f30c02ec682b-maas-api-service-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:24.217571 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:24.217539 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da6f250e-6516-4a04-a37a-f30c02ec682b-kube-api-access-8hdnk" pod "da6f250e-6516-4a04-a37a-f30c02ec682b" (UID: "da6f250e-6516-4a04-a37a-f30c02ec682b"). InnerVolumeSpecName "kube-api-access-8hdnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:00:24.233541 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:24.233511 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892780-jmsct"] Nov 01 21:00:24.241364 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:24.241342 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892780-jmsct"] Nov 01 21:00:24.317764 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:24.317725 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hdnk\" (UniqueName: \"kubernetes.io/projected/da6f250e-6516-4a04-a37a-f30c02ec682b-kube-api-access-8hdnk\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:24.469219 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:24.469129 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892780-q4ns9"] Nov 01 21:00:24.473409 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:24.473375 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892780-q4ns9"] Nov 01 21:00:24.478037 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:24.478015 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b27e66-29892780-hvgx4" Nov 01 21:00:24.478037 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:24.478020 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b27e66-29892780-hvgx4" event={"ID":"7369891a-d9d2-42f1-8165-f817e9c00984","Type":"ContainerDied","Data":"ac161d6640924630e39871b79cf7498dcde038e0511fe40cf9b26ce694131483"} Nov 01 21:00:24.478205 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:24.478077 4484 scope.go:122] "RemoveContainer" containerID="e7c802121a4e393414de6bc15052257c2221bab158bccd11d8c6357063ee278a" Nov 01 21:00:24.479563 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:24.479540 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892780-9tm2f" Nov 01 21:00:24.479659 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:24.479586 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892780-9tm2f" event={"ID":"d0f5aabc-58f2-4166-af2d-39aa67592cb1","Type":"ContainerDied","Data":"1661dc1e34f2174cccb6ecbbf7541d5e0f8ead6921d63bab91b21052597366a7"} Nov 01 21:00:24.482615 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:24.482587 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892780-7pl4v" Nov 01 21:00:24.482730 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:24.482627 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892780-7pl4v" event={"ID":"da6f250e-6516-4a04-a37a-f30c02ec682b","Type":"ContainerDied","Data":"9221b19be74cc18e50987c0073c0fd8442013d6701c47e33ae190cc9f3975ead"} Nov 01 21:00:24.489636 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:24.489617 4484 scope.go:122] "RemoveContainer" containerID="ebd29d2f86acfcbbbeb047888e6e5f6f91f1c0a2da89f7b9cb0579f26b166341" Nov 01 21:00:24.501452 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:24.501427 4484 scope.go:122] "RemoveContainer" containerID="e4d6d7f222eebd847c2ef7e294a6f5640223540cd4189cf4c4b03c92c6dd6fe3" Nov 01 21:00:24.570769 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:24.570734 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:24.814355 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:24.814321 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892780-9tm2f"] Nov 01 21:00:24.818870 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:24.818840 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892780-9tm2f"] Nov 01 21:00:24.940295 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:24.940259 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892780-7pl4v"] Nov 01 21:00:24.944130 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:24.944107 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892780-7pl4v"] Nov 01 21:00:25.052291 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:25.052260 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b27e66-29892780-hvgx4"] Nov 01 21:00:25.056083 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:25.056060 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b27e66-29892780-hvgx4"] Nov 01 21:00:25.374223 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:25.374193 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38c14ac7-4673-46b7-a721-652a972f3ba5" path="/var/lib/kubelet/pods/38c14ac7-4673-46b7-a721-652a972f3ba5/volumes" Nov 01 21:00:25.374703 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:25.374688 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="474d07b8-cbb2-4beb-a17f-fe445d0c9ad4" path="/var/lib/kubelet/pods/474d07b8-cbb2-4beb-a17f-fe445d0c9ad4/volumes" Nov 01 21:00:25.375119 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:25.375105 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52fb948e-15a6-400e-be44-2e88a1fb1501" path="/var/lib/kubelet/pods/52fb948e-15a6-400e-be44-2e88a1fb1501/volumes" Nov 01 21:00:25.375814 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:25.375794 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7369891a-d9d2-42f1-8165-f817e9c00984" path="/var/lib/kubelet/pods/7369891a-d9d2-42f1-8165-f817e9c00984/volumes" Nov 01 21:00:25.376246 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:25.376232 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0f5aabc-58f2-4166-af2d-39aa67592cb1" path="/var/lib/kubelet/pods/d0f5aabc-58f2-4166-af2d-39aa67592cb1/volumes" Nov 01 21:00:25.376618 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:25.376606 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da6f250e-6516-4a04-a37a-f30c02ec682b" path="/var/lib/kubelet/pods/da6f250e-6516-4a04-a37a-f30c02ec682b/volumes" Nov 01 21:00:25.377257 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:25.377243 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5aa9680-52ae-4173-875b-1097a5963fc0" path="/var/lib/kubelet/pods/e5aa9680-52ae-4173-875b-1097a5963fc0/volumes" Nov 01 21:00:26.593591 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:26.593557 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:26.826586 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:26.826531 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:27.017088 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.004550 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-dab736-7b55d47c59-lrvjr"] Nov 01 21:00:27.024054 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.023369 4484 cpu_manager.go:414] "containerMap: removing container" podUID="cfd728c4-7f6b-4197-a147-1f340427f414" containerName="cleanup" Nov 01 21:00:27.024054 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.023403 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfd728c4-7f6b-4197-a147-1f340427f414" containerName="cleanup" Nov 01 21:00:27.024054 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.023433 4484 cpu_manager.go:414] "containerMap: removing container" podUID="b19e6e75-0ef8-440d-8617-e5c71bfde0d2" containerName="cleanup" Nov 01 21:00:27.024054 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.023441 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b19e6e75-0ef8-440d-8617-e5c71bfde0d2" containerName="cleanup" Nov 01 21:00:27.024054 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.023454 4484 cpu_manager.go:414] "containerMap: removing container" podUID="da6f250e-6516-4a04-a37a-f30c02ec682b" containerName="cleanup" Nov 01 21:00:27.024054 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.023463 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da6f250e-6516-4a04-a37a-f30c02ec682b" containerName="cleanup" Nov 01 21:00:27.024054 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.023478 4484 cpu_manager.go:414] "containerMap: removing container" podUID="7369891a-d9d2-42f1-8165-f817e9c00984" containerName="cleanup" Nov 01 21:00:27.024054 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.023485 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7369891a-d9d2-42f1-8165-f817e9c00984" containerName="cleanup" Nov 01 21:00:27.024054 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.023496 4484 cpu_manager.go:414] "containerMap: removing container" podUID="38c14ac7-4673-46b7-a721-652a972f3ba5" containerName="cleanup" Nov 01 21:00:27.024054 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.023502 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38c14ac7-4673-46b7-a721-652a972f3ba5" containerName="cleanup" Nov 01 21:00:27.024054 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.023515 4484 cpu_manager.go:414] "containerMap: removing container" podUID="e5aa9680-52ae-4173-875b-1097a5963fc0" containerName="cleanup" Nov 01 21:00:27.024054 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.023522 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5aa9680-52ae-4173-875b-1097a5963fc0" containerName="cleanup" Nov 01 21:00:27.024054 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.023535 4484 cpu_manager.go:414] "containerMap: removing container" podUID="d0f5aabc-58f2-4166-af2d-39aa67592cb1" containerName="cleanup" Nov 01 21:00:27.024054 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.023542 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0f5aabc-58f2-4166-af2d-39aa67592cb1" containerName="cleanup" Nov 01 21:00:27.024054 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.023552 4484 cpu_manager.go:414] "containerMap: removing container" podUID="da6f250e-6516-4a04-a37a-f30c02ec682b" containerName="cleanup" Nov 01 21:00:27.024054 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.023558 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da6f250e-6516-4a04-a37a-f30c02ec682b" containerName="cleanup" Nov 01 21:00:27.024054 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.023574 4484 cpu_manager.go:414] "containerMap: removing container" podUID="e5aa9680-52ae-4173-875b-1097a5963fc0" containerName="cleanup" Nov 01 21:00:27.024054 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.023582 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5aa9680-52ae-4173-875b-1097a5963fc0" containerName="cleanup" Nov 01 21:00:27.024054 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.023594 4484 cpu_manager.go:414] "containerMap: removing container" podUID="7369891a-d9d2-42f1-8165-f817e9c00984" containerName="cleanup" Nov 01 21:00:27.024054 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.023601 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7369891a-d9d2-42f1-8165-f817e9c00984" containerName="cleanup" Nov 01 21:00:27.024054 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.023616 4484 cpu_manager.go:414] "containerMap: removing container" podUID="cfd728c4-7f6b-4197-a147-1f340427f414" containerName="cleanup" Nov 01 21:00:27.024054 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.023624 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfd728c4-7f6b-4197-a147-1f340427f414" containerName="cleanup" Nov 01 21:00:27.024054 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.023654 4484 cpu_manager.go:414] "containerMap: removing container" podUID="474d07b8-cbb2-4beb-a17f-fe445d0c9ad4" containerName="cleanup" Nov 01 21:00:27.024054 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.023661 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="474d07b8-cbb2-4beb-a17f-fe445d0c9ad4" containerName="cleanup" Nov 01 21:00:27.024054 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.023671 4484 cpu_manager.go:414] "containerMap: removing container" podUID="cfd728c4-7f6b-4197-a147-1f340427f414" containerName="cleanup" Nov 01 21:00:27.024054 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.023677 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfd728c4-7f6b-4197-a147-1f340427f414" containerName="cleanup" Nov 01 21:00:27.024054 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.023698 4484 cpu_manager.go:414] "containerMap: removing container" podUID="38c14ac7-4673-46b7-a721-652a972f3ba5" containerName="cleanup" Nov 01 21:00:27.024054 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.023705 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38c14ac7-4673-46b7-a721-652a972f3ba5" containerName="cleanup" Nov 01 21:00:27.024054 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.023718 4484 cpu_manager.go:414] "containerMap: removing container" podUID="474d07b8-cbb2-4beb-a17f-fe445d0c9ad4" containerName="cleanup" Nov 01 21:00:27.024054 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.023725 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="474d07b8-cbb2-4beb-a17f-fe445d0c9ad4" containerName="cleanup" Nov 01 21:00:27.024054 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.023735 4484 cpu_manager.go:414] "containerMap: removing container" podUID="52fb948e-15a6-400e-be44-2e88a1fb1501" containerName="cleanup" Nov 01 21:00:27.024054 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.023742 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52fb948e-15a6-400e-be44-2e88a1fb1501" containerName="cleanup" Nov 01 21:00:27.024054 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.023769 4484 cpu_manager.go:414] "containerMap: removing container" podUID="52fb948e-15a6-400e-be44-2e88a1fb1501" containerName="cleanup" Nov 01 21:00:27.024054 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.023781 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52fb948e-15a6-400e-be44-2e88a1fb1501" containerName="cleanup" Nov 01 21:00:27.024054 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.023800 4484 cpu_manager.go:414] "containerMap: removing container" podUID="d0f5aabc-58f2-4166-af2d-39aa67592cb1" containerName="cleanup" Nov 01 21:00:27.024054 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.023809 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0f5aabc-58f2-4166-af2d-39aa67592cb1" containerName="cleanup" Nov 01 21:00:27.024054 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.023820 4484 cpu_manager.go:414] "containerMap: removing container" podUID="b19e6e75-0ef8-440d-8617-e5c71bfde0d2" containerName="cleanup" Nov 01 21:00:27.024054 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.023827 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b19e6e75-0ef8-440d-8617-e5c71bfde0d2" containerName="cleanup" Nov 01 21:00:27.024054 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.023837 4484 cpu_manager.go:414] "containerMap: removing container" podUID="474d07b8-cbb2-4beb-a17f-fe445d0c9ad4" containerName="cleanup" Nov 01 21:00:27.024054 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.023844 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="474d07b8-cbb2-4beb-a17f-fe445d0c9ad4" containerName="cleanup" Nov 01 21:00:27.024054 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.024073 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="b19e6e75-0ef8-440d-8617-e5c71bfde0d2" containerName="cleanup" Nov 01 21:00:27.024054 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.024086 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="474d07b8-cbb2-4beb-a17f-fe445d0c9ad4" containerName="cleanup" Nov 01 21:00:27.026964 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.024099 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="52fb948e-15a6-400e-be44-2e88a1fb1501" containerName="cleanup" Nov 01 21:00:27.026964 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.024112 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfd728c4-7f6b-4197-a147-1f340427f414" containerName="cleanup" Nov 01 21:00:27.026964 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.024122 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="b19e6e75-0ef8-440d-8617-e5c71bfde0d2" containerName="cleanup" Nov 01 21:00:27.026964 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.024130 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="da6f250e-6516-4a04-a37a-f30c02ec682b" containerName="cleanup" Nov 01 21:00:27.026964 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.024138 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="7369891a-d9d2-42f1-8165-f817e9c00984" containerName="cleanup" Nov 01 21:00:27.026964 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.024147 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="38c14ac7-4673-46b7-a721-652a972f3ba5" containerName="cleanup" Nov 01 21:00:27.026964 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.024156 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5aa9680-52ae-4173-875b-1097a5963fc0" containerName="cleanup" Nov 01 21:00:27.026964 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.024165 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0f5aabc-58f2-4166-af2d-39aa67592cb1" containerName="cleanup" Nov 01 21:00:27.026964 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.024172 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="da6f250e-6516-4a04-a37a-f30c02ec682b" containerName="cleanup" Nov 01 21:00:27.026964 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.024183 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5aa9680-52ae-4173-875b-1097a5963fc0" containerName="cleanup" Nov 01 21:00:27.026964 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.024192 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="7369891a-d9d2-42f1-8165-f817e9c00984" containerName="cleanup" Nov 01 21:00:27.026964 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.024204 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="da6f250e-6516-4a04-a37a-f30c02ec682b" containerName="cleanup" Nov 01 21:00:27.026964 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.024214 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfd728c4-7f6b-4197-a147-1f340427f414" containerName="cleanup" Nov 01 21:00:27.026964 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.024223 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="38c14ac7-4673-46b7-a721-652a972f3ba5" containerName="cleanup" Nov 01 21:00:27.026964 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.024232 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="474d07b8-cbb2-4beb-a17f-fe445d0c9ad4" containerName="cleanup" Nov 01 21:00:27.026964 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.024243 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="52fb948e-15a6-400e-be44-2e88a1fb1501" containerName="cleanup" Nov 01 21:00:27.026964 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.024253 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0f5aabc-58f2-4166-af2d-39aa67592cb1" containerName="cleanup" Nov 01 21:00:27.044350 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.030733 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-dab736-7b55d47c59-lrvjr" Nov 01 21:00:27.277621 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.111903 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-dab736-dockercfg-v6xb4\"" Nov 01 21:00:27.277621 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.112016 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-dab736\"" Nov 01 21:00:27.277621 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.217849 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/749e3d84-a7c5-4340-bda6-d16818decb9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-dab736-7b55d47c59-lrvjr\" (UID: \"749e3d84-a7c5-4340-bda6-d16818decb9b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-dab736-7b55d47c59-lrvjr" Nov 01 21:00:27.277621 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.218025 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zswjn\" (UniqueName: \"kubernetes.io/projected/749e3d84-a7c5-4340-bda6-d16818decb9b-kube-api-access-zswjn\") pod \"payload-pre-processing-e2e-shared-b-w4-dab736-7b55d47c59-lrvjr\" (UID: \"749e3d84-a7c5-4340-bda6-d16818decb9b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-dab736-7b55d47c59-lrvjr" Nov 01 21:00:27.347771 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.329204 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/749e3d84-a7c5-4340-bda6-d16818decb9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-dab736-7b55d47c59-lrvjr\" (UID: \"749e3d84-a7c5-4340-bda6-d16818decb9b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-dab736-7b55d47c59-lrvjr" Nov 01 21:00:27.347771 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.329413 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zswjn\" (UniqueName: \"kubernetes.io/projected/749e3d84-a7c5-4340-bda6-d16818decb9b-kube-api-access-zswjn\") pod \"payload-pre-processing-e2e-shared-b-w4-dab736-7b55d47c59-lrvjr\" (UID: \"749e3d84-a7c5-4340-bda6-d16818decb9b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-dab736-7b55d47c59-lrvjr" Nov 01 21:00:27.347771 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.330740 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/749e3d84-a7c5-4340-bda6-d16818decb9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-dab736-7b55d47c59-lrvjr\" (UID: \"749e3d84-a7c5-4340-bda6-d16818decb9b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-dab736-7b55d47c59-lrvjr" Nov 01 21:00:27.419125 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.413273 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zswjn\" (UniqueName: \"kubernetes.io/projected/749e3d84-a7c5-4340-bda6-d16818decb9b-kube-api-access-zswjn\") pod \"payload-pre-processing-e2e-shared-b-w4-dab736-7b55d47c59-lrvjr\" (UID: \"749e3d84-a7c5-4340-bda6-d16818decb9b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-dab736-7b55d47c59-lrvjr" Nov 01 21:00:27.423497 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.419521 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-dab736-7b55d47c59-lrvjr"] Nov 01 21:00:27.423497 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.419588 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-dab736-5b68f8cd7c-947ht"] Nov 01 21:00:27.423497 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.420782 4484 cpu_manager.go:414] "containerMap: removing container" podUID="b19e6e75-0ef8-440d-8617-e5c71bfde0d2" containerName="cleanup" Nov 01 21:00:27.423497 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.420800 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b19e6e75-0ef8-440d-8617-e5c71bfde0d2" containerName="cleanup" Nov 01 21:00:27.423497 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.420815 4484 cpu_manager.go:414] "containerMap: removing container" podUID="7369891a-d9d2-42f1-8165-f817e9c00984" containerName="cleanup" Nov 01 21:00:27.423497 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.420824 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7369891a-d9d2-42f1-8165-f817e9c00984" containerName="cleanup" Nov 01 21:00:27.423497 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.420834 4484 cpu_manager.go:414] "containerMap: removing container" podUID="38c14ac7-4673-46b7-a721-652a972f3ba5" containerName="cleanup" Nov 01 21:00:27.423497 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.420841 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38c14ac7-4673-46b7-a721-652a972f3ba5" containerName="cleanup" Nov 01 21:00:27.423497 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.420854 4484 cpu_manager.go:414] "containerMap: removing container" podUID="d0f5aabc-58f2-4166-af2d-39aa67592cb1" containerName="cleanup" Nov 01 21:00:27.423497 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.420861 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0f5aabc-58f2-4166-af2d-39aa67592cb1" containerName="cleanup" Nov 01 21:00:27.423497 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.420875 4484 cpu_manager.go:414] "containerMap: removing container" podUID="da6f250e-6516-4a04-a37a-f30c02ec682b" containerName="cleanup" Nov 01 21:00:27.423497 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.420883 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da6f250e-6516-4a04-a37a-f30c02ec682b" containerName="cleanup" Nov 01 21:00:27.423497 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.420900 4484 cpu_manager.go:414] "containerMap: removing container" podUID="e5aa9680-52ae-4173-875b-1097a5963fc0" containerName="cleanup" Nov 01 21:00:27.423497 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.420907 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5aa9680-52ae-4173-875b-1097a5963fc0" containerName="cleanup" Nov 01 21:00:27.423497 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.420953 4484 cpu_manager.go:414] "containerMap: removing container" podUID="52fb948e-15a6-400e-be44-2e88a1fb1501" containerName="cleanup" Nov 01 21:00:27.423497 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.420960 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52fb948e-15a6-400e-be44-2e88a1fb1501" containerName="cleanup" Nov 01 21:00:27.423497 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.421172 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="474d07b8-cbb2-4beb-a17f-fe445d0c9ad4" containerName="cleanup" Nov 01 21:00:27.423497 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.421187 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="52fb948e-15a6-400e-be44-2e88a1fb1501" containerName="cleanup" Nov 01 21:00:27.423497 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.421201 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfd728c4-7f6b-4197-a147-1f340427f414" containerName="cleanup" Nov 01 21:00:27.423497 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.421213 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="b19e6e75-0ef8-440d-8617-e5c71bfde0d2" containerName="cleanup" Nov 01 21:00:27.423497 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.421223 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="38c14ac7-4673-46b7-a721-652a972f3ba5" containerName="cleanup" Nov 01 21:00:27.423497 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.421234 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0f5aabc-58f2-4166-af2d-39aa67592cb1" containerName="cleanup" Nov 01 21:00:27.423497 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.421243 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5aa9680-52ae-4173-875b-1097a5963fc0" containerName="cleanup" Nov 01 21:00:27.423497 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.421253 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="7369891a-d9d2-42f1-8165-f817e9c00984" containerName="cleanup" Nov 01 21:00:27.434256 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.434224 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-dab736-5b68f8cd7c-947ht"] Nov 01 21:00:27.434419 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.434363 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-dab736-5b68f8cd7c-947ht" Nov 01 21:00:27.534200 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.534116 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltvfn\" (UniqueName: \"kubernetes.io/projected/a1da92eb-3960-46cf-836d-2b45d4aa2497-kube-api-access-ltvfn\") pod \"payload-processing-e2e-shared-b-w4-dab736-5b68f8cd7c-947ht\" (UID: \"a1da92eb-3960-46cf-836d-2b45d4aa2497\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-dab736-5b68f8cd7c-947ht" Nov 01 21:00:27.534200 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.534178 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1da92eb-3960-46cf-836d-2b45d4aa2497-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-dab736-5b68f8cd7c-947ht\" (UID: \"a1da92eb-3960-46cf-836d-2b45d4aa2497\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-dab736-5b68f8cd7c-947ht" Nov 01 21:00:27.635625 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.635585 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltvfn\" (UniqueName: \"kubernetes.io/projected/a1da92eb-3960-46cf-836d-2b45d4aa2497-kube-api-access-ltvfn\") pod \"payload-processing-e2e-shared-b-w4-dab736-5b68f8cd7c-947ht\" (UID: \"a1da92eb-3960-46cf-836d-2b45d4aa2497\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-dab736-5b68f8cd7c-947ht" Nov 01 21:00:27.636062 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.635646 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1da92eb-3960-46cf-836d-2b45d4aa2497-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-dab736-5b68f8cd7c-947ht\" (UID: \"a1da92eb-3960-46cf-836d-2b45d4aa2497\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-dab736-5b68f8cd7c-947ht" Nov 01 21:00:27.636406 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.636386 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1da92eb-3960-46cf-836d-2b45d4aa2497-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-dab736-5b68f8cd7c-947ht\" (UID: \"a1da92eb-3960-46cf-836d-2b45d4aa2497\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-dab736-5b68f8cd7c-947ht" Nov 01 21:00:27.694378 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.694339 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltvfn\" (UniqueName: \"kubernetes.io/projected/a1da92eb-3960-46cf-836d-2b45d4aa2497-kube-api-access-ltvfn\") pod \"payload-processing-e2e-shared-b-w4-dab736-5b68f8cd7c-947ht\" (UID: \"a1da92eb-3960-46cf-836d-2b45d4aa2497\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-dab736-5b68f8cd7c-947ht" Nov 01 21:00:27.708300 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.708269 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-dab736-7b55d47c59-lrvjr" Nov 01 21:00:27.735065 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.735038 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:27.769112 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:27.767694 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-dab736-5b68f8cd7c-947ht" Nov 01 21:00:28.171359 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:28.171315 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-dab736-7b55d47c59-lrvjr"] Nov 01 21:00:28.171359 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:00:28.171356 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod749e3d84_a7c5_4340_bda6_d16818decb9b.slice/crio-48ae3aa5cbfec8006316556bad1b24ae94d3b3632497f44e385d790f15234462 WatchSource:0}: Error finding container 48ae3aa5cbfec8006316556bad1b24ae94d3b3632497f44e385d790f15234462: Status 404 returned error can't find the container with id 48ae3aa5cbfec8006316556bad1b24ae94d3b3632497f44e385d790f15234462 Nov 01 21:00:28.274438 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:00:28.274405 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1da92eb_3960_46cf_836d_2b45d4aa2497.slice/crio-b2bd753ebd68c574638ca0aabfd08669ba343cb3c83de9ef32457e0b23d2c6f0 WatchSource:0}: Error finding container b2bd753ebd68c574638ca0aabfd08669ba343cb3c83de9ef32457e0b23d2c6f0: Status 404 returned error can't find the container with id b2bd753ebd68c574638ca0aabfd08669ba343cb3c83de9ef32457e0b23d2c6f0 Nov 01 21:00:28.274684 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:28.274662 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-dab736-5b68f8cd7c-947ht"] Nov 01 21:00:28.521975 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:28.521935 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-dab736-5b68f8cd7c-947ht" event={"ID":"a1da92eb-3960-46cf-836d-2b45d4aa2497","Type":"ContainerStarted","Data":"27e31e8a844e74f4ea8db2c23f6fdad7b74b7d4c9829daf2b898f54c5e4c1705"} Nov 01 21:00:28.522192 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:28.522025 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-dab736-5b68f8cd7c-947ht" event={"ID":"a1da92eb-3960-46cf-836d-2b45d4aa2497","Type":"ContainerStarted","Data":"b2bd753ebd68c574638ca0aabfd08669ba343cb3c83de9ef32457e0b23d2c6f0"} Nov 01 21:00:28.522279 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:28.522195 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-dab736-5b68f8cd7c-947ht" Nov 01 21:00:28.524029 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:28.523965 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-dab736-7b55d47c59-lrvjr" event={"ID":"749e3d84-a7c5-4340-bda6-d16818decb9b","Type":"ContainerStarted","Data":"04e1aa87c0bc3128bb580e445d6244d06b1d0e4c7f10eb5ad73137312b623123"} Nov 01 21:00:28.524160 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:28.524035 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-dab736-7b55d47c59-lrvjr" Nov 01 21:00:28.524160 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:28.524060 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-dab736-7b55d47c59-lrvjr" event={"ID":"749e3d84-a7c5-4340-bda6-d16818decb9b","Type":"ContainerStarted","Data":"48ae3aa5cbfec8006316556bad1b24ae94d3b3632497f44e385d790f15234462"} Nov 01 21:00:28.641968 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:28.640447 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-dab736-5b68f8cd7c-947ht" podStartSLOduration=1.6404301220000002 podStartE2EDuration="1.640430122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:00:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:00:28.637041467 +0000 UTC m=+1779.414681603" watchObservedRunningTime="2026-11-01 21:00:28.640430122 +0000 UTC m=+1779.418070258" Nov 01 21:00:32.233450 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:32.233397 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-dab736-7b55d47c59-lrvjr" podStartSLOduration=6.233384083 podStartE2EDuration="6.233384083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:00:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:00:28.767890126 +0000 UTC m=+1779.545530262" watchObservedRunningTime="2026-11-01 21:00:32.233384083 +0000 UTC m=+1783.011024217" Nov 01 21:00:32.233892 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:32.233528 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9a7e213-42tzt"] Nov 01 21:00:32.237973 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:32.237956 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a7e213-42tzt" Nov 01 21:00:32.295105 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:32.295071 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9a7e213-42tzt"] Nov 01 21:00:32.325282 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:32.325245 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/752bf457-c713-429d-a6d2-9550a30c3efd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9a7e213-42tzt\" (UID: \"752bf457-c713-429d-a6d2-9550a30c3efd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a7e213-42tzt" Nov 01 21:00:32.325485 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:32.325418 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/752bf457-c713-429d-a6d2-9550a30c3efd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9a7e213-42tzt\" (UID: \"752bf457-c713-429d-a6d2-9550a30c3efd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a7e213-42tzt" Nov 01 21:00:32.325574 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:32.325554 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njts4\" (UniqueName: \"kubernetes.io/projected/752bf457-c713-429d-a6d2-9550a30c3efd-kube-api-access-njts4\") pod \"maas-api-revoke-sub-c9a7e213-42tzt\" (UID: \"752bf457-c713-429d-a6d2-9550a30c3efd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a7e213-42tzt" Nov 01 21:00:32.340145 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:32.340116 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:00:32.340145 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:32.340120 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:00:32.341803 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:32.341781 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:00:32.341898 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:32.341841 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:00:32.427307 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:32.427276 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njts4\" (UniqueName: \"kubernetes.io/projected/752bf457-c713-429d-a6d2-9550a30c3efd-kube-api-access-njts4\") pod \"maas-api-revoke-sub-c9a7e213-42tzt\" (UID: \"752bf457-c713-429d-a6d2-9550a30c3efd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a7e213-42tzt" Nov 01 21:00:32.427483 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:32.427455 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/752bf457-c713-429d-a6d2-9550a30c3efd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9a7e213-42tzt\" (UID: \"752bf457-c713-429d-a6d2-9550a30c3efd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a7e213-42tzt" Nov 01 21:00:32.427543 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:32.427513 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/752bf457-c713-429d-a6d2-9550a30c3efd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9a7e213-42tzt\" (UID: \"752bf457-c713-429d-a6d2-9550a30c3efd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a7e213-42tzt" Nov 01 21:00:32.428159 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:32.428131 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/752bf457-c713-429d-a6d2-9550a30c3efd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9a7e213-42tzt\" (UID: \"752bf457-c713-429d-a6d2-9550a30c3efd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a7e213-42tzt" Nov 01 21:00:32.429812 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:32.429794 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/752bf457-c713-429d-a6d2-9550a30c3efd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9a7e213-42tzt\" (UID: \"752bf457-c713-429d-a6d2-9550a30c3efd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a7e213-42tzt" Nov 01 21:00:32.445789 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:32.445758 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-29baa8-6c565db695-5m42b" Nov 01 21:00:32.484785 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:32.484714 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njts4\" (UniqueName: \"kubernetes.io/projected/752bf457-c713-429d-a6d2-9550a30c3efd-kube-api-access-njts4\") pod \"maas-api-revoke-sub-c9a7e213-42tzt\" (UID: \"752bf457-c713-429d-a6d2-9550a30c3efd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a7e213-42tzt" Nov 01 21:00:32.553282 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:32.553245 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a7e213-42tzt" Nov 01 21:00:32.944757 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:00:32.944720 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod752bf457_c713_429d_a6d2_9550a30c3efd.slice/crio-e98dbdff89ab9ead0eb920bd292c0608c4cef4f1e216a06ec1567a7223bfbb2b WatchSource:0}: Error finding container e98dbdff89ab9ead0eb920bd292c0608c4cef4f1e216a06ec1567a7223bfbb2b: Status 404 returned error can't find the container with id e98dbdff89ab9ead0eb920bd292c0608c4cef4f1e216a06ec1567a7223bfbb2b Nov 01 21:00:32.944907 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:32.944878 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9a7e213-42tzt"] Nov 01 21:00:33.021547 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:33.021517 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:33.469568 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:33.469543 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-29baa8-6bbff57b65-6djp8" Nov 01 21:00:33.565799 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:33.565704 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a7e213-42tzt" event={"ID":"752bf457-c713-429d-a6d2-9550a30c3efd","Type":"ContainerStarted","Data":"38608d0602c90d2bf9f6e6c0ccba139cdcef2fad47a302f59fdcd10933b1480c"} Nov 01 21:00:33.565799 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:33.565767 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a7e213-42tzt" event={"ID":"752bf457-c713-429d-a6d2-9550a30c3efd","Type":"ContainerStarted","Data":"e98dbdff89ab9ead0eb920bd292c0608c4cef4f1e216a06ec1567a7223bfbb2b"} Nov 01 21:00:33.706273 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:33.705145 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a7e213-42tzt" podStartSLOduration=1.7051212169999999 podStartE2EDuration="1.705121217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:00:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:00:33.702821245 +0000 UTC m=+1784.480461381" watchObservedRunningTime="2026-11-01 21:00:33.705121217 +0000 UTC m=+1784.482761353" Nov 01 21:00:34.578083 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:34.578043 4484 generic.go:356] "Generic (PLEG): container finished" podID="752bf457-c713-429d-a6d2-9550a30c3efd" containerID="38608d0602c90d2bf9f6e6c0ccba139cdcef2fad47a302f59fdcd10933b1480c" exitCode=0 Nov 01 21:00:34.578498 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:34.578140 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a7e213-42tzt" event={"ID":"752bf457-c713-429d-a6d2-9550a30c3efd","Type":"ContainerDied","Data":"38608d0602c90d2bf9f6e6c0ccba139cdcef2fad47a302f59fdcd10933b1480c"} Nov 01 21:00:35.781393 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:35.781368 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a7e213-42tzt" Nov 01 21:00:35.796386 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:35.796357 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/752bf457-c713-429d-a6d2-9550a30c3efd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/752bf457-c713-429d-a6d2-9550a30c3efd-maas-api-openshift-ca\") pod \"752bf457-c713-429d-a6d2-9550a30c3efd\" (UID: \"752bf457-c713-429d-a6d2-9550a30c3efd\") " Nov 01 21:00:35.796552 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:35.796506 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/752bf457-c713-429d-a6d2-9550a30c3efd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/752bf457-c713-429d-a6d2-9550a30c3efd-maas-api-xks-ca\") pod \"752bf457-c713-429d-a6d2-9550a30c3efd\" (UID: \"752bf457-c713-429d-a6d2-9550a30c3efd\") " Nov 01 21:00:35.796618 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:35.796606 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/752bf457-c713-429d-a6d2-9550a30c3efd-kube-api-access-njts4\" (UniqueName: \"kubernetes.io/projected/752bf457-c713-429d-a6d2-9550a30c3efd-kube-api-access-njts4\") pod \"752bf457-c713-429d-a6d2-9550a30c3efd\" (UID: \"752bf457-c713-429d-a6d2-9550a30c3efd\") " Nov 01 21:00:35.796759 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:35.796734 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/752bf457-c713-429d-a6d2-9550a30c3efd-maas-api-openshift-ca" pod "752bf457-c713-429d-a6d2-9550a30c3efd" (UID: "752bf457-c713-429d-a6d2-9550a30c3efd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:00:35.797255 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:35.797234 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/752bf457-c713-429d-a6d2-9550a30c3efd-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:35.798558 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:35.798533 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/752bf457-c713-429d-a6d2-9550a30c3efd-maas-api-xks-ca" pod "752bf457-c713-429d-a6d2-9550a30c3efd" (UID: "752bf457-c713-429d-a6d2-9550a30c3efd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:00:35.798699 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:35.798680 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/752bf457-c713-429d-a6d2-9550a30c3efd-kube-api-access-njts4" pod "752bf457-c713-429d-a6d2-9550a30c3efd" (UID: "752bf457-c713-429d-a6d2-9550a30c3efd"). InnerVolumeSpecName "kube-api-access-njts4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:00:35.899256 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:35.899162 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/752bf457-c713-429d-a6d2-9550a30c3efd-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:35.899256 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:35.899202 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njts4\" (UniqueName: \"kubernetes.io/projected/752bf457-c713-429d-a6d2-9550a30c3efd-kube-api-access-njts4\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:36.598126 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:36.598096 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a7e213-42tzt" Nov 01 21:00:36.598126 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:36.598106 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a7e213-42tzt" event={"ID":"752bf457-c713-429d-a6d2-9550a30c3efd","Type":"ContainerDied","Data":"e98dbdff89ab9ead0eb920bd292c0608c4cef4f1e216a06ec1567a7223bfbb2b"} Nov 01 21:00:36.598386 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:36.598167 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e98dbdff89ab9ead0eb920bd292c0608c4cef4f1e216a06ec1567a7223bfbb2b" Nov 01 21:00:37.839970 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:37.839936 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:39.535111 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:39.535079 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-dab736-7b55d47c59-lrvjr" Nov 01 21:00:39.535572 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:39.535155 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-dab736-5b68f8cd7c-947ht" Nov 01 21:00:42.758625 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:42.758486 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:43.132473 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:43.132438 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:45.768571 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:00:45.768535 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:00:46.535592 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:46.535251 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:47.178433 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:47.178319 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:47.625591 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:47.625556 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z"] Nov 01 21:00:47.626453 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:47.626434 4484 cpu_manager.go:414] "containerMap: removing container" podUID="752bf457-c713-429d-a6d2-9550a30c3efd" containerName="revoke-keys" Nov 01 21:00:47.626505 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:47.626454 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="752bf457-c713-429d-a6d2-9550a30c3efd" containerName="revoke-keys" Nov 01 21:00:47.626623 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:47.626613 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="752bf457-c713-429d-a6d2-9550a30c3efd" containerName="revoke-keys" Nov 01 21:00:47.629263 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:47.629244 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z" Nov 01 21:00:47.693414 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:47.690397 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21a7e141-0aee-418a-99ca-1b5c3993c303-credential-socket\") pod \"e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z\" (UID: \"21a7e141-0aee-418a-99ca-1b5c3993c303\") " pod="openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z" Nov 01 21:00:47.693414 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:47.690526 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21a7e141-0aee-418a-99ca-1b5c3993c303-workload-certs\") pod \"e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z\" (UID: \"21a7e141-0aee-418a-99ca-1b5c3993c303\") " pod="openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z" Nov 01 21:00:47.693414 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:47.690692 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21a7e141-0aee-418a-99ca-1b5c3993c303-istio-token\") pod \"e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z\" (UID: \"21a7e141-0aee-418a-99ca-1b5c3993c303\") " pod="openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z" Nov 01 21:00:47.693414 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:47.690766 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21a7e141-0aee-418a-99ca-1b5c3993c303-istio-podinfo\") pod \"e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z\" (UID: \"21a7e141-0aee-418a-99ca-1b5c3993c303\") " pod="openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z" Nov 01 21:00:47.693414 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:47.690810 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21a7e141-0aee-418a-99ca-1b5c3993c303-workload-socket\") pod \"e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z\" (UID: \"21a7e141-0aee-418a-99ca-1b5c3993c303\") " pod="openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z" Nov 01 21:00:47.693414 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:47.690848 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21a7e141-0aee-418a-99ca-1b5c3993c303-istio-envoy\") pod \"e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z\" (UID: \"21a7e141-0aee-418a-99ca-1b5c3993c303\") " pod="openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z" Nov 01 21:00:47.693414 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:47.690931 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21a7e141-0aee-418a-99ca-1b5c3993c303-istiod-ca-cert\") pod \"e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z\" (UID: \"21a7e141-0aee-418a-99ca-1b5c3993c303\") " pod="openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z" Nov 01 21:00:47.693414 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:47.691137 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21a7e141-0aee-418a-99ca-1b5c3993c303-istio-data\") pod \"e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z\" (UID: \"21a7e141-0aee-418a-99ca-1b5c3993c303\") " pod="openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z" Nov 01 21:00:47.693414 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:47.691234 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs5qf\" (UniqueName: \"kubernetes.io/projected/21a7e141-0aee-418a-99ca-1b5c3993c303-kube-api-access-gs5qf\") pod \"e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z\" (UID: \"21a7e141-0aee-418a-99ca-1b5c3993c303\") " pod="openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z" Nov 01 21:00:47.718022 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:47.715310 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-d08fd0-openshift-default-dockercfg-ccmzq\"" Nov 01 21:00:47.752718 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:47.752666 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:47.815142 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:47.801722 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21a7e141-0aee-418a-99ca-1b5c3993c303-istiod-ca-cert\") pod \"e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z\" (UID: \"21a7e141-0aee-418a-99ca-1b5c3993c303\") " pod="openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z" Nov 01 21:00:47.815531 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:47.815463 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21a7e141-0aee-418a-99ca-1b5c3993c303-istio-data\") pod \"e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z\" (UID: \"21a7e141-0aee-418a-99ca-1b5c3993c303\") " pod="openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z" Nov 01 21:00:47.815819 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:47.815800 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs5qf\" (UniqueName: \"kubernetes.io/projected/21a7e141-0aee-418a-99ca-1b5c3993c303-kube-api-access-gs5qf\") pod \"e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z\" (UID: \"21a7e141-0aee-418a-99ca-1b5c3993c303\") " pod="openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z" Nov 01 21:00:47.817407 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:47.816470 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z"] Nov 01 21:00:47.817407 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:47.810522 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21a7e141-0aee-418a-99ca-1b5c3993c303-istiod-ca-cert\") pod \"e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z\" (UID: \"21a7e141-0aee-418a-99ca-1b5c3993c303\") " pod="openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z" Nov 01 21:00:47.817407 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:47.816809 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21a7e141-0aee-418a-99ca-1b5c3993c303-istio-data\") pod \"e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z\" (UID: \"21a7e141-0aee-418a-99ca-1b5c3993c303\") " pod="openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z" Nov 01 21:00:47.817731 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:47.817710 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21a7e141-0aee-418a-99ca-1b5c3993c303-credential-socket\") pod \"e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z\" (UID: \"21a7e141-0aee-418a-99ca-1b5c3993c303\") " pod="openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z" Nov 01 21:00:47.818077 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:47.818055 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21a7e141-0aee-418a-99ca-1b5c3993c303-workload-certs\") pod \"e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z\" (UID: \"21a7e141-0aee-418a-99ca-1b5c3993c303\") " pod="openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z" Nov 01 21:00:47.818354 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:47.818337 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21a7e141-0aee-418a-99ca-1b5c3993c303-istio-token\") pod \"e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z\" (UID: \"21a7e141-0aee-418a-99ca-1b5c3993c303\") " pod="openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z" Nov 01 21:00:47.819413 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:47.819388 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21a7e141-0aee-418a-99ca-1b5c3993c303-istio-podinfo\") pod \"e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z\" (UID: \"21a7e141-0aee-418a-99ca-1b5c3993c303\") " pod="openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z" Nov 01 21:00:47.819582 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:47.819564 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21a7e141-0aee-418a-99ca-1b5c3993c303-workload-socket\") pod \"e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z\" (UID: \"21a7e141-0aee-418a-99ca-1b5c3993c303\") " pod="openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z" Nov 01 21:00:47.819698 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:47.819683 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21a7e141-0aee-418a-99ca-1b5c3993c303-istio-envoy\") pod \"e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z\" (UID: \"21a7e141-0aee-418a-99ca-1b5c3993c303\") " pod="openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z" Nov 01 21:00:47.821187 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:47.820416 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21a7e141-0aee-418a-99ca-1b5c3993c303-credential-socket\") pod \"e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z\" (UID: \"21a7e141-0aee-418a-99ca-1b5c3993c303\") " pod="openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z" Nov 01 21:00:47.821336 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:47.820645 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21a7e141-0aee-418a-99ca-1b5c3993c303-workload-certs\") pod \"e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z\" (UID: \"21a7e141-0aee-418a-99ca-1b5c3993c303\") " pod="openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z" Nov 01 21:00:47.821876 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:47.821854 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21a7e141-0aee-418a-99ca-1b5c3993c303-workload-socket\") pod \"e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z\" (UID: \"21a7e141-0aee-418a-99ca-1b5c3993c303\") " pod="openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z" Nov 01 21:00:47.833279 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:47.833171 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21a7e141-0aee-418a-99ca-1b5c3993c303-istio-podinfo\") pod \"e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z\" (UID: \"21a7e141-0aee-418a-99ca-1b5c3993c303\") " pod="openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z" Nov 01 21:00:47.893159 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:47.893054 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs5qf\" (UniqueName: \"kubernetes.io/projected/21a7e141-0aee-418a-99ca-1b5c3993c303-kube-api-access-gs5qf\") pod \"e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z\" (UID: \"21a7e141-0aee-418a-99ca-1b5c3993c303\") " pod="openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z" Nov 01 21:00:47.896610 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:47.896580 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21a7e141-0aee-418a-99ca-1b5c3993c303-istio-token\") pod \"e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z\" (UID: \"21a7e141-0aee-418a-99ca-1b5c3993c303\") " pod="openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z" Nov 01 21:00:47.952029 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:47.951915 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21a7e141-0aee-418a-99ca-1b5c3993c303-istio-envoy\") pod \"e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z\" (UID: \"21a7e141-0aee-418a-99ca-1b5c3993c303\") " pod="openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z" Nov 01 21:00:47.952447 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:47.952390 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z" Nov 01 21:00:47.993248 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:47.991274 4484 generic.go:356] "Generic (PLEG): container finished" podID="b77b5e66-a42b-4800-a34f-d415b42913d1" containerID="4afcb3d14e0d6c7c7e19fce05ade68290b13463bc828c8fb42f69d54a03804dd" exitCode=0 Nov 01 21:00:47.993248 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:47.991471 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk" event={"ID":"b77b5e66-a42b-4800-a34f-d415b42913d1","Type":"ContainerDied","Data":"4afcb3d14e0d6c7c7e19fce05ade68290b13463bc828c8fb42f69d54a03804dd"} Nov 01 21:00:47.993248 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:47.992062 4484 scope.go:122] "RemoveContainer" containerID="4afcb3d14e0d6c7c7e19fce05ade68290b13463bc828c8fb42f69d54a03804dd" Nov 01 21:00:48.043635 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:48.032455 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:48.167860 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:48.167570 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:00:48.167860 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:48.167696 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:00:48.167860 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:48.167751 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:00:48.192986 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:48.192953 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:48.684705 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:48.684679 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z"] Nov 01 21:00:48.685581 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:00:48.685550 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21a7e141_0aee_418a_99ca_1b5c3993c303.slice/crio-99fd9e282990a1cefa59ff224c3c7a2f7b0314f00833eabfe50afc6916bccabf WatchSource:0}: Error finding container 99fd9e282990a1cefa59ff224c3c7a2f7b0314f00833eabfe50afc6916bccabf: Status 404 returned error can't find the container with id 99fd9e282990a1cefa59ff224c3c7a2f7b0314f00833eabfe50afc6916bccabf Nov 01 21:00:48.688496 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:48.688461 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:00:48.688575 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:48.688534 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:00:48.688575 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:48.688566 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:00:49.003824 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:49.003786 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z" event={"ID":"21a7e141-0aee-418a-99ca-1b5c3993c303","Type":"ContainerStarted","Data":"506e6634ce972c722bec301c2d798fe34ee03b584c2dae5dbfb440fa2ef5a6b1"} Nov 01 21:00:49.004090 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:49.003849 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z" event={"ID":"21a7e141-0aee-418a-99ca-1b5c3993c303","Type":"ContainerStarted","Data":"99fd9e282990a1cefa59ff224c3c7a2f7b0314f00833eabfe50afc6916bccabf"} Nov 01 21:00:49.006166 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:49.006135 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk" event={"ID":"b77b5e66-a42b-4800-a34f-d415b42913d1","Type":"ContainerStarted","Data":"f15450a4c0bce764238a93f2cc364b4cc4b3d91b94f98e1de4b43ecafac0a2f3"} Nov 01 21:00:49.016634 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:49.016607 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z" Nov 01 21:00:49.018045 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:49.018025 4484 patch_prober.go:29] interesting pod/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Nov 01 21:00:49.018105 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:49.018070 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z" podUID="21a7e141-0aee-418a-99ca-1b5c3993c303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Nov 01 21:00:49.125538 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:49.125377 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z" podStartSLOduration=2.125359217 podStartE2EDuration="2.125359217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:00:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:00:49.121312894 +0000 UTC m=+1799.898953043" watchObservedRunningTime="2026-11-01 21:00:49.125359217 +0000 UTC m=+1799.902999350" Nov 01 21:00:49.873748 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:49.873713 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk" Nov 01 21:00:49.874206 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:49.873915 4484 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Nov 01 21:00:49.874206 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:49.873976 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk" podUID="b77b5e66-a42b-4800-a34f-d415b42913d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Nov 01 21:00:50.035007 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:50.034968 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z" Nov 01 21:00:50.036195 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:50.036167 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z" Nov 01 21:00:50.036576 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:50.036562 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z" Nov 01 21:00:50.873499 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:50.873464 4484 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Nov 01 21:00:50.873666 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:50.873514 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk" podUID="b77b5e66-a42b-4800-a34f-d415b42913d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Nov 01 21:00:51.874046 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:51.874012 4484 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Nov 01 21:00:51.874493 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:51.874063 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk" podUID="b77b5e66-a42b-4800-a34f-d415b42913d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Nov 01 21:00:52.032062 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:52.032028 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:52.225542 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:52.225455 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:52.576792 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:52.576755 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:52.873496 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:52.873406 4484 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Nov 01 21:00:52.873496 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:52.873474 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk" podUID="b77b5e66-a42b-4800-a34f-d415b42913d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Nov 01 21:00:53.377504 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:53.377466 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:53.681341 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:53.681159 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:53.873242 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:53.873209 4484 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Nov 01 21:00:53.873419 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:53.873274 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk" podUID="b77b5e66-a42b-4800-a34f-d415b42913d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Nov 01 21:00:54.435018 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:54.432895 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:54.874382 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:54.874349 4484 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Nov 01 21:00:54.874566 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:54.874415 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk" podUID="b77b5e66-a42b-4800-a34f-d415b42913d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Nov 01 21:00:54.973023 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:54.971268 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:55.264422 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:55.264393 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:55.517925 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:55.517834 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:55.667917 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:55.667845 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d08fd0-84d7f997dc-6zrkr"] Nov 01 21:00:55.686892 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:55.686819 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d08fd0-84d7f997dc-6zrkr" Nov 01 21:00:55.765380 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:55.765345 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-d08fd0\"" Nov 01 21:00:55.765864 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:55.765840 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-d08fd0-dockercfg-s6r86\"" Nov 01 21:00:55.793068 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:55.786131 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d08fd0-84d7f997dc-6zrkr"] Nov 01 21:00:55.873262 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:55.868618 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csts4\" (UniqueName: \"kubernetes.io/projected/2f7a17d2-43cc-4001-814a-8446e8054743-kube-api-access-csts4\") pod \"payload-pre-processing-e2e-api-b-d08fd0-84d7f997dc-6zrkr\" (UID: \"2f7a17d2-43cc-4001-814a-8446e8054743\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d08fd0-84d7f997dc-6zrkr" Nov 01 21:00:55.873262 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:55.869071 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f7a17d2-43cc-4001-814a-8446e8054743-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d08fd0-84d7f997dc-6zrkr\" (UID: \"2f7a17d2-43cc-4001-814a-8446e8054743\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d08fd0-84d7f997dc-6zrkr" Nov 01 21:00:55.895042 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:55.891452 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk" Nov 01 21:00:55.895042 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:55.893231 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk" Nov 01 21:00:55.895042 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:55.894746 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk" Nov 01 21:00:56.009472 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:56.005678 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csts4\" (UniqueName: \"kubernetes.io/projected/2f7a17d2-43cc-4001-814a-8446e8054743-kube-api-access-csts4\") pod \"payload-pre-processing-e2e-api-b-d08fd0-84d7f997dc-6zrkr\" (UID: \"2f7a17d2-43cc-4001-814a-8446e8054743\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d08fd0-84d7f997dc-6zrkr" Nov 01 21:00:56.009472 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:56.006141 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f7a17d2-43cc-4001-814a-8446e8054743-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d08fd0-84d7f997dc-6zrkr\" (UID: \"2f7a17d2-43cc-4001-814a-8446e8054743\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d08fd0-84d7f997dc-6zrkr" Nov 01 21:00:56.012421 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:56.012104 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f7a17d2-43cc-4001-814a-8446e8054743-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d08fd0-84d7f997dc-6zrkr\" (UID: \"2f7a17d2-43cc-4001-814a-8446e8054743\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d08fd0-84d7f997dc-6zrkr" Nov 01 21:00:56.064376 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:56.064301 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csts4\" (UniqueName: \"kubernetes.io/projected/2f7a17d2-43cc-4001-814a-8446e8054743-kube-api-access-csts4\") pod \"payload-pre-processing-e2e-api-b-d08fd0-84d7f997dc-6zrkr\" (UID: \"2f7a17d2-43cc-4001-814a-8446e8054743\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d08fd0-84d7f997dc-6zrkr" Nov 01 21:00:56.316103 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:56.316021 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d08fd0-84d7f997dc-6zrkr" Nov 01 21:00:56.699471 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:00:56.699442 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f7a17d2_43cc_4001_814a_8446e8054743.slice/crio-73340e02325e2a06e413afa032865a6516462d0940c1610895aa10ee6e936697 WatchSource:0}: Error finding container 73340e02325e2a06e413afa032865a6516462d0940c1610895aa10ee6e936697: Status 404 returned error can't find the container with id 73340e02325e2a06e413afa032865a6516462d0940c1610895aa10ee6e936697 Nov 01 21:00:56.699856 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:56.699753 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d08fd0-84d7f997dc-6zrkr"] Nov 01 21:00:57.118426 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:57.118391 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d08fd0-84d7f997dc-6zrkr" event={"ID":"2f7a17d2-43cc-4001-814a-8446e8054743","Type":"ContainerStarted","Data":"c3ee14f5d5bc4f886ed639c213dd54da775db2a9961215e238729198c83c0dc6"} Nov 01 21:00:57.118631 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:57.118454 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d08fd0-84d7f997dc-6zrkr" event={"ID":"2f7a17d2-43cc-4001-814a-8446e8054743","Type":"ContainerStarted","Data":"73340e02325e2a06e413afa032865a6516462d0940c1610895aa10ee6e936697"} Nov 01 21:00:57.118631 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:57.118494 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d08fd0-84d7f997dc-6zrkr" Nov 01 21:00:57.239248 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:57.239196 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d08fd0-84d7f997dc-6zrkr" podStartSLOduration=2.239182553 podStartE2EDuration="2.239182553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:00:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:00:57.234308312 +0000 UTC m=+1808.011948447" watchObservedRunningTime="2026-11-01 21:00:57.239182553 +0000 UTC m=+1808.016822688" Nov 01 21:00:58.862222 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:58.862189 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:59.066597 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:59.066548 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:59.226725 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:59.226647 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:59.399689 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:00:59.399652 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:02.343325 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:02.343288 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:02.670144 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:02.633166 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:02.886919 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:02.886881 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dd840a-68df564877-hxfz7"] Nov 01 21:01:02.912017 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:02.898057 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-dd840a-68df564877-hxfz7" Nov 01 21:01:02.999374 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:02.993984 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff449990-bd5a-4b0c-bd0a-7880dbcdd90f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-dd840a-68df564877-hxfz7\" (UID: \"ff449990-bd5a-4b0c-bd0a-7880dbcdd90f\") " pod="openshift-ingress/payload-processing-e2e-api-a-dd840a-68df564877-hxfz7" Nov 01 21:01:02.999374 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:02.994184 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj4pm\" (UniqueName: \"kubernetes.io/projected/ff449990-bd5a-4b0c-bd0a-7880dbcdd90f-kube-api-access-vj4pm\") pod \"payload-processing-e2e-api-a-dd840a-68df564877-hxfz7\" (UID: \"ff449990-bd5a-4b0c-bd0a-7880dbcdd90f\") " pod="openshift-ingress/payload-processing-e2e-api-a-dd840a-68df564877-hxfz7" Nov 01 21:01:02.999374 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:02.998974 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-dd840a\"" Nov 01 21:01:02.999374 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:02.999168 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-dd840a-dockercfg-m2j7g\"" Nov 01 21:01:03.031039 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:03.030989 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dd840a-68df564877-hxfz7"] Nov 01 21:01:03.157193 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:03.157145 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff449990-bd5a-4b0c-bd0a-7880dbcdd90f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-dd840a-68df564877-hxfz7\" (UID: \"ff449990-bd5a-4b0c-bd0a-7880dbcdd90f\") " pod="openshift-ingress/payload-processing-e2e-api-a-dd840a-68df564877-hxfz7" Nov 01 21:01:03.157682 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:03.157660 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj4pm\" (UniqueName: \"kubernetes.io/projected/ff449990-bd5a-4b0c-bd0a-7880dbcdd90f-kube-api-access-vj4pm\") pod \"payload-processing-e2e-api-a-dd840a-68df564877-hxfz7\" (UID: \"ff449990-bd5a-4b0c-bd0a-7880dbcdd90f\") " pod="openshift-ingress/payload-processing-e2e-api-a-dd840a-68df564877-hxfz7" Nov 01 21:01:03.169178 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:03.169143 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff449990-bd5a-4b0c-bd0a-7880dbcdd90f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-dd840a-68df564877-hxfz7\" (UID: \"ff449990-bd5a-4b0c-bd0a-7880dbcdd90f\") " pod="openshift-ingress/payload-processing-e2e-api-a-dd840a-68df564877-hxfz7" Nov 01 21:01:03.199501 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:03.169255 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dd840a-5c9b4679f5-wspqf"] Nov 01 21:01:03.229811 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:03.229775 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj4pm\" (UniqueName: \"kubernetes.io/projected/ff449990-bd5a-4b0c-bd0a-7880dbcdd90f-kube-api-access-vj4pm\") pod \"payload-processing-e2e-api-a-dd840a-68df564877-hxfz7\" (UID: \"ff449990-bd5a-4b0c-bd0a-7880dbcdd90f\") " pod="openshift-ingress/payload-processing-e2e-api-a-dd840a-68df564877-hxfz7" Nov 01 21:01:03.232835 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:03.232802 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dd840a-5c9b4679f5-wspqf"] Nov 01 21:01:03.233056 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:03.232923 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd840a-5c9b4679f5-wspqf" Nov 01 21:01:03.270347 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:03.270311 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-dd840a-68df564877-hxfz7" Nov 01 21:01:03.369417 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:03.369352 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40f370d3-946f-4cf1-805d-f245ca48e9a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-dd840a-5c9b4679f5-wspqf\" (UID: \"40f370d3-946f-4cf1-805d-f245ca48e9a9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd840a-5c9b4679f5-wspqf" Nov 01 21:01:03.369913 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:03.369688 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j889c\" (UniqueName: \"kubernetes.io/projected/40f370d3-946f-4cf1-805d-f245ca48e9a9-kube-api-access-j889c\") pod \"payload-pre-processing-e2e-api-a-dd840a-5c9b4679f5-wspqf\" (UID: \"40f370d3-946f-4cf1-805d-f245ca48e9a9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd840a-5c9b4679f5-wspqf" Nov 01 21:01:03.471577 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:03.471496 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j889c\" (UniqueName: \"kubernetes.io/projected/40f370d3-946f-4cf1-805d-f245ca48e9a9-kube-api-access-j889c\") pod \"payload-pre-processing-e2e-api-a-dd840a-5c9b4679f5-wspqf\" (UID: \"40f370d3-946f-4cf1-805d-f245ca48e9a9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd840a-5c9b4679f5-wspqf" Nov 01 21:01:03.471738 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:03.471601 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40f370d3-946f-4cf1-805d-f245ca48e9a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-dd840a-5c9b4679f5-wspqf\" (UID: \"40f370d3-946f-4cf1-805d-f245ca48e9a9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd840a-5c9b4679f5-wspqf" Nov 01 21:01:03.472376 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:03.472355 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40f370d3-946f-4cf1-805d-f245ca48e9a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-dd840a-5c9b4679f5-wspqf\" (UID: \"40f370d3-946f-4cf1-805d-f245ca48e9a9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd840a-5c9b4679f5-wspqf" Nov 01 21:01:03.529323 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:03.529282 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j889c\" (UniqueName: \"kubernetes.io/projected/40f370d3-946f-4cf1-805d-f245ca48e9a9-kube-api-access-j889c\") pod \"payload-pre-processing-e2e-api-a-dd840a-5c9b4679f5-wspqf\" (UID: \"40f370d3-946f-4cf1-805d-f245ca48e9a9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd840a-5c9b4679f5-wspqf" Nov 01 21:01:03.574144 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:03.574105 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd840a-5c9b4679f5-wspqf" Nov 01 21:01:03.665074 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:03.664430 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dd840a-68df564877-hxfz7"] Nov 01 21:01:03.955165 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:01:03.955129 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40f370d3_946f_4cf1_805d_f245ca48e9a9.slice/crio-8bc1c560c6e26b2a99166fccadc555914a677eded24b1997898ca1c926b3a91c WatchSource:0}: Error finding container 8bc1c560c6e26b2a99166fccadc555914a677eded24b1997898ca1c926b3a91c: Status 404 returned error can't find the container with id 8bc1c560c6e26b2a99166fccadc555914a677eded24b1997898ca1c926b3a91c Nov 01 21:01:03.955844 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:03.955809 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dd840a-5c9b4679f5-wspqf"] Nov 01 21:01:04.075264 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:04.075233 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:04.232795 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:04.232686 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd840a-5c9b4679f5-wspqf" event={"ID":"40f370d3-946f-4cf1-805d-f245ca48e9a9","Type":"ContainerStarted","Data":"22270c441f39a163bcb8e02c6745570f15daa1e5a3e99ef9c051425059aca257"} Nov 01 21:01:04.232795 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:04.232755 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd840a-5c9b4679f5-wspqf" event={"ID":"40f370d3-946f-4cf1-805d-f245ca48e9a9","Type":"ContainerStarted","Data":"8bc1c560c6e26b2a99166fccadc555914a677eded24b1997898ca1c926b3a91c"} Nov 01 21:01:04.232795 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:04.232786 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd840a-5c9b4679f5-wspqf" Nov 01 21:01:04.234591 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:04.234564 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-dd840a-68df564877-hxfz7" event={"ID":"ff449990-bd5a-4b0c-bd0a-7880dbcdd90f","Type":"ContainerStarted","Data":"38b724e5f20c3f6cacc642104f715c9ba8022cb92ae23789209c89a4668e2729"} Nov 01 21:01:04.234723 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:04.234616 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-dd840a-68df564877-hxfz7" event={"ID":"ff449990-bd5a-4b0c-bd0a-7880dbcdd90f","Type":"ContainerStarted","Data":"f8ab68ef3f92fbc3c87a7910f4a7103bde3e7b2ad5de1c44c1eb128346409b9d"} Nov 01 21:01:04.234723 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:04.234641 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-dd840a-68df564877-hxfz7" Nov 01 21:01:04.351274 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:04.351217 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd840a-5c9b4679f5-wspqf" podStartSLOduration=1.351201599 podStartE2EDuration="1.351201599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:01:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:01:04.346576243 +0000 UTC m=+1815.124216390" watchObservedRunningTime="2026-11-01 21:01:04.351201599 +0000 UTC m=+1815.128841734" Nov 01 21:01:05.044533 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:05.044475 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-dd840a-68df564877-hxfz7" podStartSLOduration=3.044458601 podStartE2EDuration="3.044458601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:01:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:01:04.482805674 +0000 UTC m=+1815.260445810" watchObservedRunningTime="2026-11-01 21:01:05.044458601 +0000 UTC m=+1815.822098751" Nov 01 21:01:05.047511 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:05.047475 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cced842-gp2px"] Nov 01 21:01:05.097918 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:05.097879 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cced842-gp2px"] Nov 01 21:01:05.374649 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:05.374565 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9d19cf1-edee-4256-b612-40b9e800c64a" path="/var/lib/kubelet/pods/a9d19cf1-edee-4256-b612-40b9e800c64a/volumes" Nov 01 21:01:08.129792 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:08.129762 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d08fd0-84d7f997dc-6zrkr" Nov 01 21:01:09.390259 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:09.390118 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:14.153464 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:14.153410 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:15.245230 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:15.245201 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd840a-5c9b4679f5-wspqf" Nov 01 21:01:15.245737 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:15.245268 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-dd840a-68df564877-hxfz7" Nov 01 21:01:15.769672 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:01:15.769632 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:01:19.060341 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:19.060301 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:21.887801 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:21.887765 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:24.150249 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:24.150213 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:28.874466 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:28.874423 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:29.033558 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:29.033522 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:29.194777 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:29.194691 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:30.465364 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:30.465322 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:32.102412 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:32.102334 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m"] Nov 01 21:01:32.108736 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:32.108715 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m" Nov 01 21:01:32.169362 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:32.169326 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m"] Nov 01 21:01:32.209394 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:32.209342 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c818ecc-7536-4e57-93f6-40ba5942d318-workload-socket\") pod \"e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m\" (UID: \"3c818ecc-7536-4e57-93f6-40ba5942d318\") " pod="openshift-ingress/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m" Nov 01 21:01:32.209394 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:32.209396 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c818ecc-7536-4e57-93f6-40ba5942d318-istio-token\") pod \"e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m\" (UID: \"3c818ecc-7536-4e57-93f6-40ba5942d318\") " pod="openshift-ingress/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m" Nov 01 21:01:32.209632 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:32.209536 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c818ecc-7536-4e57-93f6-40ba5942d318-credential-socket\") pod \"e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m\" (UID: \"3c818ecc-7536-4e57-93f6-40ba5942d318\") " pod="openshift-ingress/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m" Nov 01 21:01:32.209759 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:32.209742 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c818ecc-7536-4e57-93f6-40ba5942d318-istiod-ca-cert\") pod \"e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m\" (UID: \"3c818ecc-7536-4e57-93f6-40ba5942d318\") " pod="openshift-ingress/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m" Nov 01 21:01:32.209826 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:32.209811 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c818ecc-7536-4e57-93f6-40ba5942d318-istio-podinfo\") pod \"e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m\" (UID: \"3c818ecc-7536-4e57-93f6-40ba5942d318\") " pod="openshift-ingress/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m" Nov 01 21:01:32.209930 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:32.209905 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c818ecc-7536-4e57-93f6-40ba5942d318-istio-data\") pod \"e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m\" (UID: \"3c818ecc-7536-4e57-93f6-40ba5942d318\") " pod="openshift-ingress/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m" Nov 01 21:01:32.210033 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:32.209961 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c818ecc-7536-4e57-93f6-40ba5942d318-workload-certs\") pod \"e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m\" (UID: \"3c818ecc-7536-4e57-93f6-40ba5942d318\") " pod="openshift-ingress/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m" Nov 01 21:01:32.210154 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:32.210127 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9c4k\" (UniqueName: \"kubernetes.io/projected/3c818ecc-7536-4e57-93f6-40ba5942d318-kube-api-access-r9c4k\") pod \"e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m\" (UID: \"3c818ecc-7536-4e57-93f6-40ba5942d318\") " pod="openshift-ingress/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m" Nov 01 21:01:32.210250 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:32.210233 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c818ecc-7536-4e57-93f6-40ba5942d318-istio-envoy\") pod \"e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m\" (UID: \"3c818ecc-7536-4e57-93f6-40ba5942d318\") " pod="openshift-ingress/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m" Nov 01 21:01:32.211193 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:32.211173 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-081c8e-openshift-default-dockercfg-6wtdg\"" Nov 01 21:01:32.214460 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:32.214435 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:32.317513 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:32.316465 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c818ecc-7536-4e57-93f6-40ba5942d318-istiod-ca-cert\") pod \"e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m\" (UID: \"3c818ecc-7536-4e57-93f6-40ba5942d318\") " pod="openshift-ingress/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m" Nov 01 21:01:32.317859 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:32.317419 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c818ecc-7536-4e57-93f6-40ba5942d318-istiod-ca-cert\") pod \"e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m\" (UID: \"3c818ecc-7536-4e57-93f6-40ba5942d318\") " pod="openshift-ingress/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m" Nov 01 21:01:32.318869 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:32.318154 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c818ecc-7536-4e57-93f6-40ba5942d318-istio-podinfo\") pod \"e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m\" (UID: \"3c818ecc-7536-4e57-93f6-40ba5942d318\") " pod="openshift-ingress/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m" Nov 01 21:01:32.319533 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:32.319472 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c818ecc-7536-4e57-93f6-40ba5942d318-istio-data\") pod \"e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m\" (UID: \"3c818ecc-7536-4e57-93f6-40ba5942d318\") " pod="openshift-ingress/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m" Nov 01 21:01:32.325798 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:32.319068 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c818ecc-7536-4e57-93f6-40ba5942d318-istio-data\") pod \"e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m\" (UID: \"3c818ecc-7536-4e57-93f6-40ba5942d318\") " pod="openshift-ingress/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m" Nov 01 21:01:32.326103 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:32.326069 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c818ecc-7536-4e57-93f6-40ba5942d318-workload-certs\") pod \"e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m\" (UID: \"3c818ecc-7536-4e57-93f6-40ba5942d318\") " pod="openshift-ingress/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m" Nov 01 21:01:32.326352 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:32.326333 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9c4k\" (UniqueName: \"kubernetes.io/projected/3c818ecc-7536-4e57-93f6-40ba5942d318-kube-api-access-r9c4k\") pod \"e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m\" (UID: \"3c818ecc-7536-4e57-93f6-40ba5942d318\") " pod="openshift-ingress/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m" Nov 01 21:01:32.330023 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:32.326549 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c818ecc-7536-4e57-93f6-40ba5942d318-istio-envoy\") pod \"e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m\" (UID: \"3c818ecc-7536-4e57-93f6-40ba5942d318\") " pod="openshift-ingress/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m" Nov 01 21:01:32.330281 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:32.330255 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c818ecc-7536-4e57-93f6-40ba5942d318-workload-socket\") pod \"e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m\" (UID: \"3c818ecc-7536-4e57-93f6-40ba5942d318\") " pod="openshift-ingress/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m" Nov 01 21:01:32.330433 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:32.330417 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c818ecc-7536-4e57-93f6-40ba5942d318-istio-token\") pod \"e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m\" (UID: \"3c818ecc-7536-4e57-93f6-40ba5942d318\") " pod="openshift-ingress/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m" Nov 01 21:01:32.330628 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:32.330598 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c818ecc-7536-4e57-93f6-40ba5942d318-credential-socket\") pod \"e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m\" (UID: \"3c818ecc-7536-4e57-93f6-40ba5942d318\") " pod="openshift-ingress/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m" Nov 01 21:01:32.334852 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:32.331415 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c818ecc-7536-4e57-93f6-40ba5942d318-credential-socket\") pod \"e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m\" (UID: \"3c818ecc-7536-4e57-93f6-40ba5942d318\") " pod="openshift-ingress/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m" Nov 01 21:01:32.335383 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:32.335361 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c818ecc-7536-4e57-93f6-40ba5942d318-workload-certs\") pod \"e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m\" (UID: \"3c818ecc-7536-4e57-93f6-40ba5942d318\") " pod="openshift-ingress/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m" Nov 01 21:01:32.336497 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:32.336242 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c818ecc-7536-4e57-93f6-40ba5942d318-workload-socket\") pod \"e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m\" (UID: \"3c818ecc-7536-4e57-93f6-40ba5942d318\") " pod="openshift-ingress/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m" Nov 01 21:01:32.339615 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:32.339592 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c818ecc-7536-4e57-93f6-40ba5942d318-istio-envoy\") pod \"e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m\" (UID: \"3c818ecc-7536-4e57-93f6-40ba5942d318\") " pod="openshift-ingress/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m" Nov 01 21:01:32.357759 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:32.357632 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c818ecc-7536-4e57-93f6-40ba5942d318-istio-podinfo\") pod \"e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m\" (UID: \"3c818ecc-7536-4e57-93f6-40ba5942d318\") " pod="openshift-ingress/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m" Nov 01 21:01:32.391020 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:32.390576 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:32.414170 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:32.414103 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c818ecc-7536-4e57-93f6-40ba5942d318-istio-token\") pod \"e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m\" (UID: \"3c818ecc-7536-4e57-93f6-40ba5942d318\") " pod="openshift-ingress/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m" Nov 01 21:01:32.421023 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:32.419744 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9c4k\" (UniqueName: \"kubernetes.io/projected/3c818ecc-7536-4e57-93f6-40ba5942d318-kube-api-access-r9c4k\") pod \"e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m\" (UID: \"3c818ecc-7536-4e57-93f6-40ba5942d318\") " pod="openshift-ingress/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m" Nov 01 21:01:32.446162 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:32.446096 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m" Nov 01 21:01:32.891164 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:01:32.891130 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c818ecc_7536_4e57_93f6_40ba5942d318.slice/crio-95615dbdcc1e42084f8f337d0836d355df22245924453a562c2b173af5c6e884 WatchSource:0}: Error finding container 95615dbdcc1e42084f8f337d0836d355df22245924453a562c2b173af5c6e884: Status 404 returned error can't find the container with id 95615dbdcc1e42084f8f337d0836d355df22245924453a562c2b173af5c6e884 Nov 01 21:01:32.891764 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:32.891701 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m"] Nov 01 21:01:32.893989 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:32.893958 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:01:32.894114 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:32.894055 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:01:32.894114 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:32.894097 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:01:33.520660 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:33.520622 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m" event={"ID":"3c818ecc-7536-4e57-93f6-40ba5942d318","Type":"ContainerStarted","Data":"e1ec29d393c8e11d13cb254f36e949be173583f41b56f7ffab9cbe9783651dee"} Nov 01 21:01:33.521070 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:33.520679 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m" event={"ID":"3c818ecc-7536-4e57-93f6-40ba5942d318","Type":"ContainerStarted","Data":"95615dbdcc1e42084f8f337d0836d355df22245924453a562c2b173af5c6e884"} Nov 01 21:01:33.642636 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:33.642557 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m" podStartSLOduration=1.642543951 podStartE2EDuration="1.642543951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:01:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:01:33.640254707 +0000 UTC m=+1844.417894842" watchObservedRunningTime="2026-11-01 21:01:33.642543951 +0000 UTC m=+1844.420184128" Nov 01 21:01:33.925355 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:33.925277 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:34.448092 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:34.448049 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m" Nov 01 21:01:34.453980 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:34.453948 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m" Nov 01 21:01:34.529289 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:34.529253 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m" Nov 01 21:01:34.530438 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:34.530407 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m" Nov 01 21:01:35.054150 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:35.054119 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:35.808855 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:35.808810 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:36.321195 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:36.321098 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:36.509155 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:36.509119 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:36.680613 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:36.680526 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:36.841297 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:36.841261 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:39.125891 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:39.125853 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:39.788847 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:39.788804 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:40.018020 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:40.002442 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:41.360801 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:41.360764 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:41.555436 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:41.555400 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:43.468423 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:43.468346 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:44.392949 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:44.392906 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:45.768745 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:01:45.768704 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:01:47.023714 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:47.023681 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:47.307071 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:47.302563 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:47.479211 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:47.478023 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-22ed7b-57f689dcd7-8vspb"] Nov 01 21:01:47.493818 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:47.488474 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-22ed7b-57f689dcd7-8vspb" Nov 01 21:01:47.566821 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:47.566738 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-22ed7b-dockercfg-9rtnd\"" Nov 01 21:01:47.658582 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:47.567896 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-22ed7b\"" Nov 01 21:01:47.720309 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:47.720256 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f54111b-de15-4afa-a188-c8c5f7d14a2d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-22ed7b-57f689dcd7-8vspb\" (UID: \"7f54111b-de15-4afa-a188-c8c5f7d14a2d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-22ed7b-57f689dcd7-8vspb" Nov 01 21:01:47.720647 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:47.720622 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj4nw\" (UniqueName: \"kubernetes.io/projected/7f54111b-de15-4afa-a188-c8c5f7d14a2d-kube-api-access-mj4nw\") pod \"payload-pre-processing-e2e-api-a-22ed7b-57f689dcd7-8vspb\" (UID: \"7f54111b-de15-4afa-a188-c8c5f7d14a2d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-22ed7b-57f689dcd7-8vspb" Nov 01 21:01:47.853803 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:47.846378 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-22ed7b-57f689dcd7-8vspb"] Nov 01 21:01:47.858209 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:47.858177 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-22ed7b-5768c96547-5qpms"] Nov 01 21:01:47.876825 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:47.876786 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f54111b-de15-4afa-a188-c8c5f7d14a2d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-22ed7b-57f689dcd7-8vspb\" (UID: \"7f54111b-de15-4afa-a188-c8c5f7d14a2d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-22ed7b-57f689dcd7-8vspb" Nov 01 21:01:47.879592 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:47.879541 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj4nw\" (UniqueName: \"kubernetes.io/projected/7f54111b-de15-4afa-a188-c8c5f7d14a2d-kube-api-access-mj4nw\") pod \"payload-pre-processing-e2e-api-a-22ed7b-57f689dcd7-8vspb\" (UID: \"7f54111b-de15-4afa-a188-c8c5f7d14a2d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-22ed7b-57f689dcd7-8vspb" Nov 01 21:01:47.881910 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:47.881874 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f54111b-de15-4afa-a188-c8c5f7d14a2d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-22ed7b-57f689dcd7-8vspb\" (UID: \"7f54111b-de15-4afa-a188-c8c5f7d14a2d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-22ed7b-57f689dcd7-8vspb" Nov 01 21:01:47.899099 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:47.898298 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-22ed7b-5768c96547-5qpms" Nov 01 21:01:47.920164 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:47.920135 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-22ed7b-5768c96547-5qpms"] Nov 01 21:01:47.938375 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:47.938344 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj4nw\" (UniqueName: \"kubernetes.io/projected/7f54111b-de15-4afa-a188-c8c5f7d14a2d-kube-api-access-mj4nw\") pod \"payload-pre-processing-e2e-api-a-22ed7b-57f689dcd7-8vspb\" (UID: \"7f54111b-de15-4afa-a188-c8c5f7d14a2d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-22ed7b-57f689dcd7-8vspb" Nov 01 21:01:47.981428 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:47.981391 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8twj8\" (UniqueName: \"kubernetes.io/projected/430400df-fb2d-4390-9208-fc312d705254-kube-api-access-8twj8\") pod \"payload-processing-e2e-api-a-22ed7b-5768c96547-5qpms\" (UID: \"430400df-fb2d-4390-9208-fc312d705254\") " pod="openshift-ingress/payload-processing-e2e-api-a-22ed7b-5768c96547-5qpms" Nov 01 21:01:47.981652 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:47.981625 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/430400df-fb2d-4390-9208-fc312d705254-plugins-config-volume\") pod \"payload-processing-e2e-api-a-22ed7b-5768c96547-5qpms\" (UID: \"430400df-fb2d-4390-9208-fc312d705254\") " pod="openshift-ingress/payload-processing-e2e-api-a-22ed7b-5768c96547-5qpms" Nov 01 21:01:48.083669 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:48.083574 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8twj8\" (UniqueName: \"kubernetes.io/projected/430400df-fb2d-4390-9208-fc312d705254-kube-api-access-8twj8\") pod \"payload-processing-e2e-api-a-22ed7b-5768c96547-5qpms\" (UID: \"430400df-fb2d-4390-9208-fc312d705254\") " pod="openshift-ingress/payload-processing-e2e-api-a-22ed7b-5768c96547-5qpms" Nov 01 21:01:48.084172 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:48.083778 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/430400df-fb2d-4390-9208-fc312d705254-plugins-config-volume\") pod \"payload-processing-e2e-api-a-22ed7b-5768c96547-5qpms\" (UID: \"430400df-fb2d-4390-9208-fc312d705254\") " pod="openshift-ingress/payload-processing-e2e-api-a-22ed7b-5768c96547-5qpms" Nov 01 21:01:48.084513 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:48.084478 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/430400df-fb2d-4390-9208-fc312d705254-plugins-config-volume\") pod \"payload-processing-e2e-api-a-22ed7b-5768c96547-5qpms\" (UID: \"430400df-fb2d-4390-9208-fc312d705254\") " pod="openshift-ingress/payload-processing-e2e-api-a-22ed7b-5768c96547-5qpms" Nov 01 21:01:48.141661 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:48.141628 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8twj8\" (UniqueName: \"kubernetes.io/projected/430400df-fb2d-4390-9208-fc312d705254-kube-api-access-8twj8\") pod \"payload-processing-e2e-api-a-22ed7b-5768c96547-5qpms\" (UID: \"430400df-fb2d-4390-9208-fc312d705254\") " pod="openshift-ingress/payload-processing-e2e-api-a-22ed7b-5768c96547-5qpms" Nov 01 21:01:48.150837 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:48.150808 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-22ed7b-57f689dcd7-8vspb" Nov 01 21:01:48.226832 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:48.226795 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-22ed7b-5768c96547-5qpms" Nov 01 21:01:48.538363 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:48.538321 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-22ed7b-57f689dcd7-8vspb"] Nov 01 21:01:48.538539 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:01:48.538513 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f54111b_de15_4afa_a188_c8c5f7d14a2d.slice/crio-207f48415871ff6308a4d9040fe270c26ace95771cbff011fcfd789b11526607 WatchSource:0}: Error finding container 207f48415871ff6308a4d9040fe270c26ace95771cbff011fcfd789b11526607: Status 404 returned error can't find the container with id 207f48415871ff6308a4d9040fe270c26ace95771cbff011fcfd789b11526607 Nov 01 21:01:48.637332 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:01:48.637304 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod430400df_fb2d_4390_9208_fc312d705254.slice/crio-9e69767f0b6b251c7ae1ee09ff78fc8a9f89c0f32f23034298f587df08cfe5a5 WatchSource:0}: Error finding container 9e69767f0b6b251c7ae1ee09ff78fc8a9f89c0f32f23034298f587df08cfe5a5: Status 404 returned error can't find the container with id 9e69767f0b6b251c7ae1ee09ff78fc8a9f89c0f32f23034298f587df08cfe5a5 Nov 01 21:01:48.638200 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:48.638177 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-22ed7b-5768c96547-5qpms"] Nov 01 21:01:48.866108 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:48.865989 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-22ed7b-57f689dcd7-8vspb" event={"ID":"7f54111b-de15-4afa-a188-c8c5f7d14a2d","Type":"ContainerStarted","Data":"a3a76e29696289c13b08abce97d542366503051b07d36d1d06e19e19bc4b225d"} Nov 01 21:01:48.866108 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:48.866069 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-22ed7b-57f689dcd7-8vspb" event={"ID":"7f54111b-de15-4afa-a188-c8c5f7d14a2d","Type":"ContainerStarted","Data":"207f48415871ff6308a4d9040fe270c26ace95771cbff011fcfd789b11526607"} Nov 01 21:01:48.866108 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:48.866100 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-22ed7b-57f689dcd7-8vspb" Nov 01 21:01:48.867818 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:48.867783 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-22ed7b-5768c96547-5qpms" event={"ID":"430400df-fb2d-4390-9208-fc312d705254","Type":"ContainerStarted","Data":"696d1ee186f7008f36a3641a228eebb9ebbe5076949519e17d2bd78ca87dcc77"} Nov 01 21:01:48.868020 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:48.867837 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-22ed7b-5768c96547-5qpms" event={"ID":"430400df-fb2d-4390-9208-fc312d705254","Type":"ContainerStarted","Data":"9e69767f0b6b251c7ae1ee09ff78fc8a9f89c0f32f23034298f587df08cfe5a5"} Nov 01 21:01:48.868020 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:48.867867 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-22ed7b-5768c96547-5qpms" Nov 01 21:01:48.987046 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:48.986963 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-22ed7b-57f689dcd7-8vspb" podStartSLOduration=1.986949873 podStartE2EDuration="1.986949873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:01:48.982258121 +0000 UTC m=+1859.759898257" watchObservedRunningTime="2026-11-01 21:01:48.986949873 +0000 UTC m=+1859.764590007" Nov 01 21:01:49.099123 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:49.099073 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-22ed7b-5768c96547-5qpms" podStartSLOduration=2.099055501 podStartE2EDuration="2.099055501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:01:49.096944668 +0000 UTC m=+1859.874584802" watchObservedRunningTime="2026-11-01 21:01:49.099055501 +0000 UTC m=+1859.876695635" Nov 01 21:01:49.099634 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:49.099579 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:49.266349 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:49.266317 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:49.445847 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:49.445546 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:53.980744 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:53.980709 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:54.185023 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:54.175943 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:58.932514 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:58.932462 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:59.879673 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:59.879640 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-22ed7b-57f689dcd7-8vspb" Nov 01 21:01:59.879885 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:01:59.879714 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-22ed7b-5768c96547-5qpms" Nov 01 21:02:00.154983 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:00.154879 4484 scope.go:122] "RemoveContainer" containerID="13df01b9dcd89db86a44fe58a69b1abe0b74244a9339270a8a6443b2582db9e0" Nov 01 21:02:03.070471 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:03.070373 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:04.858684 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:04.858647 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:05.213947 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:05.213859 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:05.482854 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:05.482775 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:07.485161 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:07.485129 4484 patch_prober.go:29] interesting pod/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 21:02:07.494818 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:07.485185 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h" podUID="c2fa50d2-8fa0-4997-9fdf-354318d94561" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 21:02:08.153130 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:08.153097 4484 generic.go:356] "Generic (PLEG): container finished" podID="c2fa50d2-8fa0-4997-9fdf-354318d94561" containerID="208dd4fa036e63c78be6383d20c5d27a5acb11fe9023605beafda5d7a153a014" exitCode=0 Nov 01 21:02:08.153317 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:08.153166 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h" event={"ID":"c2fa50d2-8fa0-4997-9fdf-354318d94561","Type":"ContainerDied","Data":"208dd4fa036e63c78be6383d20c5d27a5acb11fe9023605beafda5d7a153a014"} Nov 01 21:02:08.153616 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:08.153602 4484 scope.go:122] "RemoveContainer" containerID="208dd4fa036e63c78be6383d20c5d27a5acb11fe9023605beafda5d7a153a014" Nov 01 21:02:08.155382 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:08.155354 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:02:08.155454 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:08.155423 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:02:08.155491 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:08.155456 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:02:09.163720 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:09.163685 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h" event={"ID":"c2fa50d2-8fa0-4997-9fdf-354318d94561","Type":"ContainerStarted","Data":"7b41bb1933550373133d88f2be5aef3a2731f762e951e4c83f680ced0208eb22"} Nov 01 21:02:09.456198 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:09.456093 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:09.927209 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:09.927171 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:09.969346 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:09.969315 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h" Nov 01 21:02:09.969746 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:09.969650 4484 patch_prober.go:29] interesting pod/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Nov 01 21:02:09.969931 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:09.969910 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h" podUID="c2fa50d2-8fa0-4997-9fdf-354318d94561" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Nov 01 21:02:10.966801 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:10.966762 4484 patch_prober.go:29] interesting pod/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Nov 01 21:02:10.967257 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:10.966831 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h" podUID="c2fa50d2-8fa0-4997-9fdf-354318d94561" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Nov 01 21:02:11.966316 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:11.966281 4484 patch_prober.go:29] interesting pod/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Nov 01 21:02:11.966503 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:11.966345 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h" podUID="c2fa50d2-8fa0-4997-9fdf-354318d94561" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Nov 01 21:02:12.966285 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:12.966250 4484 patch_prober.go:29] interesting pod/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Nov 01 21:02:12.966803 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:12.966313 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h" podUID="c2fa50d2-8fa0-4997-9fdf-354318d94561" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Nov 01 21:02:12.967213 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:12.967184 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:13.139502 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:13.139459 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:13.966362 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:13.966322 4484 patch_prober.go:29] interesting pod/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Nov 01 21:02:13.966806 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:13.966397 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h" podUID="c2fa50d2-8fa0-4997-9fdf-354318d94561" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Nov 01 21:02:14.103931 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:14.103882 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:14.286008 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:14.285967 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:14.971210 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:14.971176 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h" Nov 01 21:02:14.971703 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:14.971688 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h" Nov 01 21:02:14.972401 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:14.972381 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h" Nov 01 21:02:15.093326 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:15.093291 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:15.375864 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:15.375827 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:15.563006 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:15.562662 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:15.880631 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:02:15.880593 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:02:20.241111 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:20.241074 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:20.407831 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:20.407793 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:20.566252 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:20.566161 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:25.424403 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:25.424369 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:30.220033 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:30.215041 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:30.383097 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:30.382347 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:35.177820 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:35.177776 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:36.813202 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:36.813167 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:36.983332 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:36.983298 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:39.904022 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:39.903955 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:43.829434 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:43.829400 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:44.751470 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:44.751435 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:45.768968 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:02:45.768930 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:02:49.547944 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:49.547911 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:50.046519 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:50.046472 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f620899-69lws"] Nov 01 21:02:50.100607 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:50.100555 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f620899-69lws"] Nov 01 21:02:50.107983 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:50.107949 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gq6xw"] Nov 01 21:02:50.108260 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:50.108231 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gq6xw" podUID="17ef1a63-e718-4f21-958a-b353687cbff9" containerName="discovery" containerID="cri-o://378cfaa22a0f1b84a9575991ac3b2a4e3af1e0f52b1b55c0fc388feb9d76c848" gracePeriod=30 Nov 01 21:02:50.343974 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:50.343877 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:50.411779 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:50.411752 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gq6xw" Nov 01 21:02:50.542317 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:50.542283 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:50.542648 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:50.542627 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17ef1a63-e718-4f21-958a-b353687cbff9-istio-token\" (UniqueName: \"kubernetes.io/projected/17ef1a63-e718-4f21-958a-b353687cbff9-istio-token\") pod \"17ef1a63-e718-4f21-958a-b353687cbff9\" (UID: \"17ef1a63-e718-4f21-958a-b353687cbff9\") " Nov 01 21:02:50.542708 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:50.542682 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17ef1a63-e718-4f21-958a-b353687cbff9-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/17ef1a63-e718-4f21-958a-b353687cbff9-local-certs\") pod \"17ef1a63-e718-4f21-958a-b353687cbff9\" (UID: \"17ef1a63-e718-4f21-958a-b353687cbff9\") " Nov 01 21:02:50.542816 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:50.542802 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17ef1a63-e718-4f21-958a-b353687cbff9-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/17ef1a63-e718-4f21-958a-b353687cbff9-istio-kubeconfig\") pod \"17ef1a63-e718-4f21-958a-b353687cbff9\" (UID: \"17ef1a63-e718-4f21-958a-b353687cbff9\") " Nov 01 21:02:50.543032 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:50.542974 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17ef1a63-e718-4f21-958a-b353687cbff9-cacerts\" (UniqueName: \"kubernetes.io/secret/17ef1a63-e718-4f21-958a-b353687cbff9-cacerts\") pod \"17ef1a63-e718-4f21-958a-b353687cbff9\" (UID: \"17ef1a63-e718-4f21-958a-b353687cbff9\") " Nov 01 21:02:50.543218 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:50.543149 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17ef1a63-e718-4f21-958a-b353687cbff9-kube-api-access-86p56\" (UniqueName: \"kubernetes.io/projected/17ef1a63-e718-4f21-958a-b353687cbff9-kube-api-access-86p56\") pod \"17ef1a63-e718-4f21-958a-b353687cbff9\" (UID: \"17ef1a63-e718-4f21-958a-b353687cbff9\") " Nov 01 21:02:50.543320 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:50.543288 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17ef1a63-e718-4f21-958a-b353687cbff9-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/17ef1a63-e718-4f21-958a-b353687cbff9-istio-csr-ca-configmap\") pod \"17ef1a63-e718-4f21-958a-b353687cbff9\" (UID: \"17ef1a63-e718-4f21-958a-b353687cbff9\") " Nov 01 21:02:50.543419 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:50.543380 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17ef1a63-e718-4f21-958a-b353687cbff9-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/17ef1a63-e718-4f21-958a-b353687cbff9-istio-csr-dns-cert\") pod \"17ef1a63-e718-4f21-958a-b353687cbff9\" (UID: \"17ef1a63-e718-4f21-958a-b353687cbff9\") " Nov 01 21:02:50.545015 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:50.544951 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17ef1a63-e718-4f21-958a-b353687cbff9-istio-csr-ca-configmap" pod "17ef1a63-e718-4f21-958a-b353687cbff9" (UID: "17ef1a63-e718-4f21-958a-b353687cbff9"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:02:50.549253 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:50.549224 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17ef1a63-e718-4f21-958a-b353687cbff9-local-certs" pod "17ef1a63-e718-4f21-958a-b353687cbff9" (UID: "17ef1a63-e718-4f21-958a-b353687cbff9"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:02:50.552915 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:50.549534 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17ef1a63-e718-4f21-958a-b353687cbff9-istio-csr-dns-cert" pod "17ef1a63-e718-4f21-958a-b353687cbff9" (UID: "17ef1a63-e718-4f21-958a-b353687cbff9"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:02:50.552915 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:50.550901 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17ef1a63-e718-4f21-958a-b353687cbff9-cacerts" pod "17ef1a63-e718-4f21-958a-b353687cbff9" (UID: "17ef1a63-e718-4f21-958a-b353687cbff9"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:02:50.552915 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:50.551179 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17ef1a63-e718-4f21-958a-b353687cbff9-istio-token" pod "17ef1a63-e718-4f21-958a-b353687cbff9" (UID: "17ef1a63-e718-4f21-958a-b353687cbff9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:02:50.552915 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:50.551206 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17ef1a63-e718-4f21-958a-b353687cbff9-istio-kubeconfig" pod "17ef1a63-e718-4f21-958a-b353687cbff9" (UID: "17ef1a63-e718-4f21-958a-b353687cbff9"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:02:50.555494 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:50.555460 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17ef1a63-e718-4f21-958a-b353687cbff9-kube-api-access-86p56" pod "17ef1a63-e718-4f21-958a-b353687cbff9" (UID: "17ef1a63-e718-4f21-958a-b353687cbff9"). InnerVolumeSpecName "kube-api-access-86p56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:02:50.649058 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:50.648930 4484 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/17ef1a63-e718-4f21-958a-b353687cbff9-istio-csr-ca-configmap\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:02:50.649058 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:50.648973 4484 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/17ef1a63-e718-4f21-958a-b353687cbff9-istio-csr-dns-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:02:50.649058 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:50.648990 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17ef1a63-e718-4f21-958a-b353687cbff9-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:02:50.649058 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:50.649045 4484 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/17ef1a63-e718-4f21-958a-b353687cbff9-local-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:02:50.649769 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:50.649065 4484 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/17ef1a63-e718-4f21-958a-b353687cbff9-istio-kubeconfig\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:02:50.649769 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:50.649079 4484 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/17ef1a63-e718-4f21-958a-b353687cbff9-cacerts\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:02:50.649769 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:50.649093 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86p56\" (UniqueName: \"kubernetes.io/projected/17ef1a63-e718-4f21-958a-b353687cbff9-kube-api-access-86p56\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:02:50.719690 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:50.719653 4484 generic.go:356] "Generic (PLEG): container finished" podID="17ef1a63-e718-4f21-958a-b353687cbff9" containerID="378cfaa22a0f1b84a9575991ac3b2a4e3af1e0f52b1b55c0fc388feb9d76c848" exitCode=0 Nov 01 21:02:50.719873 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:50.719718 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gq6xw" Nov 01 21:02:50.719873 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:50.719738 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gq6xw" event={"ID":"17ef1a63-e718-4f21-958a-b353687cbff9","Type":"ContainerDied","Data":"378cfaa22a0f1b84a9575991ac3b2a4e3af1e0f52b1b55c0fc388feb9d76c848"} Nov 01 21:02:50.719873 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:50.719787 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gq6xw" event={"ID":"17ef1a63-e718-4f21-958a-b353687cbff9","Type":"ContainerDied","Data":"dc06938f2195edcd900dc508386d87faf44ceccdd6ebe3eccfb0903e025e4097"} Nov 01 21:02:50.719873 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:50.719813 4484 scope.go:122] "RemoveContainer" containerID="378cfaa22a0f1b84a9575991ac3b2a4e3af1e0f52b1b55c0fc388feb9d76c848" Nov 01 21:02:50.733465 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:50.733445 4484 scope.go:122] "RemoveContainer" containerID="378cfaa22a0f1b84a9575991ac3b2a4e3af1e0f52b1b55c0fc388feb9d76c848" Nov 01 21:02:50.733792 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:02:50.733773 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"378cfaa22a0f1b84a9575991ac3b2a4e3af1e0f52b1b55c0fc388feb9d76c848\": container with ID starting with 378cfaa22a0f1b84a9575991ac3b2a4e3af1e0f52b1b55c0fc388feb9d76c848 not found: ID does not exist" containerID="378cfaa22a0f1b84a9575991ac3b2a4e3af1e0f52b1b55c0fc388feb9d76c848" Nov 01 21:02:50.733954 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:50.733798 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"378cfaa22a0f1b84a9575991ac3b2a4e3af1e0f52b1b55c0fc388feb9d76c848"} err="failed to get container status \"378cfaa22a0f1b84a9575991ac3b2a4e3af1e0f52b1b55c0fc388feb9d76c848\": rpc error: code = NotFound desc = could not find container \"378cfaa22a0f1b84a9575991ac3b2a4e3af1e0f52b1b55c0fc388feb9d76c848\": container with ID starting with 378cfaa22a0f1b84a9575991ac3b2a4e3af1e0f52b1b55c0fc388feb9d76c848 not found: ID does not exist" Nov 01 21:02:50.892254 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:50.892220 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gq6xw"] Nov 01 21:02:50.898801 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:50.898770 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gq6xw"] Nov 01 21:02:51.078167 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:51.078131 4484 patch_prober.go:29] interesting pod/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 21:02:51.079202 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:51.078203 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z" podUID="21a7e141-0aee-418a-99ca-1b5c3993c303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 21:02:51.375906 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:51.375821 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17ef1a63-e718-4f21-958a-b353687cbff9" path="/var/lib/kubelet/pods/17ef1a63-e718-4f21-958a-b353687cbff9/volumes" Nov 01 21:02:51.376346 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:51.376331 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a7deca0-b10e-4b93-b0c8-9721b9db23b7" path="/var/lib/kubelet/pods/1a7deca0-b10e-4b93-b0c8-9721b9db23b7/volumes" Nov 01 21:02:51.729424 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:51.729328 4484 generic.go:356] "Generic (PLEG): container finished" podID="21a7e141-0aee-418a-99ca-1b5c3993c303" containerID="506e6634ce972c722bec301c2d798fe34ee03b584c2dae5dbfb440fa2ef5a6b1" exitCode=0 Nov 01 21:02:51.729424 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:51.729398 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z" event={"ID":"21a7e141-0aee-418a-99ca-1b5c3993c303","Type":"ContainerDied","Data":"506e6634ce972c722bec301c2d798fe34ee03b584c2dae5dbfb440fa2ef5a6b1"} Nov 01 21:02:51.729940 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:51.729921 4484 scope.go:122] "RemoveContainer" containerID="506e6634ce972c722bec301c2d798fe34ee03b584c2dae5dbfb440fa2ef5a6b1" Nov 01 21:02:51.731860 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:51.731828 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:02:51.731954 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:51.731899 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:02:51.731954 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:51.731933 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:02:52.750738 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:52.750706 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z" event={"ID":"21a7e141-0aee-418a-99ca-1b5c3993c303","Type":"ContainerStarted","Data":"766aebdb3136f65bcf6239952a9446716f79d8ccef470d5e2f44baffc7f4e155"} Nov 01 21:02:53.020554 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:53.020458 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z" Nov 01 21:02:53.020913 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:53.020815 4484 patch_prober.go:29] interesting pod/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Nov 01 21:02:53.021142 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:53.021119 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z" podUID="21a7e141-0aee-418a-99ca-1b5c3993c303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Nov 01 21:02:54.025020 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:54.024418 4484 patch_prober.go:29] interesting pod/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Nov 01 21:02:54.025020 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:54.024502 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z" podUID="21a7e141-0aee-418a-99ca-1b5c3993c303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Nov 01 21:02:54.372816 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:54.372723 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:55.025020 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:55.024464 4484 patch_prober.go:29] interesting pod/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Nov 01 21:02:55.025020 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:55.024530 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z" podUID="21a7e141-0aee-418a-99ca-1b5c3993c303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Nov 01 21:02:56.017517 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:56.017476 4484 patch_prober.go:29] interesting pod/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Nov 01 21:02:56.017833 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:56.017798 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z" podUID="21a7e141-0aee-418a-99ca-1b5c3993c303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Nov 01 21:02:57.025022 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:57.019400 4484 patch_prober.go:29] interesting pod/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Nov 01 21:02:57.025022 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:57.019466 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z" podUID="21a7e141-0aee-418a-99ca-1b5c3993c303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Nov 01 21:02:58.022755 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:58.022482 4484 patch_prober.go:29] interesting pod/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Nov 01 21:02:58.022755 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:58.022558 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z" podUID="21a7e141-0aee-418a-99ca-1b5c3993c303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Nov 01 21:02:59.021109 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:59.021077 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z" Nov 01 21:02:59.021619 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:59.021597 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z" Nov 01 21:02:59.022414 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:59.022393 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z" Nov 01 21:02:59.114200 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:02:59.114166 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:03:00.193074 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:00.193038 4484 scope.go:122] "RemoveContainer" containerID="639cf15de3f98e5c2a3688d9c3baf1b0cfeb3ae0f5fc6bb4e427b90279d80954" Nov 01 21:03:00.204121 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:00.204102 4484 scope.go:122] "RemoveContainer" containerID="1dab405e5b2cc18eda77a2e06a5d8fe6fe8c65b70a3e67b3a04d5b123b535de6" Nov 01 21:03:03.868891 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:03.868809 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:03:08.372721 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:08.372684 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:03:08.554085 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:08.554046 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:03:08.848076 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:08.848046 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:03:13.842370 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:13.842331 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:03:14.078831 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:14.078597 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:03:14.244948 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:14.244912 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:03:14.262819 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:14.262783 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-de8145-68zmw"] Nov 01 21:03:14.263736 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:14.263714 4484 cpu_manager.go:414] "containerMap: removing container" podUID="17ef1a63-e718-4f21-958a-b353687cbff9" containerName="discovery" Nov 01 21:03:14.263865 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:14.263738 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17ef1a63-e718-4f21-958a-b353687cbff9" containerName="discovery" Nov 01 21:03:14.263969 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:14.263954 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="17ef1a63-e718-4f21-958a-b353687cbff9" containerName="discovery" Nov 01 21:03:14.267969 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:14.267949 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-de8145-68zmw" Nov 01 21:03:14.323863 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:14.323833 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-de8145-68zmw"] Nov 01 21:03:14.354575 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:14.354542 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:03:14.369468 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:14.369438 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:03:14.369468 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:14.369438 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:03:14.369734 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:14.369495 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:03:14.395775 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:14.395740 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4ab60f6-33ce-447f-884d-a1384735cc84-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-de8145-68zmw\" (UID: \"a4ab60f6-33ce-447f-884d-a1384735cc84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-de8145-68zmw" Nov 01 21:03:14.395951 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:14.395871 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrqls\" (UniqueName: \"kubernetes.io/projected/a4ab60f6-33ce-447f-884d-a1384735cc84-kube-api-access-lrqls\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-de8145-68zmw\" (UID: \"a4ab60f6-33ce-447f-884d-a1384735cc84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-de8145-68zmw" Nov 01 21:03:14.396171 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:14.396151 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4ab60f6-33ce-447f-884d-a1384735cc84-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-de8145-68zmw\" (UID: \"a4ab60f6-33ce-447f-884d-a1384735cc84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-de8145-68zmw" Nov 01 21:03:14.497989 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:14.497904 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4ab60f6-33ce-447f-884d-a1384735cc84-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-de8145-68zmw\" (UID: \"a4ab60f6-33ce-447f-884d-a1384735cc84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-de8145-68zmw" Nov 01 21:03:14.497989 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:14.497944 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrqls\" (UniqueName: \"kubernetes.io/projected/a4ab60f6-33ce-447f-884d-a1384735cc84-kube-api-access-lrqls\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-de8145-68zmw\" (UID: \"a4ab60f6-33ce-447f-884d-a1384735cc84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-de8145-68zmw" Nov 01 21:03:14.498215 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:14.498169 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4ab60f6-33ce-447f-884d-a1384735cc84-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-de8145-68zmw\" (UID: \"a4ab60f6-33ce-447f-884d-a1384735cc84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-de8145-68zmw" Nov 01 21:03:14.498758 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:14.498732 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4ab60f6-33ce-447f-884d-a1384735cc84-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-de8145-68zmw\" (UID: \"a4ab60f6-33ce-447f-884d-a1384735cc84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-de8145-68zmw" Nov 01 21:03:14.500426 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:14.500397 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4ab60f6-33ce-447f-884d-a1384735cc84-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-de8145-68zmw\" (UID: \"a4ab60f6-33ce-447f-884d-a1384735cc84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-de8145-68zmw" Nov 01 21:03:14.556347 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:14.556313 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrqls\" (UniqueName: \"kubernetes.io/projected/a4ab60f6-33ce-447f-884d-a1384735cc84-kube-api-access-lrqls\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-de8145-68zmw\" (UID: \"a4ab60f6-33ce-447f-884d-a1384735cc84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-de8145-68zmw" Nov 01 21:03:14.583396 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:14.583358 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-de8145-68zmw" Nov 01 21:03:14.967638 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:03:14.967604 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4ab60f6_33ce_447f_884d_a1384735cc84.slice/crio-3afa23227b0ab7c8dc9ca41402655f14df95423de00e450bad61a019e4eb43e2 WatchSource:0}: Error finding container 3afa23227b0ab7c8dc9ca41402655f14df95423de00e450bad61a019e4eb43e2: Status 404 returned error can't find the container with id 3afa23227b0ab7c8dc9ca41402655f14df95423de00e450bad61a019e4eb43e2 Nov 01 21:03:14.968370 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:14.968347 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-de8145-68zmw"] Nov 01 21:03:14.986492 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:14.986466 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-de8145-68zmw" event={"ID":"a4ab60f6-33ce-447f-884d-a1384735cc84","Type":"ContainerStarted","Data":"3afa23227b0ab7c8dc9ca41402655f14df95423de00e450bad61a019e4eb43e2"} Nov 01 21:03:15.769128 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:03:15.769082 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:03:15.995729 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:15.995687 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-de8145-68zmw" event={"ID":"a4ab60f6-33ce-447f-884d-a1384735cc84","Type":"ContainerStarted","Data":"b7e770acf075c92550fa312043aa00c3c5ce476e52b5ea1f3a9d178c4f0869f6"} Nov 01 21:03:16.113444 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:16.113392 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-de8145-68zmw" podStartSLOduration=2.113374796 podStartE2EDuration="2.113374796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:03:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:03:16.109838761 +0000 UTC m=+1946.887478896" watchObservedRunningTime="2026-11-01 21:03:16.113374796 +0000 UTC m=+1946.891014930" Nov 01 21:03:17.046788 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:17.045874 4484 generic.go:356] "Generic (PLEG): container finished" podID="a4ab60f6-33ce-447f-884d-a1384735cc84" containerID="b7e770acf075c92550fa312043aa00c3c5ce476e52b5ea1f3a9d178c4f0869f6" exitCode=0 Nov 01 21:03:17.046788 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:17.046068 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-de8145-68zmw" event={"ID":"a4ab60f6-33ce-447f-884d-a1384735cc84","Type":"ContainerDied","Data":"b7e770acf075c92550fa312043aa00c3c5ce476e52b5ea1f3a9d178c4f0869f6"} Nov 01 21:03:18.249232 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:18.249206 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-de8145-68zmw" Nov 01 21:03:18.264421 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:18.264396 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4ab60f6-33ce-447f-884d-a1384735cc84-kube-api-access-lrqls\" (UniqueName: \"kubernetes.io/projected/a4ab60f6-33ce-447f-884d-a1384735cc84-kube-api-access-lrqls\") pod \"a4ab60f6-33ce-447f-884d-a1384735cc84\" (UID: \"a4ab60f6-33ce-447f-884d-a1384735cc84\") " Nov 01 21:03:18.264574 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:18.264476 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4ab60f6-33ce-447f-884d-a1384735cc84-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4ab60f6-33ce-447f-884d-a1384735cc84-maas-api-openshift-ca\") pod \"a4ab60f6-33ce-447f-884d-a1384735cc84\" (UID: \"a4ab60f6-33ce-447f-884d-a1384735cc84\") " Nov 01 21:03:18.264685 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:18.264670 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4ab60f6-33ce-447f-884d-a1384735cc84-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4ab60f6-33ce-447f-884d-a1384735cc84-maas-api-xks-ca\") pod \"a4ab60f6-33ce-447f-884d-a1384735cc84\" (UID: \"a4ab60f6-33ce-447f-884d-a1384735cc84\") " Nov 01 21:03:18.264819 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:18.264796 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4ab60f6-33ce-447f-884d-a1384735cc84-maas-api-openshift-ca" pod "a4ab60f6-33ce-447f-884d-a1384735cc84" (UID: "a4ab60f6-33ce-447f-884d-a1384735cc84"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:03:18.265292 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:18.265273 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4ab60f6-33ce-447f-884d-a1384735cc84-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:18.266540 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:18.266520 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4ab60f6-33ce-447f-884d-a1384735cc84-kube-api-access-lrqls" pod "a4ab60f6-33ce-447f-884d-a1384735cc84" (UID: "a4ab60f6-33ce-447f-884d-a1384735cc84"). InnerVolumeSpecName "kube-api-access-lrqls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:03:18.266676 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:18.266662 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4ab60f6-33ce-447f-884d-a1384735cc84-maas-api-xks-ca" pod "a4ab60f6-33ce-447f-884d-a1384735cc84" (UID: "a4ab60f6-33ce-447f-884d-a1384735cc84"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:03:18.367556 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:18.367481 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4ab60f6-33ce-447f-884d-a1384735cc84-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:18.367556 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:18.367516 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrqls\" (UniqueName: \"kubernetes.io/projected/a4ab60f6-33ce-447f-884d-a1384735cc84-kube-api-access-lrqls\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:19.069321 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:19.069287 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-de8145-68zmw" Nov 01 21:03:19.069598 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:19.069287 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-de8145-68zmw" event={"ID":"a4ab60f6-33ce-447f-884d-a1384735cc84","Type":"ContainerDied","Data":"3afa23227b0ab7c8dc9ca41402655f14df95423de00e450bad61a019e4eb43e2"} Nov 01 21:03:19.069598 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:19.069462 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3afa23227b0ab7c8dc9ca41402655f14df95423de00e450bad61a019e4eb43e2" Nov 01 21:03:19.118366 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:19.118331 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:03:19.283386 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:19.283354 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:03:19.583647 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:19.583614 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:03:24.420142 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:24.420104 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:03:26.023967 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:26.023932 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-de8145-9dcc65c7f-q6cff"] Nov 01 21:03:26.143675 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:26.143639 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-de8145-56fb74df5-8nbgw"] Nov 01 21:03:26.418221 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:26.418138 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:03:26.633901 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:26.633863 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:03:27.414044 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:27.414008 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a9b439-d7ph6"] Nov 01 21:03:27.414872 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:27.414857 4484 cpu_manager.go:414] "containerMap: removing container" podUID="a4ab60f6-33ce-447f-884d-a1384735cc84" containerName="revoke-keys" Nov 01 21:03:27.414872 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:27.414873 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4ab60f6-33ce-447f-884d-a1384735cc84" containerName="revoke-keys" Nov 01 21:03:27.415028 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:27.415018 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4ab60f6-33ce-447f-884d-a1384735cc84" containerName="revoke-keys" Nov 01 21:03:27.419222 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:27.419204 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a9b439-d7ph6" Nov 01 21:03:27.515019 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:27.513090 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87296964-4ebc-4bd6-9473-527c86358226-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-a9b439-d7ph6\" (UID: \"87296964-4ebc-4bd6-9473-527c86358226\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a9b439-d7ph6" Nov 01 21:03:27.515019 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:27.513146 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87296964-4ebc-4bd6-9473-527c86358226-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-a9b439-d7ph6\" (UID: \"87296964-4ebc-4bd6-9473-527c86358226\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a9b439-d7ph6" Nov 01 21:03:27.515019 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:27.513292 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpf6b\" (UniqueName: \"kubernetes.io/projected/87296964-4ebc-4bd6-9473-527c86358226-kube-api-access-qpf6b\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-a9b439-d7ph6\" (UID: \"87296964-4ebc-4bd6-9473-527c86358226\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a9b439-d7ph6" Nov 01 21:03:27.594953 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:27.594825 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:03:27.595709 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:27.595688 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:03:27.595983 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:27.595968 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:03:27.611058 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:27.598299 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:03:27.684230 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:27.624734 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87296964-4ebc-4bd6-9473-527c86358226-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-a9b439-d7ph6\" (UID: \"87296964-4ebc-4bd6-9473-527c86358226\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a9b439-d7ph6" Nov 01 21:03:27.684230 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:27.624790 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87296964-4ebc-4bd6-9473-527c86358226-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-a9b439-d7ph6\" (UID: \"87296964-4ebc-4bd6-9473-527c86358226\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a9b439-d7ph6" Nov 01 21:03:27.684230 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:27.624962 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpf6b\" (UniqueName: \"kubernetes.io/projected/87296964-4ebc-4bd6-9473-527c86358226-kube-api-access-qpf6b\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-a9b439-d7ph6\" (UID: \"87296964-4ebc-4bd6-9473-527c86358226\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a9b439-d7ph6" Nov 01 21:03:27.684230 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:27.627875 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87296964-4ebc-4bd6-9473-527c86358226-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-a9b439-d7ph6\" (UID: \"87296964-4ebc-4bd6-9473-527c86358226\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a9b439-d7ph6" Nov 01 21:03:27.684230 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:27.674445 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a9b439-d7ph6"] Nov 01 21:03:27.694221 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:27.694107 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87296964-4ebc-4bd6-9473-527c86358226-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-a9b439-d7ph6\" (UID: \"87296964-4ebc-4bd6-9473-527c86358226\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a9b439-d7ph6" Nov 01 21:03:27.759492 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:27.759415 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpf6b\" (UniqueName: \"kubernetes.io/projected/87296964-4ebc-4bd6-9473-527c86358226-kube-api-access-qpf6b\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-a9b439-d7ph6\" (UID: \"87296964-4ebc-4bd6-9473-527c86358226\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a9b439-d7ph6" Nov 01 21:03:28.034123 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:28.034085 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a9b439-d7ph6" Nov 01 21:03:28.415457 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:28.415428 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a9b439-d7ph6"] Nov 01 21:03:28.416944 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:03:28.416913 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87296964_4ebc_4bd6_9473_527c86358226.slice/crio-4504a164a66bed4da250895c4ea2307f959ac12113ce08220669f957710c809e WatchSource:0}: Error finding container 4504a164a66bed4da250895c4ea2307f959ac12113ce08220669f957710c809e: Status 404 returned error can't find the container with id 4504a164a66bed4da250895c4ea2307f959ac12113ce08220669f957710c809e Nov 01 21:03:28.952769 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:28.952734 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:03:29.103966 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:29.103929 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:03:29.256196 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:29.256159 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a9b439-d7ph6" event={"ID":"87296964-4ebc-4bd6-9473-527c86358226","Type":"ContainerStarted","Data":"44cb9caf6cb1e23f4ad56089cfb37e048be8926bf43b1c9d72e95b0a932995fc"} Nov 01 21:03:29.256406 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:29.256215 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a9b439-d7ph6" event={"ID":"87296964-4ebc-4bd6-9473-527c86358226","Type":"ContainerStarted","Data":"4504a164a66bed4da250895c4ea2307f959ac12113ce08220669f957710c809e"} Nov 01 21:03:29.371425 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:29.371368 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a9b439-d7ph6" podStartSLOduration=2.371351468 podStartE2EDuration="2.371351468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:03:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:03:29.371173426 +0000 UTC m=+1960.148813562" watchObservedRunningTime="2026-11-01 21:03:29.371351468 +0000 UTC m=+1960.148991604" Nov 01 21:03:29.780507 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:29.780464 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:03:30.264920 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:30.264885 4484 generic.go:356] "Generic (PLEG): container finished" podID="87296964-4ebc-4bd6-9473-527c86358226" containerID="44cb9caf6cb1e23f4ad56089cfb37e048be8926bf43b1c9d72e95b0a932995fc" exitCode=0 Nov 01 21:03:30.265107 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:30.264943 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a9b439-d7ph6" event={"ID":"87296964-4ebc-4bd6-9473-527c86358226","Type":"ContainerDied","Data":"44cb9caf6cb1e23f4ad56089cfb37e048be8926bf43b1c9d72e95b0a932995fc"} Nov 01 21:03:31.025335 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:31.025284 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-de8145-9dcc65c7f-q6cff" podUID="d895fa0a-1661-4924-80fc-b05b07017767" containerName="payload-processing" containerID="cri-o://d3279eee1c0b0e85e2cdb48591a025e2b6d058d91bbc1aa7dc4a453a02e674f0" gracePeriod=25 Nov 01 21:03:31.144342 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:31.144294 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-de8145-56fb74df5-8nbgw" podUID="6a7f6c58-4b9f-4c61-abe2-75e1f9fba161" containerName="payload-pre-processing" containerID="cri-o://3c6fbb2cc68d4c8c7cda0d36276d88e0ac63579ea1c8c64ef70af6c17a5b7c05" gracePeriod=25 Nov 01 21:03:31.286948 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:31.286872 4484 generic.go:356] "Generic (PLEG): container finished" podID="d895fa0a-1661-4924-80fc-b05b07017767" containerID="d3279eee1c0b0e85e2cdb48591a025e2b6d058d91bbc1aa7dc4a453a02e674f0" exitCode=0 Nov 01 21:03:31.287115 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:31.286944 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-de8145-9dcc65c7f-q6cff" event={"ID":"d895fa0a-1661-4924-80fc-b05b07017767","Type":"ContainerDied","Data":"d3279eee1c0b0e85e2cdb48591a025e2b6d058d91bbc1aa7dc4a453a02e674f0"} Nov 01 21:03:31.343022 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:31.342973 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-de8145-9dcc65c7f-q6cff" Nov 01 21:03:31.423155 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:31.423115 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d895fa0a-1661-4924-80fc-b05b07017767-kube-api-access-5j4m2\" (UniqueName: \"kubernetes.io/projected/d895fa0a-1661-4924-80fc-b05b07017767-kube-api-access-5j4m2\") pod \"d895fa0a-1661-4924-80fc-b05b07017767\" (UID: \"d895fa0a-1661-4924-80fc-b05b07017767\") " Nov 01 21:03:31.423335 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:31.423276 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d895fa0a-1661-4924-80fc-b05b07017767-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d895fa0a-1661-4924-80fc-b05b07017767-plugins-config-volume\") pod \"d895fa0a-1661-4924-80fc-b05b07017767\" (UID: \"d895fa0a-1661-4924-80fc-b05b07017767\") " Nov 01 21:03:31.425986 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:31.425811 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d895fa0a-1661-4924-80fc-b05b07017767-plugins-config-volume" pod "d895fa0a-1661-4924-80fc-b05b07017767" (UID: "d895fa0a-1661-4924-80fc-b05b07017767"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:03:31.440261 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:31.440227 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d895fa0a-1661-4924-80fc-b05b07017767-kube-api-access-5j4m2" pod "d895fa0a-1661-4924-80fc-b05b07017767" (UID: "d895fa0a-1661-4924-80fc-b05b07017767"). InnerVolumeSpecName "kube-api-access-5j4m2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:03:31.469905 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:31.469873 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-de8145-56fb74df5-8nbgw" Nov 01 21:03:31.497624 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:31.497590 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a9b439-d7ph6" Nov 01 21:03:31.517267 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:31.517232 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms"] Nov 01 21:03:31.517500 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:31.517476 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms" podUID="6bc9b444-1915-4ba3-a7f7-789549742e85" containerName="istio-proxy" containerID="cri-o://e8cdf47147083f8aab89976bba8b2812b6ada4417df3e325740374d4965cbe8d" gracePeriod=30 Nov 01 21:03:31.525804 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:31.525769 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a7f6c58-4b9f-4c61-abe2-75e1f9fba161-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a7f6c58-4b9f-4c61-abe2-75e1f9fba161-plugins-config-volume\") pod \"6a7f6c58-4b9f-4c61-abe2-75e1f9fba161\" (UID: \"6a7f6c58-4b9f-4c61-abe2-75e1f9fba161\") " Nov 01 21:03:31.525968 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:31.525868 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a7f6c58-4b9f-4c61-abe2-75e1f9fba161-kube-api-access-fr6kx\" (UniqueName: \"kubernetes.io/projected/6a7f6c58-4b9f-4c61-abe2-75e1f9fba161-kube-api-access-fr6kx\") pod \"6a7f6c58-4b9f-4c61-abe2-75e1f9fba161\" (UID: \"6a7f6c58-4b9f-4c61-abe2-75e1f9fba161\") " Nov 01 21:03:31.526244 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:31.526217 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a7f6c58-4b9f-4c61-abe2-75e1f9fba161-plugins-config-volume" pod "6a7f6c58-4b9f-4c61-abe2-75e1f9fba161" (UID: "6a7f6c58-4b9f-4c61-abe2-75e1f9fba161"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:03:31.526767 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:31.526738 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a7f6c58-4b9f-4c61-abe2-75e1f9fba161-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:31.526884 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:31.526771 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5j4m2\" (UniqueName: \"kubernetes.io/projected/d895fa0a-1661-4924-80fc-b05b07017767-kube-api-access-5j4m2\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:31.526884 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:31.526787 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d895fa0a-1661-4924-80fc-b05b07017767-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:31.528919 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:31.528887 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a7f6c58-4b9f-4c61-abe2-75e1f9fba161-kube-api-access-fr6kx" pod "6a7f6c58-4b9f-4c61-abe2-75e1f9fba161" (UID: "6a7f6c58-4b9f-4c61-abe2-75e1f9fba161"). InnerVolumeSpecName "kube-api-access-fr6kx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:03:31.627928 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:31.627825 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87296964-4ebc-4bd6-9473-527c86358226-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87296964-4ebc-4bd6-9473-527c86358226-maas-api-xks-ca\") pod \"87296964-4ebc-4bd6-9473-527c86358226\" (UID: \"87296964-4ebc-4bd6-9473-527c86358226\") " Nov 01 21:03:31.627928 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:31.627874 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87296964-4ebc-4bd6-9473-527c86358226-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87296964-4ebc-4bd6-9473-527c86358226-maas-api-openshift-ca\") pod \"87296964-4ebc-4bd6-9473-527c86358226\" (UID: \"87296964-4ebc-4bd6-9473-527c86358226\") " Nov 01 21:03:31.628218 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:31.628095 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87296964-4ebc-4bd6-9473-527c86358226-kube-api-access-qpf6b\" (UniqueName: \"kubernetes.io/projected/87296964-4ebc-4bd6-9473-527c86358226-kube-api-access-qpf6b\") pod \"87296964-4ebc-4bd6-9473-527c86358226\" (UID: \"87296964-4ebc-4bd6-9473-527c86358226\") " Nov 01 21:03:31.628379 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:31.628342 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87296964-4ebc-4bd6-9473-527c86358226-maas-api-openshift-ca" pod "87296964-4ebc-4bd6-9473-527c86358226" (UID: "87296964-4ebc-4bd6-9473-527c86358226"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:03:31.628919 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:31.628890 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87296964-4ebc-4bd6-9473-527c86358226-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:31.629073 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:31.628922 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr6kx\" (UniqueName: \"kubernetes.io/projected/6a7f6c58-4b9f-4c61-abe2-75e1f9fba161-kube-api-access-fr6kx\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:31.630520 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:31.630490 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87296964-4ebc-4bd6-9473-527c86358226-kube-api-access-qpf6b" pod "87296964-4ebc-4bd6-9473-527c86358226" (UID: "87296964-4ebc-4bd6-9473-527c86358226"). InnerVolumeSpecName "kube-api-access-qpf6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:03:31.633304 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:31.633278 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87296964-4ebc-4bd6-9473-527c86358226-maas-api-xks-ca" pod "87296964-4ebc-4bd6-9473-527c86358226" (UID: "87296964-4ebc-4bd6-9473-527c86358226"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:03:31.730816 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:31.730779 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpf6b\" (UniqueName: \"kubernetes.io/projected/87296964-4ebc-4bd6-9473-527c86358226-kube-api-access-qpf6b\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:31.730816 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:31.730813 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87296964-4ebc-4bd6-9473-527c86358226-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:31.881258 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:31.881154 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn"] Nov 01 21:03:31.881449 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:31.881393 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn" podUID="cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40" containerName="istio-proxy" containerID="cri-o://b7ae38d1714f3cf72f0431d85bfe89e750a9f756180a4e9965625a1df1452ac4" gracePeriod=30 Nov 01 21:03:32.296793 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:32.296765 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a9b439-d7ph6" Nov 01 21:03:32.297251 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:32.296803 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a9b439-d7ph6" event={"ID":"87296964-4ebc-4bd6-9473-527c86358226","Type":"ContainerDied","Data":"4504a164a66bed4da250895c4ea2307f959ac12113ce08220669f957710c809e"} Nov 01 21:03:32.297251 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:32.296859 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4504a164a66bed4da250895c4ea2307f959ac12113ce08220669f957710c809e" Nov 01 21:03:32.298454 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:32.298432 4484 generic.go:356] "Generic (PLEG): container finished" podID="6a7f6c58-4b9f-4c61-abe2-75e1f9fba161" containerID="3c6fbb2cc68d4c8c7cda0d36276d88e0ac63579ea1c8c64ef70af6c17a5b7c05" exitCode=0 Nov 01 21:03:32.298587 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:32.298478 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-de8145-56fb74df5-8nbgw" Nov 01 21:03:32.298587 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:32.298516 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-de8145-56fb74df5-8nbgw" event={"ID":"6a7f6c58-4b9f-4c61-abe2-75e1f9fba161","Type":"ContainerDied","Data":"3c6fbb2cc68d4c8c7cda0d36276d88e0ac63579ea1c8c64ef70af6c17a5b7c05"} Nov 01 21:03:32.298587 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:32.298567 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-de8145-56fb74df5-8nbgw" event={"ID":"6a7f6c58-4b9f-4c61-abe2-75e1f9fba161","Type":"ContainerDied","Data":"b9a4df89ef9cf207537964aae57b118d2782e06d31292f4bfeedb2e15b2052eb"} Nov 01 21:03:32.298766 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:32.298595 4484 scope.go:122] "RemoveContainer" containerID="3c6fbb2cc68d4c8c7cda0d36276d88e0ac63579ea1c8c64ef70af6c17a5b7c05" Nov 01 21:03:32.300123 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:32.300107 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-de8145-9dcc65c7f-q6cff" Nov 01 21:03:32.300240 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:32.300147 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-de8145-9dcc65c7f-q6cff" event={"ID":"d895fa0a-1661-4924-80fc-b05b07017767","Type":"ContainerDied","Data":"bb5eac3b4743f7469978d8972eaa33a522150afd1ced85f1ab48c3be95f7f130"} Nov 01 21:03:32.311753 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:32.311733 4484 scope.go:122] "RemoveContainer" containerID="3c6fbb2cc68d4c8c7cda0d36276d88e0ac63579ea1c8c64ef70af6c17a5b7c05" Nov 01 21:03:32.312084 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:03:32.312065 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c6fbb2cc68d4c8c7cda0d36276d88e0ac63579ea1c8c64ef70af6c17a5b7c05\": container with ID starting with 3c6fbb2cc68d4c8c7cda0d36276d88e0ac63579ea1c8c64ef70af6c17a5b7c05 not found: ID does not exist" containerID="3c6fbb2cc68d4c8c7cda0d36276d88e0ac63579ea1c8c64ef70af6c17a5b7c05" Nov 01 21:03:32.312162 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:32.312096 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c6fbb2cc68d4c8c7cda0d36276d88e0ac63579ea1c8c64ef70af6c17a5b7c05"} err="failed to get container status \"3c6fbb2cc68d4c8c7cda0d36276d88e0ac63579ea1c8c64ef70af6c17a5b7c05\": rpc error: code = NotFound desc = could not find container \"3c6fbb2cc68d4c8c7cda0d36276d88e0ac63579ea1c8c64ef70af6c17a5b7c05\": container with ID starting with 3c6fbb2cc68d4c8c7cda0d36276d88e0ac63579ea1c8c64ef70af6c17a5b7c05 not found: ID does not exist" Nov 01 21:03:32.312162 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:32.312115 4484 scope.go:122] "RemoveContainer" containerID="d3279eee1c0b0e85e2cdb48591a025e2b6d058d91bbc1aa7dc4a453a02e674f0" Nov 01 21:03:32.467732 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:32.467692 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-de8145-9dcc65c7f-q6cff"] Nov 01 21:03:32.472093 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:32.472064 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-de8145-9dcc65c7f-q6cff"] Nov 01 21:03:32.637765 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:32.637670 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-de8145-56fb74df5-8nbgw"] Nov 01 21:03:32.642260 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:32.642234 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-de8145-56fb74df5-8nbgw"] Nov 01 21:03:33.375316 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:33.375284 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a7f6c58-4b9f-4c61-abe2-75e1f9fba161" path="/var/lib/kubelet/pods/6a7f6c58-4b9f-4c61-abe2-75e1f9fba161/volumes" Nov 01 21:03:33.375774 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:33.375760 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d895fa0a-1661-4924-80fc-b05b07017767" path="/var/lib/kubelet/pods/d895fa0a-1661-4924-80fc-b05b07017767/volumes" Nov 01 21:03:34.182678 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:34.182638 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:03:34.350220 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:34.350174 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:03:34.634080 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:34.634039 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:03:34.792430 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:34.792395 4484 patch_prober.go:29] interesting pod/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 21:03:34.792612 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:34.792459 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn" podUID="cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 21:03:36.874831 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:36.874801 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms" Nov 01 21:03:36.931580 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:36.931553 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6bc9b444-1915-4ba3-a7f7-789549742e85-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bc9b444-1915-4ba3-a7f7-789549742e85-istio-podinfo\") pod \"6bc9b444-1915-4ba3-a7f7-789549742e85\" (UID: \"6bc9b444-1915-4ba3-a7f7-789549742e85\") " Nov 01 21:03:36.931708 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:36.931696 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bc9b444-1915-4ba3-a7f7-789549742e85-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bc9b444-1915-4ba3-a7f7-789549742e85-istiod-ca-cert\") pod \"6bc9b444-1915-4ba3-a7f7-789549742e85\" (UID: \"6bc9b444-1915-4ba3-a7f7-789549742e85\") " Nov 01 21:03:36.931837 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:36.931824 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bc9b444-1915-4ba3-a7f7-789549742e85-kube-api-access-4z4rj\" (UniqueName: \"kubernetes.io/projected/6bc9b444-1915-4ba3-a7f7-789549742e85-kube-api-access-4z4rj\") pod \"6bc9b444-1915-4ba3-a7f7-789549742e85\" (UID: \"6bc9b444-1915-4ba3-a7f7-789549742e85\") " Nov 01 21:03:36.931880 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:36.931856 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bc9b444-1915-4ba3-a7f7-789549742e85-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc9b444-1915-4ba3-a7f7-789549742e85-workload-socket\") pod \"6bc9b444-1915-4ba3-a7f7-789549742e85\" (UID: \"6bc9b444-1915-4ba3-a7f7-789549742e85\") " Nov 01 21:03:36.931915 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:36.931889 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bc9b444-1915-4ba3-a7f7-789549742e85-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bc9b444-1915-4ba3-a7f7-789549742e85-istio-data\") pod \"6bc9b444-1915-4ba3-a7f7-789549742e85\" (UID: \"6bc9b444-1915-4ba3-a7f7-789549742e85\") " Nov 01 21:03:36.931980 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:36.931918 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bc9b444-1915-4ba3-a7f7-789549742e85-istio-token\" (UniqueName: \"kubernetes.io/projected/6bc9b444-1915-4ba3-a7f7-789549742e85-istio-token\") pod \"6bc9b444-1915-4ba3-a7f7-789549742e85\" (UID: \"6bc9b444-1915-4ba3-a7f7-789549742e85\") " Nov 01 21:03:36.931980 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:36.931969 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bc9b444-1915-4ba3-a7f7-789549742e85-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bc9b444-1915-4ba3-a7f7-789549742e85-istio-envoy\") pod \"6bc9b444-1915-4ba3-a7f7-789549742e85\" (UID: \"6bc9b444-1915-4ba3-a7f7-789549742e85\") " Nov 01 21:03:36.932122 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:36.932101 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bc9b444-1915-4ba3-a7f7-789549742e85-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bc9b444-1915-4ba3-a7f7-789549742e85-workload-certs\") pod \"6bc9b444-1915-4ba3-a7f7-789549742e85\" (UID: \"6bc9b444-1915-4ba3-a7f7-789549742e85\") " Nov 01 21:03:36.932243 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:36.932143 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bc9b444-1915-4ba3-a7f7-789549742e85-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc9b444-1915-4ba3-a7f7-789549742e85-credential-socket\") pod \"6bc9b444-1915-4ba3-a7f7-789549742e85\" (UID: \"6bc9b444-1915-4ba3-a7f7-789549742e85\") " Nov 01 21:03:36.932243 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:36.932086 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bc9b444-1915-4ba3-a7f7-789549742e85-istiod-ca-cert" pod "6bc9b444-1915-4ba3-a7f7-789549742e85" (UID: "6bc9b444-1915-4ba3-a7f7-789549742e85"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:03:36.932243 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:36.932183 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc9b444-1915-4ba3-a7f7-789549742e85-workload-socket" pod "6bc9b444-1915-4ba3-a7f7-789549742e85" (UID: "6bc9b444-1915-4ba3-a7f7-789549742e85"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:03:36.932243 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:36.932213 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc9b444-1915-4ba3-a7f7-789549742e85-istio-data" pod "6bc9b444-1915-4ba3-a7f7-789549742e85" (UID: "6bc9b444-1915-4ba3-a7f7-789549742e85"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:03:36.932494 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:36.932414 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc9b444-1915-4ba3-a7f7-789549742e85-credential-socket" pod "6bc9b444-1915-4ba3-a7f7-789549742e85" (UID: "6bc9b444-1915-4ba3-a7f7-789549742e85"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:03:36.932494 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:36.932434 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc9b444-1915-4ba3-a7f7-789549742e85-workload-certs" pod "6bc9b444-1915-4ba3-a7f7-789549742e85" (UID: "6bc9b444-1915-4ba3-a7f7-789549742e85"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:03:36.932886 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:36.932868 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc9b444-1915-4ba3-a7f7-789549742e85-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:36.932963 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:36.932894 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bc9b444-1915-4ba3-a7f7-789549742e85-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:36.932963 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:36.932908 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bc9b444-1915-4ba3-a7f7-789549742e85-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:36.932963 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:36.932921 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc9b444-1915-4ba3-a7f7-789549742e85-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:36.932963 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:36.932937 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bc9b444-1915-4ba3-a7f7-789549742e85-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:36.934342 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:36.934311 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bc9b444-1915-4ba3-a7f7-789549742e85-kube-api-access-4z4rj" pod "6bc9b444-1915-4ba3-a7f7-789549742e85" (UID: "6bc9b444-1915-4ba3-a7f7-789549742e85"). InnerVolumeSpecName "kube-api-access-4z4rj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:03:36.934446 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:36.934426 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc9b444-1915-4ba3-a7f7-789549742e85-istio-envoy" pod "6bc9b444-1915-4ba3-a7f7-789549742e85" (UID: "6bc9b444-1915-4ba3-a7f7-789549742e85"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:03:36.934534 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:36.934474 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6bc9b444-1915-4ba3-a7f7-789549742e85-istio-podinfo" pod "6bc9b444-1915-4ba3-a7f7-789549742e85" (UID: "6bc9b444-1915-4ba3-a7f7-789549742e85"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:03:36.934595 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:36.934576 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bc9b444-1915-4ba3-a7f7-789549742e85-istio-token" pod "6bc9b444-1915-4ba3-a7f7-789549742e85" (UID: "6bc9b444-1915-4ba3-a7f7-789549742e85"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:03:37.041589 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:37.034903 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z4rj\" (UniqueName: \"kubernetes.io/projected/6bc9b444-1915-4ba3-a7f7-789549742e85-kube-api-access-4z4rj\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:37.041589 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:37.034941 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bc9b444-1915-4ba3-a7f7-789549742e85-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:37.041589 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:37.034958 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bc9b444-1915-4ba3-a7f7-789549742e85-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:37.041589 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:37.034969 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bc9b444-1915-4ba3-a7f7-789549742e85-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:37.166463 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:37.166433 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn" Nov 01 21:03:37.237872 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:37.237835 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-istio-data\") pod \"cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40\" (UID: \"cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40\") " Nov 01 21:03:37.238117 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:37.237901 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-workload-certs\") pod \"cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40\" (UID: \"cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40\") " Nov 01 21:03:37.238117 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:37.238097 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-credential-socket\") pod \"cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40\" (UID: \"cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40\") " Nov 01 21:03:37.238263 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:37.238228 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-workload-certs" pod "cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40" (UID: "cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:03:37.238263 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:37.238234 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-istio-podinfo\") pod \"cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40\" (UID: \"cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40\") " Nov 01 21:03:37.238376 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:37.238241 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-istio-data" pod "cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40" (UID: "cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:03:37.238435 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:37.238382 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-credential-socket" pod "cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40" (UID: "cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:03:37.238435 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:37.238394 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-istiod-ca-cert\") pod \"cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40\" (UID: \"cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40\") " Nov 01 21:03:37.238552 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:37.238446 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-istio-token\" (UniqueName: \"kubernetes.io/projected/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-istio-token\") pod \"cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40\" (UID: \"cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40\") " Nov 01 21:03:37.238552 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:37.238495 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-workload-socket\") pod \"cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40\" (UID: \"cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40\") " Nov 01 21:03:37.238552 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:37.238535 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-istio-envoy\") pod \"cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40\" (UID: \"cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40\") " Nov 01 21:03:37.238734 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:37.238578 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-kube-api-access-bgstf\" (UniqueName: \"kubernetes.io/projected/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-kube-api-access-bgstf\") pod \"cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40\" (UID: \"cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40\") " Nov 01 21:03:37.238816 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:37.238786 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-workload-socket" pod "cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40" (UID: "cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:03:37.238879 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:37.238807 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-istiod-ca-cert" pod "cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40" (UID: "cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:03:37.239338 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:37.239316 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:37.239338 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:37.239338 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:37.239502 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:37.239348 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:37.239502 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:37.239362 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:37.239502 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:37.239374 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:37.240474 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:37.240451 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-istio-podinfo" pod "cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40" (UID: "cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:03:37.240704 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:37.240684 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-istio-envoy" pod "cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40" (UID: "cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:03:37.241048 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:37.241031 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-istio-token" pod "cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40" (UID: "cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:03:37.241129 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:37.241071 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-kube-api-access-bgstf" pod "cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40" (UID: "cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40"). InnerVolumeSpecName "kube-api-access-bgstf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:03:37.340537 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:37.340501 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:37.340537 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:37.340532 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:37.340761 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:37.340547 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:37.340761 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:37.340560 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgstf\" (UniqueName: \"kubernetes.io/projected/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40-kube-api-access-bgstf\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:37.350175 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:37.350140 4484 generic.go:356] "Generic (PLEG): container finished" podID="cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40" containerID="b7ae38d1714f3cf72f0431d85bfe89e750a9f756180a4e9965625a1df1452ac4" exitCode=0 Nov 01 21:03:37.350362 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:37.350186 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn" Nov 01 21:03:37.350362 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:37.350216 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn" event={"ID":"cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40","Type":"ContainerDied","Data":"b7ae38d1714f3cf72f0431d85bfe89e750a9f756180a4e9965625a1df1452ac4"} Nov 01 21:03:37.350362 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:37.350304 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn" event={"ID":"cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40","Type":"ContainerDied","Data":"b2c3b2a511288a25fd80a93342e8f8cbb60486b1f38c925231b38b1c3fd2a246"} Nov 01 21:03:37.350362 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:37.350336 4484 scope.go:122] "RemoveContainer" containerID="b7ae38d1714f3cf72f0431d85bfe89e750a9f756180a4e9965625a1df1452ac4" Nov 01 21:03:37.352750 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:37.352724 4484 generic.go:356] "Generic (PLEG): container finished" podID="6bc9b444-1915-4ba3-a7f7-789549742e85" containerID="e8cdf47147083f8aab89976bba8b2812b6ada4417df3e325740374d4965cbe8d" exitCode=0 Nov 01 21:03:37.352882 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:37.352790 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms" Nov 01 21:03:37.352882 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:37.352862 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms" event={"ID":"6bc9b444-1915-4ba3-a7f7-789549742e85","Type":"ContainerDied","Data":"e8cdf47147083f8aab89976bba8b2812b6ada4417df3e325740374d4965cbe8d"} Nov 01 21:03:37.353048 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:37.352907 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms" event={"ID":"6bc9b444-1915-4ba3-a7f7-789549742e85","Type":"ContainerDied","Data":"df922b6b099c935f2d2a523ee78ef063829255e07f1104e5dd5044f8cfc72163"} Nov 01 21:03:37.362812 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:37.362785 4484 scope.go:122] "RemoveContainer" containerID="b7ae38d1714f3cf72f0431d85bfe89e750a9f756180a4e9965625a1df1452ac4" Nov 01 21:03:37.363168 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:03:37.363139 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7ae38d1714f3cf72f0431d85bfe89e750a9f756180a4e9965625a1df1452ac4\": container with ID starting with b7ae38d1714f3cf72f0431d85bfe89e750a9f756180a4e9965625a1df1452ac4 not found: ID does not exist" containerID="b7ae38d1714f3cf72f0431d85bfe89e750a9f756180a4e9965625a1df1452ac4" Nov 01 21:03:37.363282 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:37.363173 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7ae38d1714f3cf72f0431d85bfe89e750a9f756180a4e9965625a1df1452ac4"} err="failed to get container status \"b7ae38d1714f3cf72f0431d85bfe89e750a9f756180a4e9965625a1df1452ac4\": rpc error: code = NotFound desc = could not find container \"b7ae38d1714f3cf72f0431d85bfe89e750a9f756180a4e9965625a1df1452ac4\": container with ID starting with b7ae38d1714f3cf72f0431d85bfe89e750a9f756180a4e9965625a1df1452ac4 not found: ID does not exist" Nov 01 21:03:37.363282 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:37.363191 4484 scope.go:122] "RemoveContainer" containerID="e8cdf47147083f8aab89976bba8b2812b6ada4417df3e325740374d4965cbe8d" Nov 01 21:03:37.375083 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:37.375066 4484 scope.go:122] "RemoveContainer" containerID="e8cdf47147083f8aab89976bba8b2812b6ada4417df3e325740374d4965cbe8d" Nov 01 21:03:37.375704 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:03:37.375672 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8cdf47147083f8aab89976bba8b2812b6ada4417df3e325740374d4965cbe8d\": container with ID starting with e8cdf47147083f8aab89976bba8b2812b6ada4417df3e325740374d4965cbe8d not found: ID does not exist" containerID="e8cdf47147083f8aab89976bba8b2812b6ada4417df3e325740374d4965cbe8d" Nov 01 21:03:37.375770 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:37.375705 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8cdf47147083f8aab89976bba8b2812b6ada4417df3e325740374d4965cbe8d"} err="failed to get container status \"e8cdf47147083f8aab89976bba8b2812b6ada4417df3e325740374d4965cbe8d\": rpc error: code = NotFound desc = could not find container \"e8cdf47147083f8aab89976bba8b2812b6ada4417df3e325740374d4965cbe8d\": container with ID starting with e8cdf47147083f8aab89976bba8b2812b6ada4417df3e325740374d4965cbe8d not found: ID does not exist" Nov 01 21:03:37.525106 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:37.525074 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms"] Nov 01 21:03:37.528983 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:37.528960 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-gf6ms"] Nov 01 21:03:37.639527 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:37.639499 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn"] Nov 01 21:03:37.643470 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:37.643440 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-m75bn"] Nov 01 21:03:39.373394 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:39.373365 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bc9b444-1915-4ba3-a7f7-789549742e85" path="/var/lib/kubelet/pods/6bc9b444-1915-4ba3-a7f7-789549742e85/volumes" Nov 01 21:03:39.374022 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:39.373984 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40" path="/var/lib/kubelet/pods/cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40/volumes" Nov 01 21:03:39.677782 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:39.677690 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:03:41.349268 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:41.349229 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-a9b439-76bf5ff49-qqkwc"] Nov 01 21:03:41.468293 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:41.468261 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a9b439-76df769c99-g6df9"] Nov 01 21:03:41.733318 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:41.733232 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:03:41.923019 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:41.922228 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:03:44.191402 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:44.191361 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:03:44.742597 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:44.742561 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:03:45.768460 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:03:45.768422 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:03:46.349736 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:46.349686 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a9b439-76bf5ff49-qqkwc" podUID="8195e66f-4ed3-44f2-ab9c-3ff8881e428c" containerName="payload-processing" containerID="cri-o://27d12065d89e35f17d959635b9447332d78abf1853fcefebd471169c49f116b0" gracePeriod=25 Nov 01 21:03:46.469451 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:46.469402 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a9b439-76df769c99-g6df9" podUID="4e4385fa-bbd4-4fc5-b40d-43f27d0ae3ac" containerName="payload-pre-processing" containerID="cri-o://9d9577e8f2b4f42bca622d0740f5e8f5ec4f3738e67e48ad61e301037ba4e270" gracePeriod=25 Nov 01 21:03:46.672950 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:46.672923 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a9b439-76bf5ff49-qqkwc" Nov 01 21:03:46.788043 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:46.787092 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8195e66f-4ed3-44f2-ab9c-3ff8881e428c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8195e66f-4ed3-44f2-ab9c-3ff8881e428c-plugins-config-volume\") pod \"8195e66f-4ed3-44f2-ab9c-3ff8881e428c\" (UID: \"8195e66f-4ed3-44f2-ab9c-3ff8881e428c\") " Nov 01 21:03:46.788043 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:46.787491 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8195e66f-4ed3-44f2-ab9c-3ff8881e428c-kube-api-access-jqm95\" (UniqueName: \"kubernetes.io/projected/8195e66f-4ed3-44f2-ab9c-3ff8881e428c-kube-api-access-jqm95\") pod \"8195e66f-4ed3-44f2-ab9c-3ff8881e428c\" (UID: \"8195e66f-4ed3-44f2-ab9c-3ff8881e428c\") " Nov 01 21:03:46.788043 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:46.788027 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8195e66f-4ed3-44f2-ab9c-3ff8881e428c-plugins-config-volume" pod "8195e66f-4ed3-44f2-ab9c-3ff8881e428c" (UID: "8195e66f-4ed3-44f2-ab9c-3ff8881e428c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:03:46.788839 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:46.788816 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8195e66f-4ed3-44f2-ab9c-3ff8881e428c-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:46.793530 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:46.793485 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8195e66f-4ed3-44f2-ab9c-3ff8881e428c-kube-api-access-jqm95" pod "8195e66f-4ed3-44f2-ab9c-3ff8881e428c" (UID: "8195e66f-4ed3-44f2-ab9c-3ff8881e428c"). InnerVolumeSpecName "kube-api-access-jqm95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:03:46.802158 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:46.802131 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a9b439-76df769c99-g6df9" Nov 01 21:03:46.889867 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:46.889784 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e4385fa-bbd4-4fc5-b40d-43f27d0ae3ac-kube-api-access-rdmxs\" (UniqueName: \"kubernetes.io/projected/4e4385fa-bbd4-4fc5-b40d-43f27d0ae3ac-kube-api-access-rdmxs\") pod \"4e4385fa-bbd4-4fc5-b40d-43f27d0ae3ac\" (UID: \"4e4385fa-bbd4-4fc5-b40d-43f27d0ae3ac\") " Nov 01 21:03:46.889867 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:46.889852 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e4385fa-bbd4-4fc5-b40d-43f27d0ae3ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4385fa-bbd4-4fc5-b40d-43f27d0ae3ac-plugins-config-volume\") pod \"4e4385fa-bbd4-4fc5-b40d-43f27d0ae3ac\" (UID: \"4e4385fa-bbd4-4fc5-b40d-43f27d0ae3ac\") " Nov 01 21:03:46.890267 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:46.890241 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e4385fa-bbd4-4fc5-b40d-43f27d0ae3ac-plugins-config-volume" pod "4e4385fa-bbd4-4fc5-b40d-43f27d0ae3ac" (UID: "4e4385fa-bbd4-4fc5-b40d-43f27d0ae3ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:03:46.890561 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:46.890536 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqm95\" (UniqueName: \"kubernetes.io/projected/8195e66f-4ed3-44f2-ab9c-3ff8881e428c-kube-api-access-jqm95\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:46.890561 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:46.890562 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4385fa-bbd4-4fc5-b40d-43f27d0ae3ac-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:46.892427 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:46.892396 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e4385fa-bbd4-4fc5-b40d-43f27d0ae3ac-kube-api-access-rdmxs" pod "4e4385fa-bbd4-4fc5-b40d-43f27d0ae3ac" (UID: "4e4385fa-bbd4-4fc5-b40d-43f27d0ae3ac"). InnerVolumeSpecName "kube-api-access-rdmxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:03:46.992411 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:46.992365 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdmxs\" (UniqueName: \"kubernetes.io/projected/4e4385fa-bbd4-4fc5-b40d-43f27d0ae3ac-kube-api-access-rdmxs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:47.462565 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:47.462530 4484 generic.go:356] "Generic (PLEG): container finished" podID="8195e66f-4ed3-44f2-ab9c-3ff8881e428c" containerID="27d12065d89e35f17d959635b9447332d78abf1853fcefebd471169c49f116b0" exitCode=0 Nov 01 21:03:47.462763 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:47.462585 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a9b439-76bf5ff49-qqkwc" Nov 01 21:03:47.462763 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:47.462620 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a9b439-76bf5ff49-qqkwc" event={"ID":"8195e66f-4ed3-44f2-ab9c-3ff8881e428c","Type":"ContainerDied","Data":"27d12065d89e35f17d959635b9447332d78abf1853fcefebd471169c49f116b0"} Nov 01 21:03:47.462763 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:47.462667 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a9b439-76bf5ff49-qqkwc" event={"ID":"8195e66f-4ed3-44f2-ab9c-3ff8881e428c","Type":"ContainerDied","Data":"ee956cc41e4c0396c39c6dd6b248183391d41725f29ed4664522f4f063888f6b"} Nov 01 21:03:47.462763 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:47.462687 4484 scope.go:122] "RemoveContainer" containerID="27d12065d89e35f17d959635b9447332d78abf1853fcefebd471169c49f116b0" Nov 01 21:03:47.464471 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:47.464376 4484 generic.go:356] "Generic (PLEG): container finished" podID="4e4385fa-bbd4-4fc5-b40d-43f27d0ae3ac" containerID="9d9577e8f2b4f42bca622d0740f5e8f5ec4f3738e67e48ad61e301037ba4e270" exitCode=0 Nov 01 21:03:47.464596 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:47.464575 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a9b439-76df769c99-g6df9" event={"ID":"4e4385fa-bbd4-4fc5-b40d-43f27d0ae3ac","Type":"ContainerDied","Data":"9d9577e8f2b4f42bca622d0740f5e8f5ec4f3738e67e48ad61e301037ba4e270"} Nov 01 21:03:47.464719 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:47.464619 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a9b439-76df769c99-g6df9" event={"ID":"4e4385fa-bbd4-4fc5-b40d-43f27d0ae3ac","Type":"ContainerDied","Data":"ccbb04f375fea15624dc9dc24ae59b5c981fa3161df6b932fa8dbfa81ee45ab4"} Nov 01 21:03:47.464719 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:47.464639 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a9b439-76df769c99-g6df9" Nov 01 21:03:47.478057 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:47.478037 4484 scope.go:122] "RemoveContainer" containerID="27d12065d89e35f17d959635b9447332d78abf1853fcefebd471169c49f116b0" Nov 01 21:03:47.478450 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:03:47.478425 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27d12065d89e35f17d959635b9447332d78abf1853fcefebd471169c49f116b0\": container with ID starting with 27d12065d89e35f17d959635b9447332d78abf1853fcefebd471169c49f116b0 not found: ID does not exist" containerID="27d12065d89e35f17d959635b9447332d78abf1853fcefebd471169c49f116b0" Nov 01 21:03:47.478560 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:47.478456 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27d12065d89e35f17d959635b9447332d78abf1853fcefebd471169c49f116b0"} err="failed to get container status \"27d12065d89e35f17d959635b9447332d78abf1853fcefebd471169c49f116b0\": rpc error: code = NotFound desc = could not find container \"27d12065d89e35f17d959635b9447332d78abf1853fcefebd471169c49f116b0\": container with ID starting with 27d12065d89e35f17d959635b9447332d78abf1853fcefebd471169c49f116b0 not found: ID does not exist" Nov 01 21:03:47.478560 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:47.478475 4484 scope.go:122] "RemoveContainer" containerID="9d9577e8f2b4f42bca622d0740f5e8f5ec4f3738e67e48ad61e301037ba4e270" Nov 01 21:03:47.493486 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:47.493462 4484 scope.go:122] "RemoveContainer" containerID="9d9577e8f2b4f42bca622d0740f5e8f5ec4f3738e67e48ad61e301037ba4e270" Nov 01 21:03:47.494162 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:03:47.494131 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d9577e8f2b4f42bca622d0740f5e8f5ec4f3738e67e48ad61e301037ba4e270\": container with ID starting with 9d9577e8f2b4f42bca622d0740f5e8f5ec4f3738e67e48ad61e301037ba4e270 not found: ID does not exist" containerID="9d9577e8f2b4f42bca622d0740f5e8f5ec4f3738e67e48ad61e301037ba4e270" Nov 01 21:03:47.494254 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:47.494171 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d9577e8f2b4f42bca622d0740f5e8f5ec4f3738e67e48ad61e301037ba4e270"} err="failed to get container status \"9d9577e8f2b4f42bca622d0740f5e8f5ec4f3738e67e48ad61e301037ba4e270\": rpc error: code = NotFound desc = could not find container \"9d9577e8f2b4f42bca622d0740f5e8f5ec4f3738e67e48ad61e301037ba4e270\": container with ID starting with 9d9577e8f2b4f42bca622d0740f5e8f5ec4f3738e67e48ad61e301037ba4e270 not found: ID does not exist" Nov 01 21:03:47.630292 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:47.630203 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a9b439-76df769c99-g6df9"] Nov 01 21:03:47.681223 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:47.681189 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a9b439-76df769c99-g6df9"] Nov 01 21:03:47.791400 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:47.791363 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-a9b439-76bf5ff49-qqkwc"] Nov 01 21:03:47.795351 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:47.795327 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-a9b439-76bf5ff49-qqkwc"] Nov 01 21:03:49.377306 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:49.377272 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e4385fa-bbd4-4fc5-b40d-43f27d0ae3ac" path="/var/lib/kubelet/pods/4e4385fa-bbd4-4fc5-b40d-43f27d0ae3ac/volumes" Nov 01 21:03:49.378756 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:49.377880 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8195e66f-4ed3-44f2-ab9c-3ff8881e428c" path="/var/lib/kubelet/pods/8195e66f-4ed3-44f2-ab9c-3ff8881e428c/volumes" Nov 01 21:03:49.379711 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:49.379687 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:03:49.537570 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:49.537539 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:03:49.750789 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:49.750755 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:03:50.508971 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:50.508937 4484 generic.go:356] "Generic (PLEG): container finished" podID="6c3c6fec-0841-4a0c-b761-626d8c53a1cb" containerID="a5d91c1d8490cb6b27129606cf5d3c39f95dedafea0ed794a5cd959283a6e608" exitCode=0 Nov 01 21:03:50.509391 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:50.508985 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq" event={"ID":"6c3c6fec-0841-4a0c-b761-626d8c53a1cb","Type":"ContainerDied","Data":"a5d91c1d8490cb6b27129606cf5d3c39f95dedafea0ed794a5cd959283a6e608"} Nov 01 21:03:50.509467 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:50.509451 4484 scope.go:122] "RemoveContainer" containerID="a5d91c1d8490cb6b27129606cf5d3c39f95dedafea0ed794a5cd959283a6e608" Nov 01 21:03:50.511257 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:50.511223 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:03:50.511379 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:50.511312 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:03:50.511379 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:50.511362 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:03:51.397034 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:51.394983 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq" Nov 01 21:03:51.557933 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:51.557901 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq" event={"ID":"6c3c6fec-0841-4a0c-b761-626d8c53a1cb","Type":"ContainerStarted","Data":"8d9ae7629a83e488649a1e3de0872ded59eb6ec2a1317076573ca545c7d5abcd"} Nov 01 21:03:51.950596 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:51.950559 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq" Nov 01 21:03:51.957164 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:51.957121 4484 patch_prober.go:29] interesting pod/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Nov 01 21:03:51.957550 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:51.957509 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq" podUID="6c3c6fec-0841-4a0c-b761-626d8c53a1cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Nov 01 21:03:52.679225 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:52.679186 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:03:52.853273 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:52.853224 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:03:52.951077 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:52.948547 4484 patch_prober.go:29] interesting pod/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Nov 01 21:03:52.951077 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:52.948636 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq" podUID="6c3c6fec-0841-4a0c-b761-626d8c53a1cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Nov 01 21:03:53.948019 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:53.947456 4484 patch_prober.go:29] interesting pod/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Nov 01 21:03:53.948019 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:53.947539 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq" podUID="6c3c6fec-0841-4a0c-b761-626d8c53a1cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Nov 01 21:03:54.373010 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:54.372964 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:03:54.948020 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:54.947964 4484 patch_prober.go:29] interesting pod/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Nov 01 21:03:54.948553 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:54.948065 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq" podUID="6c3c6fec-0841-4a0c-b761-626d8c53a1cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Nov 01 21:03:55.180459 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:03:55.180421 4484 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@15-35470-0.service\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c3c6fec_0841_4a0c_b761_626d8c53a1cb.slice/crio-a5d91c1d8490cb6b27129606cf5d3c39f95dedafea0ed794a5cd959283a6e608.scope\": RecentStats: unable to find data in memory cache]" Nov 01 21:03:55.854100 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:03:55.853083 4484 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@15-35470-0.service\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c3c6fec_0841_4a0c_b761_626d8c53a1cb.slice/crio-a5d91c1d8490cb6b27129606cf5d3c39f95dedafea0ed794a5cd959283a6e608.scope\": RecentStats: unable to find data in memory cache]" Nov 01 21:03:55.859482 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:55.859447 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b27e66-9g52w"] Nov 01 21:03:55.860334 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:55.860315 4484 cpu_manager.go:414] "containerMap: removing container" podUID="6bc9b444-1915-4ba3-a7f7-789549742e85" containerName="istio-proxy" Nov 01 21:03:55.860334 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:55.860332 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bc9b444-1915-4ba3-a7f7-789549742e85" containerName="istio-proxy" Nov 01 21:03:55.860509 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:55.860365 4484 cpu_manager.go:414] "containerMap: removing container" podUID="8195e66f-4ed3-44f2-ab9c-3ff8881e428c" containerName="payload-processing" Nov 01 21:03:55.860509 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:55.860371 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8195e66f-4ed3-44f2-ab9c-3ff8881e428c" containerName="payload-processing" Nov 01 21:03:55.860509 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:55.860379 4484 cpu_manager.go:414] "containerMap: removing container" podUID="4e4385fa-bbd4-4fc5-b40d-43f27d0ae3ac" containerName="payload-pre-processing" Nov 01 21:03:55.860509 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:55.860386 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e4385fa-bbd4-4fc5-b40d-43f27d0ae3ac" containerName="payload-pre-processing" Nov 01 21:03:55.860509 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:55.860396 4484 cpu_manager.go:414] "containerMap: removing container" podUID="cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40" containerName="istio-proxy" Nov 01 21:03:55.860509 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:55.860403 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40" containerName="istio-proxy" Nov 01 21:03:55.860509 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:55.860412 4484 cpu_manager.go:414] "containerMap: removing container" podUID="d895fa0a-1661-4924-80fc-b05b07017767" containerName="payload-processing" Nov 01 21:03:55.860509 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:55.860417 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d895fa0a-1661-4924-80fc-b05b07017767" containerName="payload-processing" Nov 01 21:03:55.860509 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:55.860431 4484 cpu_manager.go:414] "containerMap: removing container" podUID="87296964-4ebc-4bd6-9473-527c86358226" containerName="revoke-keys" Nov 01 21:03:55.860509 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:55.860436 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87296964-4ebc-4bd6-9473-527c86358226" containerName="revoke-keys" Nov 01 21:03:55.860509 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:55.860450 4484 cpu_manager.go:414] "containerMap: removing container" podUID="6a7f6c58-4b9f-4c61-abe2-75e1f9fba161" containerName="payload-pre-processing" Nov 01 21:03:55.860509 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:55.860455 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a7f6c58-4b9f-4c61-abe2-75e1f9fba161" containerName="payload-pre-processing" Nov 01 21:03:55.861122 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:55.860556 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="8195e66f-4ed3-44f2-ab9c-3ff8881e428c" containerName="payload-processing" Nov 01 21:03:55.861122 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:55.860564 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e4385fa-bbd4-4fc5-b40d-43f27d0ae3ac" containerName="payload-pre-processing" Nov 01 21:03:55.861122 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:55.860571 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="cad3a33a-eeaf-4d9b-9a1d-6d628a38cd40" containerName="istio-proxy" Nov 01 21:03:55.861122 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:55.860579 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="d895fa0a-1661-4924-80fc-b05b07017767" containerName="payload-processing" Nov 01 21:03:55.861122 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:55.860587 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="87296964-4ebc-4bd6-9473-527c86358226" containerName="revoke-keys" Nov 01 21:03:55.861122 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:55.860593 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a7f6c58-4b9f-4c61-abe2-75e1f9fba161" containerName="payload-pre-processing" Nov 01 21:03:55.861122 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:55.860603 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bc9b444-1915-4ba3-a7f7-789549742e85" containerName="istio-proxy" Nov 01 21:03:55.864201 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:55.864178 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b27e66-9g52w" Nov 01 21:03:55.923787 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:55.923633 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:03:55.923787 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:55.923718 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:03:55.923787 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:55.923755 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:03:55.925217 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:55.923892 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:03:55.963016 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:55.953427 4484 patch_prober.go:29] interesting pod/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Nov 01 21:03:55.963016 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:55.953487 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq" podUID="6c3c6fec-0841-4a0c-b761-626d8c53a1cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Nov 01 21:03:55.965220 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:55.965176 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b27e66-9g52w"] Nov 01 21:03:55.969673 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:55.969641 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9080b783-c895-4460-aa20-467c0ace8c3f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b27e66-9g52w\" (UID: \"9080b783-c895-4460-aa20-467c0ace8c3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b27e66-9g52w" Nov 01 21:03:55.969827 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:55.969693 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9080b783-c895-4460-aa20-467c0ace8c3f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b27e66-9g52w\" (UID: \"9080b783-c895-4460-aa20-467c0ace8c3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b27e66-9g52w" Nov 01 21:03:55.969874 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:55.969860 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq2qz\" (UniqueName: \"kubernetes.io/projected/9080b783-c895-4460-aa20-467c0ace8c3f-kube-api-access-zq2qz\") pod \"maas-api-revoke-keys-e2e-api-a-b27e66-9g52w\" (UID: \"9080b783-c895-4460-aa20-467c0ace8c3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b27e66-9g52w" Nov 01 21:03:56.071137 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:56.071100 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9080b783-c895-4460-aa20-467c0ace8c3f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b27e66-9g52w\" (UID: \"9080b783-c895-4460-aa20-467c0ace8c3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b27e66-9g52w" Nov 01 21:03:56.071137 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:56.071140 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9080b783-c895-4460-aa20-467c0ace8c3f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b27e66-9g52w\" (UID: \"9080b783-c895-4460-aa20-467c0ace8c3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b27e66-9g52w" Nov 01 21:03:56.071373 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:56.071189 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq2qz\" (UniqueName: \"kubernetes.io/projected/9080b783-c895-4460-aa20-467c0ace8c3f-kube-api-access-zq2qz\") pod \"maas-api-revoke-keys-e2e-api-a-b27e66-9g52w\" (UID: \"9080b783-c895-4460-aa20-467c0ace8c3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b27e66-9g52w" Nov 01 21:03:56.072154 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:56.072129 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9080b783-c895-4460-aa20-467c0ace8c3f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b27e66-9g52w\" (UID: \"9080b783-c895-4460-aa20-467c0ace8c3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b27e66-9g52w" Nov 01 21:03:56.073794 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:56.073770 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9080b783-c895-4460-aa20-467c0ace8c3f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b27e66-9g52w\" (UID: \"9080b783-c895-4460-aa20-467c0ace8c3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b27e66-9g52w" Nov 01 21:03:56.160972 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:56.160690 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq2qz\" (UniqueName: \"kubernetes.io/projected/9080b783-c895-4460-aa20-467c0ace8c3f-kube-api-access-zq2qz\") pod \"maas-api-revoke-keys-e2e-api-a-b27e66-9g52w\" (UID: \"9080b783-c895-4460-aa20-467c0ace8c3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b27e66-9g52w" Nov 01 21:03:56.178171 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:56.178114 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b27e66-9g52w" Nov 01 21:03:56.659589 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:56.659556 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b27e66-9g52w"] Nov 01 21:03:56.980603 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:56.980563 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq" Nov 01 21:03:56.981133 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:56.980976 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq" Nov 01 21:03:56.984706 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:56.984663 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq" Nov 01 21:03:57.608088 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:57.608046 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b27e66-9g52w" event={"ID":"9080b783-c895-4460-aa20-467c0ace8c3f","Type":"ContainerStarted","Data":"b6d3070e0b2dfb4725aa64e643d90abec766cb2ca0f5992921c9662b89168ea1"} Nov 01 21:03:57.608290 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:57.608129 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b27e66-9g52w" event={"ID":"9080b783-c895-4460-aa20-467c0ace8c3f","Type":"ContainerStarted","Data":"62aeca638f3ccf24d480350be7b184bcba6837865e32c83706a44c0c2cb0349f"} Nov 01 21:03:57.727495 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:57.727433 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b27e66-9g52w" podStartSLOduration=2.727417935 podStartE2EDuration="2.727417935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:03:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:03:57.724098146 +0000 UTC m=+1988.501738302" watchObservedRunningTime="2026-11-01 21:03:57.727417935 +0000 UTC m=+1988.505058070" Nov 01 21:03:58.619681 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:58.619639 4484 generic.go:356] "Generic (PLEG): container finished" podID="9080b783-c895-4460-aa20-467c0ace8c3f" containerID="b6d3070e0b2dfb4725aa64e643d90abec766cb2ca0f5992921c9662b89168ea1" exitCode=0 Nov 01 21:03:58.620262 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:58.619727 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b27e66-9g52w" event={"ID":"9080b783-c895-4460-aa20-467c0ace8c3f","Type":"ContainerDied","Data":"b6d3070e0b2dfb4725aa64e643d90abec766cb2ca0f5992921c9662b89168ea1"} Nov 01 21:03:59.441067 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:59.440985 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:03:59.823377 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:59.823341 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b27e66-9g52w" Nov 01 21:03:59.936925 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:59.936894 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9080b783-c895-4460-aa20-467c0ace8c3f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9080b783-c895-4460-aa20-467c0ace8c3f-maas-api-openshift-ca\") pod \"9080b783-c895-4460-aa20-467c0ace8c3f\" (UID: \"9080b783-c895-4460-aa20-467c0ace8c3f\") " Nov 01 21:03:59.937118 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:59.937038 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9080b783-c895-4460-aa20-467c0ace8c3f-kube-api-access-zq2qz\" (UniqueName: \"kubernetes.io/projected/9080b783-c895-4460-aa20-467c0ace8c3f-kube-api-access-zq2qz\") pod \"9080b783-c895-4460-aa20-467c0ace8c3f\" (UID: \"9080b783-c895-4460-aa20-467c0ace8c3f\") " Nov 01 21:03:59.937118 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:59.937095 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9080b783-c895-4460-aa20-467c0ace8c3f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9080b783-c895-4460-aa20-467c0ace8c3f-maas-api-xks-ca\") pod \"9080b783-c895-4460-aa20-467c0ace8c3f\" (UID: \"9080b783-c895-4460-aa20-467c0ace8c3f\") " Nov 01 21:03:59.937415 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:59.937383 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9080b783-c895-4460-aa20-467c0ace8c3f-maas-api-openshift-ca" pod "9080b783-c895-4460-aa20-467c0ace8c3f" (UID: "9080b783-c895-4460-aa20-467c0ace8c3f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:03:59.938585 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:59.938131 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9080b783-c895-4460-aa20-467c0ace8c3f-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:59.941307 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:59.940541 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9080b783-c895-4460-aa20-467c0ace8c3f-kube-api-access-zq2qz" pod "9080b783-c895-4460-aa20-467c0ace8c3f" (UID: "9080b783-c895-4460-aa20-467c0ace8c3f"). InnerVolumeSpecName "kube-api-access-zq2qz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:03:59.946200 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:03:59.946163 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9080b783-c895-4460-aa20-467c0ace8c3f-maas-api-xks-ca" pod "9080b783-c895-4460-aa20-467c0ace8c3f" (UID: "9080b783-c895-4460-aa20-467c0ace8c3f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:04:00.039171 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:00.039137 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq2qz\" (UniqueName: \"kubernetes.io/projected/9080b783-c895-4460-aa20-467c0ace8c3f-kube-api-access-zq2qz\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:00.039171 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:00.039168 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9080b783-c895-4460-aa20-467c0ace8c3f-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:00.248457 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:00.248427 4484 scope.go:122] "RemoveContainer" containerID="5e892c2131faf44397dc1b3c79c661d6c943078e5f5ba4ad20800b32c12f48f4" Nov 01 21:04:00.259516 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:00.259490 4484 scope.go:122] "RemoveContainer" containerID="acc625b318bfa109bad380efb5b3ff9d5259e7261ce409704112f0380829e56b" Nov 01 21:04:00.294812 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:00.294786 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr"] Nov 01 21:04:00.295005 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:00.294972 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr" podUID="43443369-5e6a-4b4b-ac9d-acbb320656c1" containerName="istio-proxy" containerID="cri-o://2937525e226fa5935d72fc9212c722f405c0da4cdd57776b4a699fd7255b5013" gracePeriod=30 Nov 01 21:04:00.473048 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:00.472985 4484 patch_prober.go:29] interesting pod/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 21:04:00.473232 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:00.473092 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr" podUID="43443369-5e6a-4b4b-ac9d-acbb320656c1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 21:04:00.644344 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:00.644251 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b27e66-9g52w" event={"ID":"9080b783-c895-4460-aa20-467c0ace8c3f","Type":"ContainerDied","Data":"62aeca638f3ccf24d480350be7b184bcba6837865e32c83706a44c0c2cb0349f"} Nov 01 21:04:00.644344 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:00.644310 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62aeca638f3ccf24d480350be7b184bcba6837865e32c83706a44c0c2cb0349f" Nov 01 21:04:00.644344 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:00.644266 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b27e66-9g52w" Nov 01 21:04:01.669941 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:01.669909 4484 generic.go:356] "Generic (PLEG): container finished" podID="3c818ecc-7536-4e57-93f6-40ba5942d318" containerID="e1ec29d393c8e11d13cb254f36e949be173583f41b56f7ffab9cbe9783651dee" exitCode=0 Nov 01 21:04:01.670523 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:01.669986 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m" event={"ID":"3c818ecc-7536-4e57-93f6-40ba5942d318","Type":"ContainerDied","Data":"e1ec29d393c8e11d13cb254f36e949be173583f41b56f7ffab9cbe9783651dee"} Nov 01 21:04:01.670523 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:01.670478 4484 scope.go:122] "RemoveContainer" containerID="e1ec29d393c8e11d13cb254f36e949be173583f41b56f7ffab9cbe9783651dee" Nov 01 21:04:01.672168 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:01.672138 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:04:01.672247 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:01.672207 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:04:01.672247 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:01.672238 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:04:02.681273 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:02.681234 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m" event={"ID":"3c818ecc-7536-4e57-93f6-40ba5942d318","Type":"ContainerStarted","Data":"521531af618eb362a74aed2f6b877b3991146bfe2614cb17482b05f74b816b7d"} Nov 01 21:04:02.986550 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:02.986470 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd840a-4t6zl"] Nov 01 21:04:02.987372 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:02.987356 4484 cpu_manager.go:414] "containerMap: removing container" podUID="9080b783-c895-4460-aa20-467c0ace8c3f" containerName="revoke-keys" Nov 01 21:04:02.987425 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:02.987374 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9080b783-c895-4460-aa20-467c0ace8c3f" containerName="revoke-keys" Nov 01 21:04:02.987539 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:02.987528 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="9080b783-c895-4460-aa20-467c0ace8c3f" containerName="revoke-keys" Nov 01 21:04:02.991388 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:02.991367 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd840a-4t6zl" Nov 01 21:04:03.044800 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:03.044767 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:04:03.044800 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:03.044767 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:04:03.044800 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:03.044799 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:04:03.045116 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:03.044924 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:04:03.045355 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:03.045333 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd840a-4t6zl"] Nov 01 21:04:03.087866 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:03.087825 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d85edeb-2371-4625-97b4-b41cba9a30d5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-dd840a-4t6zl\" (UID: \"4d85edeb-2371-4625-97b4-b41cba9a30d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd840a-4t6zl" Nov 01 21:04:03.088117 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:03.088017 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d85edeb-2371-4625-97b4-b41cba9a30d5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-dd840a-4t6zl\" (UID: \"4d85edeb-2371-4625-97b4-b41cba9a30d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd840a-4t6zl" Nov 01 21:04:03.088117 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:03.088071 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzs8v\" (UniqueName: \"kubernetes.io/projected/4d85edeb-2371-4625-97b4-b41cba9a30d5-kube-api-access-xzs8v\") pod \"maas-api-revoke-keys-e2e-api-a-dd840a-4t6zl\" (UID: \"4d85edeb-2371-4625-97b4-b41cba9a30d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd840a-4t6zl" Nov 01 21:04:03.189466 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:03.189426 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d85edeb-2371-4625-97b4-b41cba9a30d5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-dd840a-4t6zl\" (UID: \"4d85edeb-2371-4625-97b4-b41cba9a30d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd840a-4t6zl" Nov 01 21:04:03.189637 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:03.189517 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d85edeb-2371-4625-97b4-b41cba9a30d5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-dd840a-4t6zl\" (UID: \"4d85edeb-2371-4625-97b4-b41cba9a30d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd840a-4t6zl" Nov 01 21:04:03.189637 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:03.189547 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzs8v\" (UniqueName: \"kubernetes.io/projected/4d85edeb-2371-4625-97b4-b41cba9a30d5-kube-api-access-xzs8v\") pod \"maas-api-revoke-keys-e2e-api-a-dd840a-4t6zl\" (UID: \"4d85edeb-2371-4625-97b4-b41cba9a30d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd840a-4t6zl" Nov 01 21:04:03.190327 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:03.190306 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d85edeb-2371-4625-97b4-b41cba9a30d5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-dd840a-4t6zl\" (UID: \"4d85edeb-2371-4625-97b4-b41cba9a30d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd840a-4t6zl" Nov 01 21:04:03.191979 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:03.191957 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d85edeb-2371-4625-97b4-b41cba9a30d5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-dd840a-4t6zl\" (UID: \"4d85edeb-2371-4625-97b4-b41cba9a30d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd840a-4t6zl" Nov 01 21:04:03.247714 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:03.247674 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzs8v\" (UniqueName: \"kubernetes.io/projected/4d85edeb-2371-4625-97b4-b41cba9a30d5-kube-api-access-xzs8v\") pod \"maas-api-revoke-keys-e2e-api-a-dd840a-4t6zl\" (UID: \"4d85edeb-2371-4625-97b4-b41cba9a30d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd840a-4t6zl" Nov 01 21:04:03.330491 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:03.330452 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd840a-4t6zl" Nov 01 21:04:03.449035 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:03.448882 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m" Nov 01 21:04:03.449576 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:03.449553 4484 patch_prober.go:29] interesting pod/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Nov 01 21:04:03.449696 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:03.449595 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m" podUID="3c818ecc-7536-4e57-93f6-40ba5942d318" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Nov 01 21:04:03.731555 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:03.731482 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd840a-4t6zl"] Nov 01 21:04:03.734682 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:04:03.734641 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d85edeb_2371_4625_97b4_b41cba9a30d5.slice/crio-1b45d67c5f507030de46167240b45ab7624d2fa28ee3ee7844cf83a62bacb840 WatchSource:0}: Error finding container 1b45d67c5f507030de46167240b45ab7624d2fa28ee3ee7844cf83a62bacb840: Status 404 returned error can't find the container with id 1b45d67c5f507030de46167240b45ab7624d2fa28ee3ee7844cf83a62bacb840 Nov 01 21:04:04.448796 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:04.448756 4484 patch_prober.go:29] interesting pod/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Nov 01 21:04:04.448982 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:04.448817 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m" podUID="3c818ecc-7536-4e57-93f6-40ba5942d318" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Nov 01 21:04:04.529339 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:04.529225 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m" Nov 01 21:04:04.534846 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:04.534795 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:04.698759 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:04.698723 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd840a-4t6zl" event={"ID":"4d85edeb-2371-4625-97b4-b41cba9a30d5","Type":"ContainerStarted","Data":"aaf96b0df739980fb14b5ec06637a2f9906b4ba05cc3c4ad9982d94dc25e51ad"} Nov 01 21:04:04.698929 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:04.698780 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd840a-4t6zl" event={"ID":"4d85edeb-2371-4625-97b4-b41cba9a30d5","Type":"ContainerStarted","Data":"1b45d67c5f507030de46167240b45ab7624d2fa28ee3ee7844cf83a62bacb840"} Nov 01 21:04:04.831249 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:04.831081 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd840a-4t6zl" podStartSLOduration=2.831058924 podStartE2EDuration="2.831058924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:04:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:04:04.820427239 +0000 UTC m=+1995.598067389" watchObservedRunningTime="2026-11-01 21:04:04.831058924 +0000 UTC m=+1995.608699056" Nov 01 21:04:05.197681 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:04:05.197599 4484 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c3c6fec_0841_4a0c_b761_626d8c53a1cb.slice/crio-a5d91c1d8490cb6b27129606cf5d3c39f95dedafea0ed794a5cd959283a6e608.scope\": RecentStats: unable to find data in memory cache], [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@15-35470-0.service\": RecentStats: unable to find data in memory cache]" Nov 01 21:04:05.448760 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:05.448699 4484 patch_prober.go:29] interesting pod/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Nov 01 21:04:05.448879 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:05.448757 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m" podUID="3c818ecc-7536-4e57-93f6-40ba5942d318" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Nov 01 21:04:05.695060 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:05.695035 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr" Nov 01 21:04:05.730935 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:05.730559 4484 generic.go:356] "Generic (PLEG): container finished" podID="4d85edeb-2371-4625-97b4-b41cba9a30d5" containerID="aaf96b0df739980fb14b5ec06637a2f9906b4ba05cc3c4ad9982d94dc25e51ad" exitCode=0 Nov 01 21:04:05.730935 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:05.730832 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd840a-4t6zl" event={"ID":"4d85edeb-2371-4625-97b4-b41cba9a30d5","Type":"ContainerDied","Data":"aaf96b0df739980fb14b5ec06637a2f9906b4ba05cc3c4ad9982d94dc25e51ad"} Nov 01 21:04:05.732722 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:05.732695 4484 generic.go:356] "Generic (PLEG): container finished" podID="43443369-5e6a-4b4b-ac9d-acbb320656c1" containerID="2937525e226fa5935d72fc9212c722f405c0da4cdd57776b4a699fd7255b5013" exitCode=0 Nov 01 21:04:05.732860 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:05.732740 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr" Nov 01 21:04:05.732860 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:05.732781 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr" event={"ID":"43443369-5e6a-4b4b-ac9d-acbb320656c1","Type":"ContainerDied","Data":"2937525e226fa5935d72fc9212c722f405c0da4cdd57776b4a699fd7255b5013"} Nov 01 21:04:05.732860 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:05.732831 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr" event={"ID":"43443369-5e6a-4b4b-ac9d-acbb320656c1","Type":"ContainerDied","Data":"041fd9f4eae5c2f129a03a8c6f21f790d83dae14c6ed47051c081bdf78d013c5"} Nov 01 21:04:05.732860 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:05.732848 4484 scope.go:122] "RemoveContainer" containerID="2937525e226fa5935d72fc9212c722f405c0da4cdd57776b4a699fd7255b5013" Nov 01 21:04:05.744047 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:05.744027 4484 scope.go:122] "RemoveContainer" containerID="2937525e226fa5935d72fc9212c722f405c0da4cdd57776b4a699fd7255b5013" Nov 01 21:04:05.744305 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:04:05.744282 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2937525e226fa5935d72fc9212c722f405c0da4cdd57776b4a699fd7255b5013\": container with ID starting with 2937525e226fa5935d72fc9212c722f405c0da4cdd57776b4a699fd7255b5013 not found: ID does not exist" containerID="2937525e226fa5935d72fc9212c722f405c0da4cdd57776b4a699fd7255b5013" Nov 01 21:04:05.744370 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:05.744316 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2937525e226fa5935d72fc9212c722f405c0da4cdd57776b4a699fd7255b5013"} err="failed to get container status \"2937525e226fa5935d72fc9212c722f405c0da4cdd57776b4a699fd7255b5013\": rpc error: code = NotFound desc = could not find container \"2937525e226fa5935d72fc9212c722f405c0da4cdd57776b4a699fd7255b5013\": container with ID starting with 2937525e226fa5935d72fc9212c722f405c0da4cdd57776b4a699fd7255b5013 not found: ID does not exist" Nov 01 21:04:05.860048 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:05.860008 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43443369-5e6a-4b4b-ac9d-acbb320656c1-kube-api-access-t9llj\" (UniqueName: \"kubernetes.io/projected/43443369-5e6a-4b4b-ac9d-acbb320656c1-kube-api-access-t9llj\") pod \"43443369-5e6a-4b4b-ac9d-acbb320656c1\" (UID: \"43443369-5e6a-4b4b-ac9d-acbb320656c1\") " Nov 01 21:04:05.860555 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:05.860059 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43443369-5e6a-4b4b-ac9d-acbb320656c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43443369-5e6a-4b4b-ac9d-acbb320656c1-workload-socket\") pod \"43443369-5e6a-4b4b-ac9d-acbb320656c1\" (UID: \"43443369-5e6a-4b4b-ac9d-acbb320656c1\") " Nov 01 21:04:05.860555 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:05.860083 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/43443369-5e6a-4b4b-ac9d-acbb320656c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43443369-5e6a-4b4b-ac9d-acbb320656c1-istio-podinfo\") pod \"43443369-5e6a-4b4b-ac9d-acbb320656c1\" (UID: \"43443369-5e6a-4b4b-ac9d-acbb320656c1\") " Nov 01 21:04:05.860555 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:05.860142 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43443369-5e6a-4b4b-ac9d-acbb320656c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43443369-5e6a-4b4b-ac9d-acbb320656c1-istiod-ca-cert\") pod \"43443369-5e6a-4b4b-ac9d-acbb320656c1\" (UID: \"43443369-5e6a-4b4b-ac9d-acbb320656c1\") " Nov 01 21:04:05.860555 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:05.860179 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43443369-5e6a-4b4b-ac9d-acbb320656c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43443369-5e6a-4b4b-ac9d-acbb320656c1-credential-socket\") pod \"43443369-5e6a-4b4b-ac9d-acbb320656c1\" (UID: \"43443369-5e6a-4b4b-ac9d-acbb320656c1\") " Nov 01 21:04:05.860555 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:05.860233 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43443369-5e6a-4b4b-ac9d-acbb320656c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43443369-5e6a-4b4b-ac9d-acbb320656c1-istio-data\") pod \"43443369-5e6a-4b4b-ac9d-acbb320656c1\" (UID: \"43443369-5e6a-4b4b-ac9d-acbb320656c1\") " Nov 01 21:04:05.860555 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:05.860312 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43443369-5e6a-4b4b-ac9d-acbb320656c1-istio-token\" (UniqueName: \"kubernetes.io/projected/43443369-5e6a-4b4b-ac9d-acbb320656c1-istio-token\") pod \"43443369-5e6a-4b4b-ac9d-acbb320656c1\" (UID: \"43443369-5e6a-4b4b-ac9d-acbb320656c1\") " Nov 01 21:04:05.860555 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:05.860326 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43443369-5e6a-4b4b-ac9d-acbb320656c1-workload-socket" pod "43443369-5e6a-4b4b-ac9d-acbb320656c1" (UID: "43443369-5e6a-4b4b-ac9d-acbb320656c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:04:05.860555 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:05.860342 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43443369-5e6a-4b4b-ac9d-acbb320656c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43443369-5e6a-4b4b-ac9d-acbb320656c1-istio-envoy\") pod \"43443369-5e6a-4b4b-ac9d-acbb320656c1\" (UID: \"43443369-5e6a-4b4b-ac9d-acbb320656c1\") " Nov 01 21:04:05.860555 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:05.860440 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43443369-5e6a-4b4b-ac9d-acbb320656c1-credential-socket" pod "43443369-5e6a-4b4b-ac9d-acbb320656c1" (UID: "43443369-5e6a-4b4b-ac9d-acbb320656c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:04:05.860555 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:05.860498 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43443369-5e6a-4b4b-ac9d-acbb320656c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43443369-5e6a-4b4b-ac9d-acbb320656c1-workload-certs\") pod \"43443369-5e6a-4b4b-ac9d-acbb320656c1\" (UID: \"43443369-5e6a-4b4b-ac9d-acbb320656c1\") " Nov 01 21:04:05.860555 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:05.860541 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43443369-5e6a-4b4b-ac9d-acbb320656c1-istiod-ca-cert" pod "43443369-5e6a-4b4b-ac9d-acbb320656c1" (UID: "43443369-5e6a-4b4b-ac9d-acbb320656c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:04:05.861165 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:05.860596 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43443369-5e6a-4b4b-ac9d-acbb320656c1-istio-data" pod "43443369-5e6a-4b4b-ac9d-acbb320656c1" (UID: "43443369-5e6a-4b4b-ac9d-acbb320656c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:04:05.861165 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:05.860829 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43443369-5e6a-4b4b-ac9d-acbb320656c1-workload-certs" pod "43443369-5e6a-4b4b-ac9d-acbb320656c1" (UID: "43443369-5e6a-4b4b-ac9d-acbb320656c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:04:05.861418 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:05.861398 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43443369-5e6a-4b4b-ac9d-acbb320656c1-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:05.861473 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:05.861425 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43443369-5e6a-4b4b-ac9d-acbb320656c1-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:05.861473 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:05.861439 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43443369-5e6a-4b4b-ac9d-acbb320656c1-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:05.861473 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:05.861453 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43443369-5e6a-4b4b-ac9d-acbb320656c1-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:05.861473 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:05.861466 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43443369-5e6a-4b4b-ac9d-acbb320656c1-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:05.862501 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:05.862480 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43443369-5e6a-4b4b-ac9d-acbb320656c1-istio-podinfo" pod "43443369-5e6a-4b4b-ac9d-acbb320656c1" (UID: "43443369-5e6a-4b4b-ac9d-acbb320656c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:04:05.862781 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:05.862754 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43443369-5e6a-4b4b-ac9d-acbb320656c1-istio-envoy" pod "43443369-5e6a-4b4b-ac9d-acbb320656c1" (UID: "43443369-5e6a-4b4b-ac9d-acbb320656c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:04:05.862899 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:05.862796 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43443369-5e6a-4b4b-ac9d-acbb320656c1-istio-token" pod "43443369-5e6a-4b4b-ac9d-acbb320656c1" (UID: "43443369-5e6a-4b4b-ac9d-acbb320656c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:04:05.862899 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:05.862864 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43443369-5e6a-4b4b-ac9d-acbb320656c1-kube-api-access-t9llj" pod "43443369-5e6a-4b4b-ac9d-acbb320656c1" (UID: "43443369-5e6a-4b4b-ac9d-acbb320656c1"). InnerVolumeSpecName "kube-api-access-t9llj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:04:05.963429 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:05.963393 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9llj\" (UniqueName: \"kubernetes.io/projected/43443369-5e6a-4b4b-ac9d-acbb320656c1-kube-api-access-t9llj\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:05.963429 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:05.963428 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43443369-5e6a-4b4b-ac9d-acbb320656c1-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:05.963429 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:05.963437 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43443369-5e6a-4b4b-ac9d-acbb320656c1-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:05.963725 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:05.963446 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43443369-5e6a-4b4b-ac9d-acbb320656c1-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:06.166982 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:06.166942 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr"] Nov 01 21:04:06.186391 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:06.186350 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-944263-openshift-default-5f78997bc9-l7fxr"] Nov 01 21:04:06.448420 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:06.448327 4484 patch_prober.go:29] interesting pod/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Nov 01 21:04:06.448420 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:06.448392 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m" podUID="3c818ecc-7536-4e57-93f6-40ba5942d318" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Nov 01 21:04:06.972909 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:06.972877 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd840a-4t6zl" Nov 01 21:04:07.086258 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:07.086167 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d85edeb-2371-4625-97b4-b41cba9a30d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d85edeb-2371-4625-97b4-b41cba9a30d5-maas-api-xks-ca\") pod \"4d85edeb-2371-4625-97b4-b41cba9a30d5\" (UID: \"4d85edeb-2371-4625-97b4-b41cba9a30d5\") " Nov 01 21:04:07.086258 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:07.086219 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d85edeb-2371-4625-97b4-b41cba9a30d5-kube-api-access-xzs8v\" (UniqueName: \"kubernetes.io/projected/4d85edeb-2371-4625-97b4-b41cba9a30d5-kube-api-access-xzs8v\") pod \"4d85edeb-2371-4625-97b4-b41cba9a30d5\" (UID: \"4d85edeb-2371-4625-97b4-b41cba9a30d5\") " Nov 01 21:04:07.086453 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:07.086288 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d85edeb-2371-4625-97b4-b41cba9a30d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d85edeb-2371-4625-97b4-b41cba9a30d5-maas-api-openshift-ca\") pod \"4d85edeb-2371-4625-97b4-b41cba9a30d5\" (UID: \"4d85edeb-2371-4625-97b4-b41cba9a30d5\") " Nov 01 21:04:07.086694 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:07.086644 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d85edeb-2371-4625-97b4-b41cba9a30d5-maas-api-openshift-ca" pod "4d85edeb-2371-4625-97b4-b41cba9a30d5" (UID: "4d85edeb-2371-4625-97b4-b41cba9a30d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:04:07.086893 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:07.086872 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d85edeb-2371-4625-97b4-b41cba9a30d5-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:07.089176 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:07.089145 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d85edeb-2371-4625-97b4-b41cba9a30d5-maas-api-xks-ca" pod "4d85edeb-2371-4625-97b4-b41cba9a30d5" (UID: "4d85edeb-2371-4625-97b4-b41cba9a30d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:04:07.089324 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:07.089179 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d85edeb-2371-4625-97b4-b41cba9a30d5-kube-api-access-xzs8v" pod "4d85edeb-2371-4625-97b4-b41cba9a30d5" (UID: "4d85edeb-2371-4625-97b4-b41cba9a30d5"). InnerVolumeSpecName "kube-api-access-xzs8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:04:07.188322 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:07.188289 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d85edeb-2371-4625-97b4-b41cba9a30d5-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:07.188322 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:07.188320 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzs8v\" (UniqueName: \"kubernetes.io/projected/4d85edeb-2371-4625-97b4-b41cba9a30d5-kube-api-access-xzs8v\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:07.361663 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:07.361566 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b27e66-6df49bd599-n7wtm"] Nov 01 21:04:07.374976 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:07.374947 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43443369-5e6a-4b4b-ac9d-acbb320656c1" path="/var/lib/kubelet/pods/43443369-5e6a-4b4b-ac9d-acbb320656c1/volumes" Nov 01 21:04:07.448796 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:07.448762 4484 patch_prober.go:29] interesting pod/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Nov 01 21:04:07.449037 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:07.448821 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m" podUID="3c818ecc-7536-4e57-93f6-40ba5942d318" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Nov 01 21:04:07.469620 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:07.469579 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b27e66-5db66b9467-z8rv7"] Nov 01 21:04:07.746578 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:07.746540 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:07.751228 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:07.751194 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd840a-4t6zl" event={"ID":"4d85edeb-2371-4625-97b4-b41cba9a30d5","Type":"ContainerDied","Data":"1b45d67c5f507030de46167240b45ab7624d2fa28ee3ee7844cf83a62bacb840"} Nov 01 21:04:07.751228 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:07.751220 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd840a-4t6zl" Nov 01 21:04:07.751467 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:07.751246 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b45d67c5f507030de46167240b45ab7624d2fa28ee3ee7844cf83a62bacb840" Nov 01 21:04:07.904221 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:07.904161 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:08.453725 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:08.453697 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m" Nov 01 21:04:08.454952 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:08.454932 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m" Nov 01 21:04:09.895591 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:09.895552 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:10.898292 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:04:10.898255 4484 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c3c6fec_0841_4a0c_b761_626d8c53a1cb.slice/crio-a5d91c1d8490cb6b27129606cf5d3c39f95dedafea0ed794a5cd959283a6e608.scope\": RecentStats: unable to find data in memory cache], [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@15-35470-0.service\": RecentStats: unable to find data in memory cache]" Nov 01 21:04:12.362094 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:12.362053 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-b27e66-6df49bd599-n7wtm" podUID="3ce576d7-7d1e-4b92-ad28-654f4426099c" containerName="payload-processing" containerID="cri-o://955b611b7c742efb3af01bbac2d28300101fc5426f619758ef87fe45a54f45c6" gracePeriod=25 Nov 01 21:04:12.470223 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:12.470170 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b27e66-5db66b9467-z8rv7" podUID="4b167c30-a633-471e-a8f4-7f6933c81f0d" containerName="payload-pre-processing" containerID="cri-o://b77bfc23b1adb869151c34434aa56228107e2ea6994a1f5995919c61803d29c0" gracePeriod=25 Nov 01 21:04:12.661010 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:12.660961 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b27e66-6df49bd599-n7wtm" Nov 01 21:04:12.690073 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:12.690034 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ce576d7-7d1e-4b92-ad28-654f4426099c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ce576d7-7d1e-4b92-ad28-654f4426099c-plugins-config-volume\") pod \"3ce576d7-7d1e-4b92-ad28-654f4426099c\" (UID: \"3ce576d7-7d1e-4b92-ad28-654f4426099c\") " Nov 01 21:04:12.690257 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:12.690240 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ce576d7-7d1e-4b92-ad28-654f4426099c-kube-api-access-p2cbh\" (UniqueName: \"kubernetes.io/projected/3ce576d7-7d1e-4b92-ad28-654f4426099c-kube-api-access-p2cbh\") pod \"3ce576d7-7d1e-4b92-ad28-654f4426099c\" (UID: \"3ce576d7-7d1e-4b92-ad28-654f4426099c\") " Nov 01 21:04:12.690500 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:12.690471 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ce576d7-7d1e-4b92-ad28-654f4426099c-plugins-config-volume" pod "3ce576d7-7d1e-4b92-ad28-654f4426099c" (UID: "3ce576d7-7d1e-4b92-ad28-654f4426099c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:04:12.690973 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:12.690947 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ce576d7-7d1e-4b92-ad28-654f4426099c-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:12.692527 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:12.692495 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ce576d7-7d1e-4b92-ad28-654f4426099c-kube-api-access-p2cbh" pod "3ce576d7-7d1e-4b92-ad28-654f4426099c" (UID: "3ce576d7-7d1e-4b92-ad28-654f4426099c"). InnerVolumeSpecName "kube-api-access-p2cbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:04:12.764238 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:12.764212 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b27e66-5db66b9467-z8rv7" Nov 01 21:04:12.791574 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:12.791536 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b167c30-a633-471e-a8f4-7f6933c81f0d-kube-api-access-sd9f8\" (UniqueName: \"kubernetes.io/projected/4b167c30-a633-471e-a8f4-7f6933c81f0d-kube-api-access-sd9f8\") pod \"4b167c30-a633-471e-a8f4-7f6933c81f0d\" (UID: \"4b167c30-a633-471e-a8f4-7f6933c81f0d\") " Nov 01 21:04:12.791874 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:12.791853 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b167c30-a633-471e-a8f4-7f6933c81f0d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b167c30-a633-471e-a8f4-7f6933c81f0d-plugins-config-volume\") pod \"4b167c30-a633-471e-a8f4-7f6933c81f0d\" (UID: \"4b167c30-a633-471e-a8f4-7f6933c81f0d\") " Nov 01 21:04:12.792658 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:12.792632 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2cbh\" (UniqueName: \"kubernetes.io/projected/3ce576d7-7d1e-4b92-ad28-654f4426099c-kube-api-access-p2cbh\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:12.792658 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:12.792635 4484 generic.go:356] "Generic (PLEG): container finished" podID="3ce576d7-7d1e-4b92-ad28-654f4426099c" containerID="955b611b7c742efb3af01bbac2d28300101fc5426f619758ef87fe45a54f45c6" exitCode=0 Nov 01 21:04:12.792803 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:12.792669 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b27e66-6df49bd599-n7wtm" event={"ID":"3ce576d7-7d1e-4b92-ad28-654f4426099c","Type":"ContainerDied","Data":"955b611b7c742efb3af01bbac2d28300101fc5426f619758ef87fe45a54f45c6"} Nov 01 21:04:12.792803 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:12.792722 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b27e66-6df49bd599-n7wtm" event={"ID":"3ce576d7-7d1e-4b92-ad28-654f4426099c","Type":"ContainerDied","Data":"0342e5ab57e55af4fd8e7c80c20cddf130726d81d53e3d63e5c06eaa7e2107dc"} Nov 01 21:04:12.792803 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:12.792750 4484 scope.go:122] "RemoveContainer" containerID="955b611b7c742efb3af01bbac2d28300101fc5426f619758ef87fe45a54f45c6" Nov 01 21:04:12.792934 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:12.792922 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b27e66-6df49bd599-n7wtm" Nov 01 21:04:12.795380 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:12.795354 4484 generic.go:356] "Generic (PLEG): container finished" podID="4b167c30-a633-471e-a8f4-7f6933c81f0d" containerID="b77bfc23b1adb869151c34434aa56228107e2ea6994a1f5995919c61803d29c0" exitCode=0 Nov 01 21:04:12.795539 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:12.795398 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b27e66-5db66b9467-z8rv7" Nov 01 21:04:12.795737 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:12.795394 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b27e66-5db66b9467-z8rv7" event={"ID":"4b167c30-a633-471e-a8f4-7f6933c81f0d","Type":"ContainerDied","Data":"b77bfc23b1adb869151c34434aa56228107e2ea6994a1f5995919c61803d29c0"} Nov 01 21:04:12.795737 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:12.795670 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b27e66-5db66b9467-z8rv7" event={"ID":"4b167c30-a633-471e-a8f4-7f6933c81f0d","Type":"ContainerDied","Data":"ade80a9aeafab87ad4f3960a2c1d556f636db4d0f7100af78286f707f196d77d"} Nov 01 21:04:12.798681 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:12.798654 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b167c30-a633-471e-a8f4-7f6933c81f0d-plugins-config-volume" pod "4b167c30-a633-471e-a8f4-7f6933c81f0d" (UID: "4b167c30-a633-471e-a8f4-7f6933c81f0d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:04:12.801112 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:12.801081 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b167c30-a633-471e-a8f4-7f6933c81f0d-kube-api-access-sd9f8" pod "4b167c30-a633-471e-a8f4-7f6933c81f0d" (UID: "4b167c30-a633-471e-a8f4-7f6933c81f0d"). InnerVolumeSpecName "kube-api-access-sd9f8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:04:12.810026 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:12.809981 4484 scope.go:122] "RemoveContainer" containerID="955b611b7c742efb3af01bbac2d28300101fc5426f619758ef87fe45a54f45c6" Nov 01 21:04:12.810396 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:04:12.810367 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"955b611b7c742efb3af01bbac2d28300101fc5426f619758ef87fe45a54f45c6\": container with ID starting with 955b611b7c742efb3af01bbac2d28300101fc5426f619758ef87fe45a54f45c6 not found: ID does not exist" containerID="955b611b7c742efb3af01bbac2d28300101fc5426f619758ef87fe45a54f45c6" Nov 01 21:04:12.810498 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:12.810405 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"955b611b7c742efb3af01bbac2d28300101fc5426f619758ef87fe45a54f45c6"} err="failed to get container status \"955b611b7c742efb3af01bbac2d28300101fc5426f619758ef87fe45a54f45c6\": rpc error: code = NotFound desc = could not find container \"955b611b7c742efb3af01bbac2d28300101fc5426f619758ef87fe45a54f45c6\": container with ID starting with 955b611b7c742efb3af01bbac2d28300101fc5426f619758ef87fe45a54f45c6 not found: ID does not exist" Nov 01 21:04:12.810498 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:12.810425 4484 scope.go:122] "RemoveContainer" containerID="b77bfc23b1adb869151c34434aa56228107e2ea6994a1f5995919c61803d29c0" Nov 01 21:04:12.821103 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:12.821082 4484 scope.go:122] "RemoveContainer" containerID="b77bfc23b1adb869151c34434aa56228107e2ea6994a1f5995919c61803d29c0" Nov 01 21:04:12.824581 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:04:12.824551 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b77bfc23b1adb869151c34434aa56228107e2ea6994a1f5995919c61803d29c0\": container with ID starting with b77bfc23b1adb869151c34434aa56228107e2ea6994a1f5995919c61803d29c0 not found: ID does not exist" containerID="b77bfc23b1adb869151c34434aa56228107e2ea6994a1f5995919c61803d29c0" Nov 01 21:04:12.824690 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:12.824603 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b77bfc23b1adb869151c34434aa56228107e2ea6994a1f5995919c61803d29c0"} err="failed to get container status \"b77bfc23b1adb869151c34434aa56228107e2ea6994a1f5995919c61803d29c0\": rpc error: code = NotFound desc = could not find container \"b77bfc23b1adb869151c34434aa56228107e2ea6994a1f5995919c61803d29c0\": container with ID starting with b77bfc23b1adb869151c34434aa56228107e2ea6994a1f5995919c61803d29c0 not found: ID does not exist" Nov 01 21:04:12.894534 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:12.894441 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b167c30-a633-471e-a8f4-7f6933c81f0d-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:12.894534 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:12.894472 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd9f8\" (UniqueName: \"kubernetes.io/projected/4b167c30-a633-471e-a8f4-7f6933c81f0d-kube-api-access-sd9f8\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:13.062147 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:13.062110 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b27e66-6df49bd599-n7wtm"] Nov 01 21:04:13.066385 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:13.066353 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b27e66-6df49bd599-n7wtm"] Nov 01 21:04:13.221654 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:13.221578 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b27e66-5db66b9467-z8rv7"] Nov 01 21:04:13.225481 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:13.225459 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b27e66-5db66b9467-z8rv7"] Nov 01 21:04:13.375861 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:13.375827 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ce576d7-7d1e-4b92-ad28-654f4426099c" path="/var/lib/kubelet/pods/3ce576d7-7d1e-4b92-ad28-654f4426099c/volumes" Nov 01 21:04:13.376421 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:13.376400 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b167c30-a633-471e-a8f4-7f6933c81f0d" path="/var/lib/kubelet/pods/4b167c30-a633-471e-a8f4-7f6933c81f0d/volumes" Nov 01 21:04:14.761804 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:14.761764 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:15.235846 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:04:15.235758 4484 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c3c6fec_0841_4a0c_b761_626d8c53a1cb.slice/crio-a5d91c1d8490cb6b27129606cf5d3c39f95dedafea0ed794a5cd959283a6e608.scope\": RecentStats: unable to find data in memory cache], [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@15-35470-0.service\": RecentStats: unable to find data in memory cache]" Nov 01 21:04:15.768314 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:04:15.768273 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:04:16.847524 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:16.847487 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dd840a-68df564877-hxfz7"] Nov 01 21:04:16.967717 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:16.967684 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dd840a-5c9b4679f5-wspqf"] Nov 01 21:04:17.233913 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:17.233820 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:17.453669 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:17.418226 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:19.318795 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:19.318760 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:21.122338 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:21.122305 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:21.848840 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:21.848795 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-dd840a-68df564877-hxfz7" podUID="ff449990-bd5a-4b0c-bd0a-7880dbcdd90f" containerName="payload-processing" containerID="cri-o://38b724e5f20c3f6cacc642104f715c9ba8022cb92ae23789209c89a4668e2729" gracePeriod=25 Nov 01 21:04:21.968571 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:21.968525 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd840a-5c9b4679f5-wspqf" podUID="40f370d3-946f-4cf1-805d-f245ca48e9a9" containerName="payload-pre-processing" containerID="cri-o://22270c441f39a163bcb8e02c6745570f15daa1e5a3e99ef9c051425059aca257" gracePeriod=25 Nov 01 21:04:22.167175 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:22.167139 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-dd840a-68df564877-hxfz7" Nov 01 21:04:22.261605 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:22.260378 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff449990-bd5a-4b0c-bd0a-7880dbcdd90f-kube-api-access-vj4pm\" (UniqueName: \"kubernetes.io/projected/ff449990-bd5a-4b0c-bd0a-7880dbcdd90f-kube-api-access-vj4pm\") pod \"ff449990-bd5a-4b0c-bd0a-7880dbcdd90f\" (UID: \"ff449990-bd5a-4b0c-bd0a-7880dbcdd90f\") " Nov 01 21:04:22.261605 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:22.260549 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff449990-bd5a-4b0c-bd0a-7880dbcdd90f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff449990-bd5a-4b0c-bd0a-7880dbcdd90f-plugins-config-volume\") pod \"ff449990-bd5a-4b0c-bd0a-7880dbcdd90f\" (UID: \"ff449990-bd5a-4b0c-bd0a-7880dbcdd90f\") " Nov 01 21:04:22.261605 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:22.261562 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff449990-bd5a-4b0c-bd0a-7880dbcdd90f-plugins-config-volume" pod "ff449990-bd5a-4b0c-bd0a-7880dbcdd90f" (UID: "ff449990-bd5a-4b0c-bd0a-7880dbcdd90f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:04:22.263566 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:22.263533 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff449990-bd5a-4b0c-bd0a-7880dbcdd90f-kube-api-access-vj4pm" pod "ff449990-bd5a-4b0c-bd0a-7880dbcdd90f" (UID: "ff449990-bd5a-4b0c-bd0a-7880dbcdd90f"). InnerVolumeSpecName "kube-api-access-vj4pm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:04:22.286181 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:22.286155 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd840a-5c9b4679f5-wspqf" Nov 01 21:04:22.362385 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:22.362347 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40f370d3-946f-4cf1-805d-f245ca48e9a9-kube-api-access-j889c\" (UniqueName: \"kubernetes.io/projected/40f370d3-946f-4cf1-805d-f245ca48e9a9-kube-api-access-j889c\") pod \"40f370d3-946f-4cf1-805d-f245ca48e9a9\" (UID: \"40f370d3-946f-4cf1-805d-f245ca48e9a9\") " Nov 01 21:04:22.362667 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:22.362645 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40f370d3-946f-4cf1-805d-f245ca48e9a9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40f370d3-946f-4cf1-805d-f245ca48e9a9-plugins-config-volume\") pod \"40f370d3-946f-4cf1-805d-f245ca48e9a9\" (UID: \"40f370d3-946f-4cf1-805d-f245ca48e9a9\") " Nov 01 21:04:22.363084 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:22.363059 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40f370d3-946f-4cf1-805d-f245ca48e9a9-plugins-config-volume" pod "40f370d3-946f-4cf1-805d-f245ca48e9a9" (UID: "40f370d3-946f-4cf1-805d-f245ca48e9a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:04:22.363290 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:22.363276 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj4pm\" (UniqueName: \"kubernetes.io/projected/ff449990-bd5a-4b0c-bd0a-7880dbcdd90f-kube-api-access-vj4pm\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:22.363332 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:22.363293 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40f370d3-946f-4cf1-805d-f245ca48e9a9-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:22.363332 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:22.363304 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff449990-bd5a-4b0c-bd0a-7880dbcdd90f-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:22.364565 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:22.364541 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40f370d3-946f-4cf1-805d-f245ca48e9a9-kube-api-access-j889c" pod "40f370d3-946f-4cf1-805d-f245ca48e9a9" (UID: "40f370d3-946f-4cf1-805d-f245ca48e9a9"). InnerVolumeSpecName "kube-api-access-j889c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:04:22.432952 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:22.432854 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:22.465119 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:22.465087 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j889c\" (UniqueName: \"kubernetes.io/projected/40f370d3-946f-4cf1-805d-f245ca48e9a9-kube-api-access-j889c\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:22.591715 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:22.591664 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:22.906674 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:22.906637 4484 generic.go:356] "Generic (PLEG): container finished" podID="ff449990-bd5a-4b0c-bd0a-7880dbcdd90f" containerID="38b724e5f20c3f6cacc642104f715c9ba8022cb92ae23789209c89a4668e2729" exitCode=0 Nov 01 21:04:22.906850 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:22.906685 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-dd840a-68df564877-hxfz7" Nov 01 21:04:22.906850 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:22.906718 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-dd840a-68df564877-hxfz7" event={"ID":"ff449990-bd5a-4b0c-bd0a-7880dbcdd90f","Type":"ContainerDied","Data":"38b724e5f20c3f6cacc642104f715c9ba8022cb92ae23789209c89a4668e2729"} Nov 01 21:04:22.906850 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:22.906782 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-dd840a-68df564877-hxfz7" event={"ID":"ff449990-bd5a-4b0c-bd0a-7880dbcdd90f","Type":"ContainerDied","Data":"f8ab68ef3f92fbc3c87a7910f4a7103bde3e7b2ad5de1c44c1eb128346409b9d"} Nov 01 21:04:22.906850 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:22.906810 4484 scope.go:122] "RemoveContainer" containerID="38b724e5f20c3f6cacc642104f715c9ba8022cb92ae23789209c89a4668e2729" Nov 01 21:04:22.908498 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:22.908477 4484 generic.go:356] "Generic (PLEG): container finished" podID="40f370d3-946f-4cf1-805d-f245ca48e9a9" containerID="22270c441f39a163bcb8e02c6745570f15daa1e5a3e99ef9c051425059aca257" exitCode=0 Nov 01 21:04:22.908584 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:22.908510 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd840a-5c9b4679f5-wspqf" event={"ID":"40f370d3-946f-4cf1-805d-f245ca48e9a9","Type":"ContainerDied","Data":"22270c441f39a163bcb8e02c6745570f15daa1e5a3e99ef9c051425059aca257"} Nov 01 21:04:22.908584 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:22.908525 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd840a-5c9b4679f5-wspqf" Nov 01 21:04:22.908584 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:22.908543 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd840a-5c9b4679f5-wspqf" event={"ID":"40f370d3-946f-4cf1-805d-f245ca48e9a9","Type":"ContainerDied","Data":"8bc1c560c6e26b2a99166fccadc555914a677eded24b1997898ca1c926b3a91c"} Nov 01 21:04:22.919745 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:22.919729 4484 scope.go:122] "RemoveContainer" containerID="38b724e5f20c3f6cacc642104f715c9ba8022cb92ae23789209c89a4668e2729" Nov 01 21:04:22.920056 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:04:22.920033 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38b724e5f20c3f6cacc642104f715c9ba8022cb92ae23789209c89a4668e2729\": container with ID starting with 38b724e5f20c3f6cacc642104f715c9ba8022cb92ae23789209c89a4668e2729 not found: ID does not exist" containerID="38b724e5f20c3f6cacc642104f715c9ba8022cb92ae23789209c89a4668e2729" Nov 01 21:04:22.920121 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:22.920066 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38b724e5f20c3f6cacc642104f715c9ba8022cb92ae23789209c89a4668e2729"} err="failed to get container status \"38b724e5f20c3f6cacc642104f715c9ba8022cb92ae23789209c89a4668e2729\": rpc error: code = NotFound desc = could not find container \"38b724e5f20c3f6cacc642104f715c9ba8022cb92ae23789209c89a4668e2729\": container with ID starting with 38b724e5f20c3f6cacc642104f715c9ba8022cb92ae23789209c89a4668e2729 not found: ID does not exist" Nov 01 21:04:22.920121 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:22.920084 4484 scope.go:122] "RemoveContainer" containerID="22270c441f39a163bcb8e02c6745570f15daa1e5a3e99ef9c051425059aca257" Nov 01 21:04:22.930471 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:22.930451 4484 scope.go:122] "RemoveContainer" containerID="22270c441f39a163bcb8e02c6745570f15daa1e5a3e99ef9c051425059aca257" Nov 01 21:04:22.930751 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:04:22.930733 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22270c441f39a163bcb8e02c6745570f15daa1e5a3e99ef9c051425059aca257\": container with ID starting with 22270c441f39a163bcb8e02c6745570f15daa1e5a3e99ef9c051425059aca257 not found: ID does not exist" containerID="22270c441f39a163bcb8e02c6745570f15daa1e5a3e99ef9c051425059aca257" Nov 01 21:04:22.930821 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:22.930760 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22270c441f39a163bcb8e02c6745570f15daa1e5a3e99ef9c051425059aca257"} err="failed to get container status \"22270c441f39a163bcb8e02c6745570f15daa1e5a3e99ef9c051425059aca257\": rpc error: code = NotFound desc = could not find container \"22270c441f39a163bcb8e02c6745570f15daa1e5a3e99ef9c051425059aca257\": container with ID starting with 22270c441f39a163bcb8e02c6745570f15daa1e5a3e99ef9c051425059aca257 not found: ID does not exist" Nov 01 21:04:23.076761 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:23.076720 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dd840a-5c9b4679f5-wspqf"] Nov 01 21:04:23.080665 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:23.080629 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dd840a-5c9b4679f5-wspqf"] Nov 01 21:04:23.192460 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:23.192379 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dd840a-68df564877-hxfz7"] Nov 01 21:04:23.199633 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:23.199605 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dd840a-68df564877-hxfz7"] Nov 01 21:04:23.373274 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:23.373241 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40f370d3-946f-4cf1-805d-f245ca48e9a9" path="/var/lib/kubelet/pods/40f370d3-946f-4cf1-805d-f245ca48e9a9/volumes" Nov 01 21:04:23.373744 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:23.373730 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff449990-bd5a-4b0c-bd0a-7880dbcdd90f" path="/var/lib/kubelet/pods/ff449990-bd5a-4b0c-bd0a-7880dbcdd90f/volumes" Nov 01 21:04:25.326572 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:04:25.326534 4484 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@15-35470-0.service\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c3c6fec_0841_4a0c_b761_626d8c53a1cb.slice/crio-a5d91c1d8490cb6b27129606cf5d3c39f95dedafea0ed794a5cd959283a6e608.scope\": RecentStats: unable to find data in memory cache]" Nov 01 21:04:25.810168 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:04:25.810130 4484 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@15-35470-0.service\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c3c6fec_0841_4a0c_b761_626d8c53a1cb.slice/crio-a5d91c1d8490cb6b27129606cf5d3c39f95dedafea0ed794a5cd959283a6e608.scope\": RecentStats: unable to find data in memory cache]" Nov 01 21:04:25.844697 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:25.844665 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:29.668933 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:29.668896 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:29.832040 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:29.831989 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:30.019785 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:30.019741 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:31.611808 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:31.611725 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:31.832589 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:31.832550 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:31.998596 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:31.998553 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:32.202259 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:32.202225 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-de8145-68zmw"] Nov 01 21:04:32.252802 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:32.252722 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-de8145-68zmw"] Nov 01 21:04:32.624985 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:32.624905 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:33.373507 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:33.373472 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4ab60f6-33ce-447f-884d-a1384735cc84" path="/var/lib/kubelet/pods/a4ab60f6-33ce-447f-884d-a1384735cc84/volumes" Nov 01 21:04:34.331733 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:34.331669 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8"] Nov 01 21:04:34.332493 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:34.332477 4484 cpu_manager.go:414] "containerMap: removing container" podUID="4d85edeb-2371-4625-97b4-b41cba9a30d5" containerName="revoke-keys" Nov 01 21:04:34.332550 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:34.332495 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d85edeb-2371-4625-97b4-b41cba9a30d5" containerName="revoke-keys" Nov 01 21:04:34.332550 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:34.332517 4484 cpu_manager.go:414] "containerMap: removing container" podUID="4b167c30-a633-471e-a8f4-7f6933c81f0d" containerName="payload-pre-processing" Nov 01 21:04:34.332550 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:34.332522 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b167c30-a633-471e-a8f4-7f6933c81f0d" containerName="payload-pre-processing" Nov 01 21:04:34.332550 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:34.332541 4484 cpu_manager.go:414] "containerMap: removing container" podUID="40f370d3-946f-4cf1-805d-f245ca48e9a9" containerName="payload-pre-processing" Nov 01 21:04:34.332550 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:34.332546 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40f370d3-946f-4cf1-805d-f245ca48e9a9" containerName="payload-pre-processing" Nov 01 21:04:34.332714 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:34.332559 4484 cpu_manager.go:414] "containerMap: removing container" podUID="43443369-5e6a-4b4b-ac9d-acbb320656c1" containerName="istio-proxy" Nov 01 21:04:34.332714 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:34.332564 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43443369-5e6a-4b4b-ac9d-acbb320656c1" containerName="istio-proxy" Nov 01 21:04:34.332714 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:34.332575 4484 cpu_manager.go:414] "containerMap: removing container" podUID="ff449990-bd5a-4b0c-bd0a-7880dbcdd90f" containerName="payload-processing" Nov 01 21:04:34.332714 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:34.332580 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff449990-bd5a-4b0c-bd0a-7880dbcdd90f" containerName="payload-processing" Nov 01 21:04:34.332714 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:34.332586 4484 cpu_manager.go:414] "containerMap: removing container" podUID="3ce576d7-7d1e-4b92-ad28-654f4426099c" containerName="payload-processing" Nov 01 21:04:34.332714 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:34.332591 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ce576d7-7d1e-4b92-ad28-654f4426099c" containerName="payload-processing" Nov 01 21:04:34.332714 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:34.332690 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="43443369-5e6a-4b4b-ac9d-acbb320656c1" containerName="istio-proxy" Nov 01 21:04:34.332714 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:34.332699 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff449990-bd5a-4b0c-bd0a-7880dbcdd90f" containerName="payload-processing" Nov 01 21:04:34.332714 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:34.332705 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ce576d7-7d1e-4b92-ad28-654f4426099c" containerName="payload-processing" Nov 01 21:04:34.332714 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:34.332717 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d85edeb-2371-4625-97b4-b41cba9a30d5" containerName="revoke-keys" Nov 01 21:04:34.333050 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:34.332725 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b167c30-a633-471e-a8f4-7f6933c81f0d" containerName="payload-pre-processing" Nov 01 21:04:34.333050 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:34.332734 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="40f370d3-946f-4cf1-805d-f245ca48e9a9" containerName="payload-pre-processing" Nov 01 21:04:34.337825 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:34.337806 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8" Nov 01 21:04:34.391137 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:34.391104 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-11a069b1-openshift-default-dockercfg-c82c9\"" Nov 01 21:04:34.448928 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:34.448888 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8"] Nov 01 21:04:34.495652 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:34.495616 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b18c47d2-1694-4961-9638-be5cb8868cc3-istio-podinfo\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8\" (UID: \"b18c47d2-1694-4961-9638-be5cb8868cc3\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8" Nov 01 21:04:34.495858 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:34.495658 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b18c47d2-1694-4961-9638-be5cb8868cc3-workload-socket\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8\" (UID: \"b18c47d2-1694-4961-9638-be5cb8868cc3\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8" Nov 01 21:04:34.495858 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:34.495685 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b18c47d2-1694-4961-9638-be5cb8868cc3-credential-socket\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8\" (UID: \"b18c47d2-1694-4961-9638-be5cb8868cc3\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8" Nov 01 21:04:34.495858 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:34.495812 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b18c47d2-1694-4961-9638-be5cb8868cc3-istio-token\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8\" (UID: \"b18c47d2-1694-4961-9638-be5cb8868cc3\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8" Nov 01 21:04:34.496031 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:34.495943 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b18c47d2-1694-4961-9638-be5cb8868cc3-istio-data\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8\" (UID: \"b18c47d2-1694-4961-9638-be5cb8868cc3\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8" Nov 01 21:04:34.496031 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:34.495979 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b18c47d2-1694-4961-9638-be5cb8868cc3-istio-envoy\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8\" (UID: \"b18c47d2-1694-4961-9638-be5cb8868cc3\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8" Nov 01 21:04:34.496121 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:34.496082 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b18c47d2-1694-4961-9638-be5cb8868cc3-workload-certs\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8\" (UID: \"b18c47d2-1694-4961-9638-be5cb8868cc3\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8" Nov 01 21:04:34.496242 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:34.496223 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b18c47d2-1694-4961-9638-be5cb8868cc3-istiod-ca-cert\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8\" (UID: \"b18c47d2-1694-4961-9638-be5cb8868cc3\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8" Nov 01 21:04:34.496317 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:34.496299 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g8gg\" (UniqueName: \"kubernetes.io/projected/b18c47d2-1694-4961-9638-be5cb8868cc3-kube-api-access-7g8gg\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8\" (UID: \"b18c47d2-1694-4961-9638-be5cb8868cc3\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8" Nov 01 21:04:34.597724 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:34.597627 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b18c47d2-1694-4961-9638-be5cb8868cc3-istio-data\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8\" (UID: \"b18c47d2-1694-4961-9638-be5cb8868cc3\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8" Nov 01 21:04:34.597724 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:34.597693 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b18c47d2-1694-4961-9638-be5cb8868cc3-istio-envoy\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8\" (UID: \"b18c47d2-1694-4961-9638-be5cb8868cc3\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8" Nov 01 21:04:34.597977 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:34.597754 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b18c47d2-1694-4961-9638-be5cb8868cc3-workload-certs\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8\" (UID: \"b18c47d2-1694-4961-9638-be5cb8868cc3\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8" Nov 01 21:04:34.597977 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:34.597867 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b18c47d2-1694-4961-9638-be5cb8868cc3-istiod-ca-cert\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8\" (UID: \"b18c47d2-1694-4961-9638-be5cb8868cc3\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8" Nov 01 21:04:34.597977 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:34.597926 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g8gg\" (UniqueName: \"kubernetes.io/projected/b18c47d2-1694-4961-9638-be5cb8868cc3-kube-api-access-7g8gg\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8\" (UID: \"b18c47d2-1694-4961-9638-be5cb8868cc3\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8" Nov 01 21:04:34.598189 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:34.598026 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b18c47d2-1694-4961-9638-be5cb8868cc3-istio-podinfo\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8\" (UID: \"b18c47d2-1694-4961-9638-be5cb8868cc3\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8" Nov 01 21:04:34.598189 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:34.598065 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b18c47d2-1694-4961-9638-be5cb8868cc3-workload-socket\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8\" (UID: \"b18c47d2-1694-4961-9638-be5cb8868cc3\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8" Nov 01 21:04:34.598189 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:34.598083 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b18c47d2-1694-4961-9638-be5cb8868cc3-istio-data\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8\" (UID: \"b18c47d2-1694-4961-9638-be5cb8868cc3\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8" Nov 01 21:04:34.598189 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:34.598101 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b18c47d2-1694-4961-9638-be5cb8868cc3-credential-socket\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8\" (UID: \"b18c47d2-1694-4961-9638-be5cb8868cc3\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8" Nov 01 21:04:34.598189 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:34.598165 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b18c47d2-1694-4961-9638-be5cb8868cc3-istio-token\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8\" (UID: \"b18c47d2-1694-4961-9638-be5cb8868cc3\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8" Nov 01 21:04:34.598189 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:34.598168 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b18c47d2-1694-4961-9638-be5cb8868cc3-workload-certs\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8\" (UID: \"b18c47d2-1694-4961-9638-be5cb8868cc3\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8" Nov 01 21:04:34.598518 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:34.598413 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b18c47d2-1694-4961-9638-be5cb8868cc3-credential-socket\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8\" (UID: \"b18c47d2-1694-4961-9638-be5cb8868cc3\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8" Nov 01 21:04:34.598518 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:34.598435 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b18c47d2-1694-4961-9638-be5cb8868cc3-workload-socket\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8\" (UID: \"b18c47d2-1694-4961-9638-be5cb8868cc3\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8" Nov 01 21:04:34.598653 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:34.598629 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b18c47d2-1694-4961-9638-be5cb8868cc3-istiod-ca-cert\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8\" (UID: \"b18c47d2-1694-4961-9638-be5cb8868cc3\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8" Nov 01 21:04:34.600564 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:34.600539 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b18c47d2-1694-4961-9638-be5cb8868cc3-istio-envoy\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8\" (UID: \"b18c47d2-1694-4961-9638-be5cb8868cc3\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8" Nov 01 21:04:34.600689 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:34.600623 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b18c47d2-1694-4961-9638-be5cb8868cc3-istio-podinfo\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8\" (UID: \"b18c47d2-1694-4961-9638-be5cb8868cc3\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8" Nov 01 21:04:34.656432 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:34.656394 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g8gg\" (UniqueName: \"kubernetes.io/projected/b18c47d2-1694-4961-9638-be5cb8868cc3-kube-api-access-7g8gg\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8\" (UID: \"b18c47d2-1694-4961-9638-be5cb8868cc3\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8" Nov 01 21:04:34.656432 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:34.656419 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b18c47d2-1694-4961-9638-be5cb8868cc3-istio-token\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8\" (UID: \"b18c47d2-1694-4961-9638-be5cb8868cc3\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8" Nov 01 21:04:34.952268 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:34.952183 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8" Nov 01 21:04:35.384958 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:35.384923 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8"] Nov 01 21:04:35.385559 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:04:35.385536 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb18c47d2_1694_4961_9638_be5cb8868cc3.slice/crio-0e39dc9779d16f4a34d8ee3961473fd9ba83cc2bd90eb1aa7e7c39f2d20d6075 WatchSource:0}: Error finding container 0e39dc9779d16f4a34d8ee3961473fd9ba83cc2bd90eb1aa7e7c39f2d20d6075: Status 404 returned error can't find the container with id 0e39dc9779d16f4a34d8ee3961473fd9ba83cc2bd90eb1aa7e7c39f2d20d6075 Nov 01 21:04:35.389111 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:35.389072 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:04:35.389198 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:35.389148 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:04:35.389198 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:35.389188 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:04:35.425582 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:04:35.425542 4484 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c3c6fec_0841_4a0c_b761_626d8c53a1cb.slice/crio-a5d91c1d8490cb6b27129606cf5d3c39f95dedafea0ed794a5cd959283a6e608.scope\": RecentStats: unable to find data in memory cache], [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@15-35470-0.service\": RecentStats: unable to find data in memory cache]" Nov 01 21:04:35.941154 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:04:35.941116 4484 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c3c6fec_0841_4a0c_b761_626d8c53a1cb.slice/crio-a5d91c1d8490cb6b27129606cf5d3c39f95dedafea0ed794a5cd959283a6e608.scope\": RecentStats: unable to find data in memory cache], [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@15-35470-0.service\": RecentStats: unable to find data in memory cache]" Nov 01 21:04:35.941154 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:04:35.941133 4484 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@15-35470-0.service\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c3c6fec_0841_4a0c_b761_626d8c53a1cb.slice/crio-a5d91c1d8490cb6b27129606cf5d3c39f95dedafea0ed794a5cd959283a6e608.scope\": RecentStats: unable to find data in memory cache]" Nov 01 21:04:36.035658 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:36.035622 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8" event={"ID":"b18c47d2-1694-4961-9638-be5cb8868cc3","Type":"ContainerStarted","Data":"771d793b32480a9618c7aa5de6bb995d3bb681998a7fdd1002773ec584fe5bcb"} Nov 01 21:04:36.035867 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:36.035684 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8" event={"ID":"b18c47d2-1694-4961-9638-be5cb8868cc3","Type":"ContainerStarted","Data":"0e39dc9779d16f4a34d8ee3961473fd9ba83cc2bd90eb1aa7e7c39f2d20d6075"} Nov 01 21:04:36.153972 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:36.153916 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8" podStartSLOduration=2.153900712 podStartE2EDuration="2.153900712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:04:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:04:36.152228867 +0000 UTC m=+2026.929869015" watchObservedRunningTime="2026-11-01 21:04:36.153900712 +0000 UTC m=+2026.931540847" Nov 01 21:04:36.953154 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:36.953120 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8" Nov 01 21:04:36.955043 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:36.955008 4484 patch_prober.go:29] interesting pod/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body= Nov 01 21:04:36.955194 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:36.955071 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8" podUID="b18c47d2-1694-4961-9638-be5cb8868cc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" Nov 01 21:04:37.032800 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:37.032580 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:37.952861 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:37.952819 4484 patch_prober.go:29] interesting pod/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body= Nov 01 21:04:37.953076 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:37.952884 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8" podUID="b18c47d2-1694-4961-9638-be5cb8868cc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" Nov 01 21:04:38.784916 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:38.784863 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:38.952959 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:38.952916 4484 patch_prober.go:29] interesting pod/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body= Nov 01 21:04:38.953195 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:38.952976 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8" podUID="b18c47d2-1694-4961-9638-be5cb8868cc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" Nov 01 21:04:38.964107 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:38.964082 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:39.083539 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:39.083447 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6"] Nov 01 21:04:39.083723 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:39.083644 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6" podUID="5caf509e-0540-4ff6-a544-5b9e83bda042" containerName="istio-proxy" containerID="cri-o://9dd7eb3f4bdea480e82cdd337391ab8250f45239e5f09eca70544e8c27650b0e" gracePeriod=30 Nov 01 21:04:39.126562 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:39.126472 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:39.932284 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:39.932239 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-29baa8-v55fq"] Nov 01 21:04:39.937829 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:39.937804 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-29baa8-v55fq" Nov 01 21:04:39.953479 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:39.953455 4484 patch_prober.go:29] interesting pod/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body= Nov 01 21:04:39.953612 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:39.953500 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8" podUID="b18c47d2-1694-4961-9638-be5cb8868cc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" Nov 01 21:04:39.989581 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:39.989553 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-29baa8-v55fq"] Nov 01 21:04:40.004465 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:40.004425 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c1812a5-2e95-4ca5-8c91-b6734bbc348e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-29baa8-v55fq\" (UID: \"2c1812a5-2e95-4ca5-8c91-b6734bbc348e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-29baa8-v55fq" Nov 01 21:04:40.004784 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:40.004759 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8npv8\" (UniqueName: \"kubernetes.io/projected/2c1812a5-2e95-4ca5-8c91-b6734bbc348e-kube-api-access-8npv8\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-29baa8-v55fq\" (UID: \"2c1812a5-2e95-4ca5-8c91-b6734bbc348e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-29baa8-v55fq" Nov 01 21:04:40.005225 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:40.005201 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c1812a5-2e95-4ca5-8c91-b6734bbc348e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-29baa8-v55fq\" (UID: \"2c1812a5-2e95-4ca5-8c91-b6734bbc348e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-29baa8-v55fq" Nov 01 21:04:40.035921 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:40.035886 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:04:40.035921 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:40.035886 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:04:40.036158 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:40.035898 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:04:40.036158 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:40.035897 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:04:40.107170 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:40.107137 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c1812a5-2e95-4ca5-8c91-b6734bbc348e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-29baa8-v55fq\" (UID: \"2c1812a5-2e95-4ca5-8c91-b6734bbc348e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-29baa8-v55fq" Nov 01 21:04:40.107528 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:40.107506 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c1812a5-2e95-4ca5-8c91-b6734bbc348e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-29baa8-v55fq\" (UID: \"2c1812a5-2e95-4ca5-8c91-b6734bbc348e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-29baa8-v55fq" Nov 01 21:04:40.108917 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:40.108892 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8npv8\" (UniqueName: \"kubernetes.io/projected/2c1812a5-2e95-4ca5-8c91-b6734bbc348e-kube-api-access-8npv8\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-29baa8-v55fq\" (UID: \"2c1812a5-2e95-4ca5-8c91-b6734bbc348e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-29baa8-v55fq" Nov 01 21:04:40.109568 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:40.108803 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c1812a5-2e95-4ca5-8c91-b6734bbc348e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-29baa8-v55fq\" (UID: \"2c1812a5-2e95-4ca5-8c91-b6734bbc348e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-29baa8-v55fq" Nov 01 21:04:40.128862 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:40.128828 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c1812a5-2e95-4ca5-8c91-b6734bbc348e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-29baa8-v55fq\" (UID: \"2c1812a5-2e95-4ca5-8c91-b6734bbc348e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-29baa8-v55fq" Nov 01 21:04:40.167309 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:40.167272 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8npv8\" (UniqueName: \"kubernetes.io/projected/2c1812a5-2e95-4ca5-8c91-b6734bbc348e-kube-api-access-8npv8\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-29baa8-v55fq\" (UID: \"2c1812a5-2e95-4ca5-8c91-b6734bbc348e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-29baa8-v55fq" Nov 01 21:04:40.254212 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:40.254172 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-29baa8-v55fq" Nov 01 21:04:40.400567 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:40.400530 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-081c8e-4j2qn"] Nov 01 21:04:40.436688 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:40.436654 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-081c8e-4j2qn" Nov 01 21:04:40.459652 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:40.459617 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-081c8e-4j2qn"] Nov 01 21:04:40.535116 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:40.535035 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a79544f1-af43-4f27-a247-9da722d1508e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-081c8e-4j2qn\" (UID: \"a79544f1-af43-4f27-a247-9da722d1508e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-081c8e-4j2qn" Nov 01 21:04:40.535116 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:40.535090 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lvqv\" (UniqueName: \"kubernetes.io/projected/a79544f1-af43-4f27-a247-9da722d1508e-kube-api-access-8lvqv\") pod \"maas-api-revoke-keys-e2e-inf-a-081c8e-4j2qn\" (UID: \"a79544f1-af43-4f27-a247-9da722d1508e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-081c8e-4j2qn" Nov 01 21:04:40.535309 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:40.535274 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a79544f1-af43-4f27-a247-9da722d1508e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-081c8e-4j2qn\" (UID: \"a79544f1-af43-4f27-a247-9da722d1508e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-081c8e-4j2qn" Nov 01 21:04:40.637420 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:40.637390 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a79544f1-af43-4f27-a247-9da722d1508e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-081c8e-4j2qn\" (UID: \"a79544f1-af43-4f27-a247-9da722d1508e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-081c8e-4j2qn" Nov 01 21:04:40.637597 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:40.637465 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lvqv\" (UniqueName: \"kubernetes.io/projected/a79544f1-af43-4f27-a247-9da722d1508e-kube-api-access-8lvqv\") pod \"maas-api-revoke-keys-e2e-inf-a-081c8e-4j2qn\" (UID: \"a79544f1-af43-4f27-a247-9da722d1508e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-081c8e-4j2qn" Nov 01 21:04:40.637597 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:40.637575 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a79544f1-af43-4f27-a247-9da722d1508e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-081c8e-4j2qn\" (UID: \"a79544f1-af43-4f27-a247-9da722d1508e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-081c8e-4j2qn" Nov 01 21:04:40.638574 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:40.638549 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a79544f1-af43-4f27-a247-9da722d1508e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-081c8e-4j2qn\" (UID: \"a79544f1-af43-4f27-a247-9da722d1508e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-081c8e-4j2qn" Nov 01 21:04:40.641490 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:40.641353 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a79544f1-af43-4f27-a247-9da722d1508e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-081c8e-4j2qn\" (UID: \"a79544f1-af43-4f27-a247-9da722d1508e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-081c8e-4j2qn" Nov 01 21:04:40.664948 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:04:40.664923 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c1812a5_2e95_4ca5_8c91_b6734bbc348e.slice/crio-845f81aad24034cdd1f55ddd33dc1bc42b92fc6f4e10f71c243b8d0cd8cd586b WatchSource:0}: Error finding container 845f81aad24034cdd1f55ddd33dc1bc42b92fc6f4e10f71c243b8d0cd8cd586b: Status 404 returned error can't find the container with id 845f81aad24034cdd1f55ddd33dc1bc42b92fc6f4e10f71c243b8d0cd8cd586b Nov 01 21:04:40.665706 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:40.665522 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-29baa8-v55fq"] Nov 01 21:04:40.695587 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:40.695562 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lvqv\" (UniqueName: \"kubernetes.io/projected/a79544f1-af43-4f27-a247-9da722d1508e-kube-api-access-8lvqv\") pod \"maas-api-revoke-keys-e2e-inf-a-081c8e-4j2qn\" (UID: \"a79544f1-af43-4f27-a247-9da722d1508e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-081c8e-4j2qn" Nov 01 21:04:40.757243 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:40.757199 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-081c8e-4j2qn" Nov 01 21:04:40.951543 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:04:40.951506 4484 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c3c6fec_0841_4a0c_b761_626d8c53a1cb.slice/crio-a5d91c1d8490cb6b27129606cf5d3c39f95dedafea0ed794a5cd959283a6e608.scope\": RecentStats: unable to find data in memory cache], [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@15-35470-0.service\": RecentStats: unable to find data in memory cache]" Nov 01 21:04:40.952973 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:40.952947 4484 patch_prober.go:29] interesting pod/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body= Nov 01 21:04:40.953074 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:40.953048 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8" podUID="b18c47d2-1694-4961-9638-be5cb8868cc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" Nov 01 21:04:41.143969 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:41.143921 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-29baa8-v55fq" event={"ID":"2c1812a5-2e95-4ca5-8c91-b6734bbc348e","Type":"ContainerStarted","Data":"845f81aad24034cdd1f55ddd33dc1bc42b92fc6f4e10f71c243b8d0cd8cd586b"} Nov 01 21:04:41.163207 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:04:41.163178 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda79544f1_af43_4f27_a247_9da722d1508e.slice/crio-c7e9874a59af283328aac44071dc36a5402edf48257875af40fd2defa447b472 WatchSource:0}: Error finding container c7e9874a59af283328aac44071dc36a5402edf48257875af40fd2defa447b472: Status 404 returned error can't find the container with id c7e9874a59af283328aac44071dc36a5402edf48257875af40fd2defa447b472 Nov 01 21:04:41.163825 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:41.163795 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-081c8e-4j2qn"] Nov 01 21:04:41.638715 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:41.638680 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:41.953657 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:41.953623 4484 patch_prober.go:29] interesting pod/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body= Nov 01 21:04:41.954096 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:41.953680 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8" podUID="b18c47d2-1694-4961-9638-be5cb8868cc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" Nov 01 21:04:42.209268 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:42.199235 4484 generic.go:356] "Generic (PLEG): container finished" podID="2c1812a5-2e95-4ca5-8c91-b6734bbc348e" containerID="90f190f7772e0c7f1ae7ca1c67437b5a9846a7c9039bac9872c2c9ed328a0312" exitCode=0 Nov 01 21:04:42.209268 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:42.199455 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-29baa8-v55fq" event={"ID":"2c1812a5-2e95-4ca5-8c91-b6734bbc348e","Type":"ContainerDied","Data":"90f190f7772e0c7f1ae7ca1c67437b5a9846a7c9039bac9872c2c9ed328a0312"} Nov 01 21:04:42.246544 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:42.237297 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-081c8e-4j2qn" event={"ID":"a79544f1-af43-4f27-a247-9da722d1508e","Type":"ContainerStarted","Data":"58d73a0440a359ed30a4ac5c2e263fbce0a3143374d24f4eb3417f45464410b4"} Nov 01 21:04:42.246544 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:42.237533 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-081c8e-4j2qn" event={"ID":"a79544f1-af43-4f27-a247-9da722d1508e","Type":"ContainerStarted","Data":"c7e9874a59af283328aac44071dc36a5402edf48257875af40fd2defa447b472"} Nov 01 21:04:42.437465 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:42.437402 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-081c8e-4j2qn" podStartSLOduration=2.437385664 podStartE2EDuration="2.437385664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:04:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:04:42.432838287 +0000 UTC m=+2033.210478421" watchObservedRunningTime="2026-11-01 21:04:42.437385664 +0000 UTC m=+2033.215025799" Nov 01 21:04:42.957486 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:42.957450 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8" Nov 01 21:04:42.957898 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:42.957876 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8" Nov 01 21:04:42.958580 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:42.958563 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8" Nov 01 21:04:43.247426 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:43.247393 4484 generic.go:356] "Generic (PLEG): container finished" podID="a79544f1-af43-4f27-a247-9da722d1508e" containerID="58d73a0440a359ed30a4ac5c2e263fbce0a3143374d24f4eb3417f45464410b4" exitCode=0 Nov 01 21:04:43.247685 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:43.247460 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-081c8e-4j2qn" event={"ID":"a79544f1-af43-4f27-a247-9da722d1508e","Type":"ContainerDied","Data":"58d73a0440a359ed30a4ac5c2e263fbce0a3143374d24f4eb3417f45464410b4"} Nov 01 21:04:43.310413 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:43.310375 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:43.452380 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:43.452350 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-29baa8-v55fq" Nov 01 21:04:43.536828 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:43.536734 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c1812a5-2e95-4ca5-8c91-b6734bbc348e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c1812a5-2e95-4ca5-8c91-b6734bbc348e-maas-api-openshift-ca\") pod \"2c1812a5-2e95-4ca5-8c91-b6734bbc348e\" (UID: \"2c1812a5-2e95-4ca5-8c91-b6734bbc348e\") " Nov 01 21:04:43.537444 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:43.537414 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c1812a5-2e95-4ca5-8c91-b6734bbc348e-maas-api-openshift-ca" pod "2c1812a5-2e95-4ca5-8c91-b6734bbc348e" (UID: "2c1812a5-2e95-4ca5-8c91-b6734bbc348e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:04:43.538069 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:43.538037 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c1812a5-2e95-4ca5-8c91-b6734bbc348e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c1812a5-2e95-4ca5-8c91-b6734bbc348e-maas-api-xks-ca\") pod \"2c1812a5-2e95-4ca5-8c91-b6734bbc348e\" (UID: \"2c1812a5-2e95-4ca5-8c91-b6734bbc348e\") " Nov 01 21:04:43.538219 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:43.538125 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c1812a5-2e95-4ca5-8c91-b6734bbc348e-kube-api-access-8npv8\" (UniqueName: \"kubernetes.io/projected/2c1812a5-2e95-4ca5-8c91-b6734bbc348e-kube-api-access-8npv8\") pod \"2c1812a5-2e95-4ca5-8c91-b6734bbc348e\" (UID: \"2c1812a5-2e95-4ca5-8c91-b6734bbc348e\") " Nov 01 21:04:43.538776 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:43.538749 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c1812a5-2e95-4ca5-8c91-b6734bbc348e-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:43.542955 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:43.542926 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c1812a5-2e95-4ca5-8c91-b6734bbc348e-maas-api-xks-ca" pod "2c1812a5-2e95-4ca5-8c91-b6734bbc348e" (UID: "2c1812a5-2e95-4ca5-8c91-b6734bbc348e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:04:43.544448 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:43.544413 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c1812a5-2e95-4ca5-8c91-b6734bbc348e-kube-api-access-8npv8" pod "2c1812a5-2e95-4ca5-8c91-b6734bbc348e" (UID: "2c1812a5-2e95-4ca5-8c91-b6734bbc348e"). InnerVolumeSpecName "kube-api-access-8npv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:04:43.546964 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:43.546930 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:43.691948 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:43.691903 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c1812a5-2e95-4ca5-8c91-b6734bbc348e-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:43.691948 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:43.691949 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8npv8\" (UniqueName: \"kubernetes.io/projected/2c1812a5-2e95-4ca5-8c91-b6734bbc348e-kube-api-access-8npv8\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:43.706443 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:43.706398 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:43.812559 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:43.812474 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a9b439-d7ph6"] Nov 01 21:04:43.816431 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:43.816408 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a9b439-d7ph6"] Nov 01 21:04:44.174126 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.174093 4484 patch_prober.go:29] interesting pod/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Nov 01 21:04:44.174511 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.174144 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6" podUID="5caf509e-0540-4ff6-a544-5b9e83bda042" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Nov 01 21:04:44.200563 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.200533 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-22ed7b-tkt8f"] Nov 01 21:04:44.201379 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.201358 4484 cpu_manager.go:414] "containerMap: removing container" podUID="2c1812a5-2e95-4ca5-8c91-b6734bbc348e" containerName="revoke-keys" Nov 01 21:04:44.201379 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.201378 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c1812a5-2e95-4ca5-8c91-b6734bbc348e" containerName="revoke-keys" Nov 01 21:04:44.201633 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.201618 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c1812a5-2e95-4ca5-8c91-b6734bbc348e" containerName="revoke-keys" Nov 01 21:04:44.205411 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.205387 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-22ed7b-tkt8f" Nov 01 21:04:44.231665 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.231630 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:44.260407 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.260378 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-29baa8-v55fq" Nov 01 21:04:44.260629 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.260378 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-29baa8-v55fq" event={"ID":"2c1812a5-2e95-4ca5-8c91-b6734bbc348e","Type":"ContainerDied","Data":"845f81aad24034cdd1f55ddd33dc1bc42b92fc6f4e10f71c243b8d0cd8cd586b"} Nov 01 21:04:44.260629 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.260573 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="845f81aad24034cdd1f55ddd33dc1bc42b92fc6f4e10f71c243b8d0cd8cd586b" Nov 01 21:04:44.261286 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.261261 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-22ed7b-tkt8f"] Nov 01 21:04:44.264323 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.264291 4484 generic.go:356] "Generic (PLEG): container finished" podID="5caf509e-0540-4ff6-a544-5b9e83bda042" containerID="9dd7eb3f4bdea480e82cdd337391ab8250f45239e5f09eca70544e8c27650b0e" exitCode=0 Nov 01 21:04:44.264442 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.264363 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6" event={"ID":"5caf509e-0540-4ff6-a544-5b9e83bda042","Type":"ContainerDied","Data":"9dd7eb3f4bdea480e82cdd337391ab8250f45239e5f09eca70544e8c27650b0e"} Nov 01 21:04:44.264442 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.264428 4484 scope.go:122] "RemoveContainer" containerID="b877616fcd50c65da91923af62eadbf103481a08c0b2aa30cab537aaf0efc123" Nov 01 21:04:44.301045 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.300980 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa2af658-15d9-48a7-8f65-36f359fee14c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-22ed7b-tkt8f\" (UID: \"fa2af658-15d9-48a7-8f65-36f359fee14c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-22ed7b-tkt8f" Nov 01 21:04:44.301195 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.301175 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa2af658-15d9-48a7-8f65-36f359fee14c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-22ed7b-tkt8f\" (UID: \"fa2af658-15d9-48a7-8f65-36f359fee14c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-22ed7b-tkt8f" Nov 01 21:04:44.301271 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.301213 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4s6v\" (UniqueName: \"kubernetes.io/projected/fa2af658-15d9-48a7-8f65-36f359fee14c-kube-api-access-v4s6v\") pod \"maas-api-revoke-keys-e2e-api-a-22ed7b-tkt8f\" (UID: \"fa2af658-15d9-48a7-8f65-36f359fee14c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-22ed7b-tkt8f" Nov 01 21:04:44.403437 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.403350 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa2af658-15d9-48a7-8f65-36f359fee14c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-22ed7b-tkt8f\" (UID: \"fa2af658-15d9-48a7-8f65-36f359fee14c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-22ed7b-tkt8f" Nov 01 21:04:44.403613 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.403533 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa2af658-15d9-48a7-8f65-36f359fee14c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-22ed7b-tkt8f\" (UID: \"fa2af658-15d9-48a7-8f65-36f359fee14c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-22ed7b-tkt8f" Nov 01 21:04:44.403613 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.403565 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4s6v\" (UniqueName: \"kubernetes.io/projected/fa2af658-15d9-48a7-8f65-36f359fee14c-kube-api-access-v4s6v\") pod \"maas-api-revoke-keys-e2e-api-a-22ed7b-tkt8f\" (UID: \"fa2af658-15d9-48a7-8f65-36f359fee14c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-22ed7b-tkt8f" Nov 01 21:04:44.404069 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.404038 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa2af658-15d9-48a7-8f65-36f359fee14c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-22ed7b-tkt8f\" (UID: \"fa2af658-15d9-48a7-8f65-36f359fee14c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-22ed7b-tkt8f" Nov 01 21:04:44.407051 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.407023 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa2af658-15d9-48a7-8f65-36f359fee14c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-22ed7b-tkt8f\" (UID: \"fa2af658-15d9-48a7-8f65-36f359fee14c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-22ed7b-tkt8f" Nov 01 21:04:44.476214 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.476190 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6" Nov 01 21:04:44.477912 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.477886 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4s6v\" (UniqueName: \"kubernetes.io/projected/fa2af658-15d9-48a7-8f65-36f359fee14c-kube-api-access-v4s6v\") pod \"maas-api-revoke-keys-e2e-api-a-22ed7b-tkt8f\" (UID: \"fa2af658-15d9-48a7-8f65-36f359fee14c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-22ed7b-tkt8f" Nov 01 21:04:44.479390 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.479372 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-081c8e-4j2qn" Nov 01 21:04:44.522760 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.522731 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-22ed7b-tkt8f" Nov 01 21:04:44.609858 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.607507 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5caf509e-0540-4ff6-a544-5b9e83bda042-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5caf509e-0540-4ff6-a544-5b9e83bda042-istiod-ca-cert\") pod \"5caf509e-0540-4ff6-a544-5b9e83bda042\" (UID: \"5caf509e-0540-4ff6-a544-5b9e83bda042\") " Nov 01 21:04:44.610505 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.609811 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5caf509e-0540-4ff6-a544-5b9e83bda042-istiod-ca-cert" pod "5caf509e-0540-4ff6-a544-5b9e83bda042" (UID: "5caf509e-0540-4ff6-a544-5b9e83bda042"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:04:44.621110 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.612263 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5caf509e-0540-4ff6-a544-5b9e83bda042-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5caf509e-0540-4ff6-a544-5b9e83bda042-credential-socket\") pod \"5caf509e-0540-4ff6-a544-5b9e83bda042\" (UID: \"5caf509e-0540-4ff6-a544-5b9e83bda042\") " Nov 01 21:04:44.621110 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.612359 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5caf509e-0540-4ff6-a544-5b9e83bda042-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5caf509e-0540-4ff6-a544-5b9e83bda042-istio-envoy\") pod \"5caf509e-0540-4ff6-a544-5b9e83bda042\" (UID: \"5caf509e-0540-4ff6-a544-5b9e83bda042\") " Nov 01 21:04:44.621110 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.612463 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a79544f1-af43-4f27-a247-9da722d1508e-kube-api-access-8lvqv\" (UniqueName: \"kubernetes.io/projected/a79544f1-af43-4f27-a247-9da722d1508e-kube-api-access-8lvqv\") pod \"a79544f1-af43-4f27-a247-9da722d1508e\" (UID: \"a79544f1-af43-4f27-a247-9da722d1508e\") " Nov 01 21:04:44.621110 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.612528 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5caf509e-0540-4ff6-a544-5b9e83bda042-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5caf509e-0540-4ff6-a544-5b9e83bda042-istio-data\") pod \"5caf509e-0540-4ff6-a544-5b9e83bda042\" (UID: \"5caf509e-0540-4ff6-a544-5b9e83bda042\") " Nov 01 21:04:44.621110 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.612569 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5caf509e-0540-4ff6-a544-5b9e83bda042-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5caf509e-0540-4ff6-a544-5b9e83bda042-workload-certs\") pod \"5caf509e-0540-4ff6-a544-5b9e83bda042\" (UID: \"5caf509e-0540-4ff6-a544-5b9e83bda042\") " Nov 01 21:04:44.621110 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.612616 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5caf509e-0540-4ff6-a544-5b9e83bda042-kube-api-access-lkwp7\" (UniqueName: \"kubernetes.io/projected/5caf509e-0540-4ff6-a544-5b9e83bda042-kube-api-access-lkwp7\") pod \"5caf509e-0540-4ff6-a544-5b9e83bda042\" (UID: \"5caf509e-0540-4ff6-a544-5b9e83bda042\") " Nov 01 21:04:44.621110 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.612642 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5caf509e-0540-4ff6-a544-5b9e83bda042-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5caf509e-0540-4ff6-a544-5b9e83bda042-istio-podinfo\") pod \"5caf509e-0540-4ff6-a544-5b9e83bda042\" (UID: \"5caf509e-0540-4ff6-a544-5b9e83bda042\") " Nov 01 21:04:44.621110 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.612675 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5caf509e-0540-4ff6-a544-5b9e83bda042-istio-token\" (UniqueName: \"kubernetes.io/projected/5caf509e-0540-4ff6-a544-5b9e83bda042-istio-token\") pod \"5caf509e-0540-4ff6-a544-5b9e83bda042\" (UID: \"5caf509e-0540-4ff6-a544-5b9e83bda042\") " Nov 01 21:04:44.621110 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.612722 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a79544f1-af43-4f27-a247-9da722d1508e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a79544f1-af43-4f27-a247-9da722d1508e-maas-api-xks-ca\") pod \"a79544f1-af43-4f27-a247-9da722d1508e\" (UID: \"a79544f1-af43-4f27-a247-9da722d1508e\") " Nov 01 21:04:44.621110 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.612797 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5caf509e-0540-4ff6-a544-5b9e83bda042-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5caf509e-0540-4ff6-a544-5b9e83bda042-workload-socket\") pod \"5caf509e-0540-4ff6-a544-5b9e83bda042\" (UID: \"5caf509e-0540-4ff6-a544-5b9e83bda042\") " Nov 01 21:04:44.621110 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.612841 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a79544f1-af43-4f27-a247-9da722d1508e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a79544f1-af43-4f27-a247-9da722d1508e-maas-api-openshift-ca\") pod \"a79544f1-af43-4f27-a247-9da722d1508e\" (UID: \"a79544f1-af43-4f27-a247-9da722d1508e\") " Nov 01 21:04:44.621110 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.613585 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5caf509e-0540-4ff6-a544-5b9e83bda042-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:44.621110 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.613928 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a79544f1-af43-4f27-a247-9da722d1508e-maas-api-openshift-ca" pod "a79544f1-af43-4f27-a247-9da722d1508e" (UID: "a79544f1-af43-4f27-a247-9da722d1508e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:04:44.621110 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.614204 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5caf509e-0540-4ff6-a544-5b9e83bda042-credential-socket" pod "5caf509e-0540-4ff6-a544-5b9e83bda042" (UID: "5caf509e-0540-4ff6-a544-5b9e83bda042"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:04:44.621110 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.616550 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:44.621110 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.616824 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5caf509e-0540-4ff6-a544-5b9e83bda042-workload-socket" pod "5caf509e-0540-4ff6-a544-5b9e83bda042" (UID: "5caf509e-0540-4ff6-a544-5b9e83bda042"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:04:44.621110 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.617190 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5caf509e-0540-4ff6-a544-5b9e83bda042-istio-data" pod "5caf509e-0540-4ff6-a544-5b9e83bda042" (UID: "5caf509e-0540-4ff6-a544-5b9e83bda042"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:04:44.621110 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.617432 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5caf509e-0540-4ff6-a544-5b9e83bda042-workload-certs" pod "5caf509e-0540-4ff6-a544-5b9e83bda042" (UID: "5caf509e-0540-4ff6-a544-5b9e83bda042"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:04:44.622982 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.622935 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a79544f1-af43-4f27-a247-9da722d1508e-kube-api-access-8lvqv" pod "a79544f1-af43-4f27-a247-9da722d1508e" (UID: "a79544f1-af43-4f27-a247-9da722d1508e"). InnerVolumeSpecName "kube-api-access-8lvqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:04:44.623134 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.623102 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a79544f1-af43-4f27-a247-9da722d1508e-maas-api-xks-ca" pod "a79544f1-af43-4f27-a247-9da722d1508e" (UID: "a79544f1-af43-4f27-a247-9da722d1508e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:04:44.624710 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.624690 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5caf509e-0540-4ff6-a544-5b9e83bda042-istio-envoy" pod "5caf509e-0540-4ff6-a544-5b9e83bda042" (UID: "5caf509e-0540-4ff6-a544-5b9e83bda042"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:04:44.624812 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.624717 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5caf509e-0540-4ff6-a544-5b9e83bda042-kube-api-access-lkwp7" pod "5caf509e-0540-4ff6-a544-5b9e83bda042" (UID: "5caf509e-0540-4ff6-a544-5b9e83bda042"). InnerVolumeSpecName "kube-api-access-lkwp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:04:44.625269 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.625237 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5caf509e-0540-4ff6-a544-5b9e83bda042-istio-token" pod "5caf509e-0540-4ff6-a544-5b9e83bda042" (UID: "5caf509e-0540-4ff6-a544-5b9e83bda042"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:04:44.627287 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.627267 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5caf509e-0540-4ff6-a544-5b9e83bda042-istio-podinfo" pod "5caf509e-0540-4ff6-a544-5b9e83bda042" (UID: "5caf509e-0540-4ff6-a544-5b9e83bda042"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:04:44.715530 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.715446 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5caf509e-0540-4ff6-a544-5b9e83bda042-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:44.715530 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.715477 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5caf509e-0540-4ff6-a544-5b9e83bda042-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:44.715530 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.715487 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lvqv\" (UniqueName: \"kubernetes.io/projected/a79544f1-af43-4f27-a247-9da722d1508e-kube-api-access-8lvqv\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:44.715530 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.715497 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5caf509e-0540-4ff6-a544-5b9e83bda042-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:44.715530 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.715506 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5caf509e-0540-4ff6-a544-5b9e83bda042-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:44.715530 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.715514 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkwp7\" (UniqueName: \"kubernetes.io/projected/5caf509e-0540-4ff6-a544-5b9e83bda042-kube-api-access-lkwp7\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:44.715530 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.715525 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5caf509e-0540-4ff6-a544-5b9e83bda042-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:44.715530 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.715533 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5caf509e-0540-4ff6-a544-5b9e83bda042-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:44.715905 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.715541 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a79544f1-af43-4f27-a247-9da722d1508e-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:44.715905 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.715550 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5caf509e-0540-4ff6-a544-5b9e83bda042-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:44.715905 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:44.715559 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a79544f1-af43-4f27-a247-9da722d1508e-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:45.004215 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:04:45.004183 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa2af658_15d9_48a7_8f65_36f359fee14c.slice/crio-b34d1f6b9989ee6a28be9a864f2751525e94005156d9175b41643e06da30b015 WatchSource:0}: Error finding container b34d1f6b9989ee6a28be9a864f2751525e94005156d9175b41643e06da30b015: Status 404 returned error can't find the container with id b34d1f6b9989ee6a28be9a864f2751525e94005156d9175b41643e06da30b015 Nov 01 21:04:45.004872 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:45.004846 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-22ed7b-tkt8f"] Nov 01 21:04:45.273455 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:45.273370 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-081c8e-4j2qn" Nov 01 21:04:45.273455 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:45.273383 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-081c8e-4j2qn" event={"ID":"a79544f1-af43-4f27-a247-9da722d1508e","Type":"ContainerDied","Data":"c7e9874a59af283328aac44071dc36a5402edf48257875af40fd2defa447b472"} Nov 01 21:04:45.273455 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:45.273445 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7e9874a59af283328aac44071dc36a5402edf48257875af40fd2defa447b472" Nov 01 21:04:45.275171 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:45.275143 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6" Nov 01 21:04:45.275320 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:45.275186 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6" event={"ID":"5caf509e-0540-4ff6-a544-5b9e83bda042","Type":"ContainerDied","Data":"40c3d24202032567d872e6549d49f8072c003776a2a2ad8a89fb46ae0a5f485d"} Nov 01 21:04:45.275320 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:45.275236 4484 scope.go:122] "RemoveContainer" containerID="9dd7eb3f4bdea480e82cdd337391ab8250f45239e5f09eca70544e8c27650b0e" Nov 01 21:04:45.277718 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:45.277693 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-22ed7b-tkt8f" event={"ID":"fa2af658-15d9-48a7-8f65-36f359fee14c","Type":"ContainerStarted","Data":"b34d1f6b9989ee6a28be9a864f2751525e94005156d9175b41643e06da30b015"} Nov 01 21:04:45.373833 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:45.373801 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87296964-4ebc-4bd6-9473-527c86358226" path="/var/lib/kubelet/pods/87296964-4ebc-4bd6-9473-527c86358226/volumes" Nov 01 21:04:45.443479 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:04:45.443400 4484 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@15-35470-0.service\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c3c6fec_0841_4a0c_b761_626d8c53a1cb.slice/crio-a5d91c1d8490cb6b27129606cf5d3c39f95dedafea0ed794a5cd959283a6e608.scope\": RecentStats: unable to find data in memory cache]" Nov 01 21:04:45.448461 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:45.448433 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6"] Nov 01 21:04:45.452295 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:45.452275 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-de8145-openshift-default-57f9f8965-mpzk6"] Nov 01 21:04:45.767562 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:04:45.767521 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:04:46.288121 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:46.288090 4484 generic.go:356] "Generic (PLEG): container finished" podID="fa2af658-15d9-48a7-8f65-36f359fee14c" containerID="d5c96cc4385de59c41be7420f068f3d4b5de59e9b76a9163c079eac500b05558" exitCode=0 Nov 01 21:04:46.288533 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:46.288174 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-22ed7b-tkt8f" event={"ID":"fa2af658-15d9-48a7-8f65-36f359fee14c","Type":"ContainerDied","Data":"d5c96cc4385de59c41be7420f068f3d4b5de59e9b76a9163c079eac500b05558"} Nov 01 21:04:47.380681 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:47.380644 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5caf509e-0540-4ff6-a544-5b9e83bda042" path="/var/lib/kubelet/pods/5caf509e-0540-4ff6-a544-5b9e83bda042/volumes" Nov 01 21:04:47.471081 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:47.471057 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-22ed7b-tkt8f" Nov 01 21:04:47.598838 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:47.598761 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa2af658-15d9-48a7-8f65-36f359fee14c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa2af658-15d9-48a7-8f65-36f359fee14c-maas-api-xks-ca\") pod \"fa2af658-15d9-48a7-8f65-36f359fee14c\" (UID: \"fa2af658-15d9-48a7-8f65-36f359fee14c\") " Nov 01 21:04:47.599036 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:47.598859 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa2af658-15d9-48a7-8f65-36f359fee14c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa2af658-15d9-48a7-8f65-36f359fee14c-maas-api-openshift-ca\") pod \"fa2af658-15d9-48a7-8f65-36f359fee14c\" (UID: \"fa2af658-15d9-48a7-8f65-36f359fee14c\") " Nov 01 21:04:47.599036 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:47.598914 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa2af658-15d9-48a7-8f65-36f359fee14c-kube-api-access-v4s6v\" (UniqueName: \"kubernetes.io/projected/fa2af658-15d9-48a7-8f65-36f359fee14c-kube-api-access-v4s6v\") pod \"fa2af658-15d9-48a7-8f65-36f359fee14c\" (UID: \"fa2af658-15d9-48a7-8f65-36f359fee14c\") " Nov 01 21:04:47.599708 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:47.599674 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa2af658-15d9-48a7-8f65-36f359fee14c-maas-api-openshift-ca" pod "fa2af658-15d9-48a7-8f65-36f359fee14c" (UID: "fa2af658-15d9-48a7-8f65-36f359fee14c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:04:47.601614 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:47.601588 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa2af658-15d9-48a7-8f65-36f359fee14c-maas-api-xks-ca" pod "fa2af658-15d9-48a7-8f65-36f359fee14c" (UID: "fa2af658-15d9-48a7-8f65-36f359fee14c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:04:47.602010 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:47.601971 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa2af658-15d9-48a7-8f65-36f359fee14c-kube-api-access-v4s6v" pod "fa2af658-15d9-48a7-8f65-36f359fee14c" (UID: "fa2af658-15d9-48a7-8f65-36f359fee14c"). InnerVolumeSpecName "kube-api-access-v4s6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:04:47.700981 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:47.700948 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa2af658-15d9-48a7-8f65-36f359fee14c-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:47.700981 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:47.700978 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4s6v\" (UniqueName: \"kubernetes.io/projected/fa2af658-15d9-48a7-8f65-36f359fee14c-kube-api-access-v4s6v\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:47.700981 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:47.700990 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa2af658-15d9-48a7-8f65-36f359fee14c-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:48.305240 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:48.305203 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-22ed7b-tkt8f" Nov 01 21:04:48.305448 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:48.305240 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-22ed7b-tkt8f" event={"ID":"fa2af658-15d9-48a7-8f65-36f359fee14c","Type":"ContainerDied","Data":"b34d1f6b9989ee6a28be9a864f2751525e94005156d9175b41643e06da30b015"} Nov 01 21:04:48.305448 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:48.305327 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b34d1f6b9989ee6a28be9a864f2751525e94005156d9175b41643e06da30b015" Nov 01 21:04:48.692468 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:48.692381 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:49.449135 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:49.449105 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98"] Nov 01 21:04:49.449852 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:49.449838 4484 cpu_manager.go:414] "containerMap: removing container" podUID="a79544f1-af43-4f27-a247-9da722d1508e" containerName="revoke-keys" Nov 01 21:04:49.449904 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:49.449853 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a79544f1-af43-4f27-a247-9da722d1508e" containerName="revoke-keys" Nov 01 21:04:49.449904 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:49.449862 4484 cpu_manager.go:414] "containerMap: removing container" podUID="5caf509e-0540-4ff6-a544-5b9e83bda042" containerName="istio-proxy" Nov 01 21:04:49.449904 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:49.449867 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5caf509e-0540-4ff6-a544-5b9e83bda042" containerName="istio-proxy" Nov 01 21:04:49.450045 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:49.449922 4484 cpu_manager.go:414] "containerMap: removing container" podUID="fa2af658-15d9-48a7-8f65-36f359fee14c" containerName="revoke-keys" Nov 01 21:04:49.450045 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:49.449928 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa2af658-15d9-48a7-8f65-36f359fee14c" containerName="revoke-keys" Nov 01 21:04:49.450130 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:49.450054 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa2af658-15d9-48a7-8f65-36f359fee14c" containerName="revoke-keys" Nov 01 21:04:49.450130 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:49.450063 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="a79544f1-af43-4f27-a247-9da722d1508e" containerName="revoke-keys" Nov 01 21:04:49.450130 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:49.450070 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="5caf509e-0540-4ff6-a544-5b9e83bda042" containerName="istio-proxy" Nov 01 21:04:49.450130 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:49.450085 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="5caf509e-0540-4ff6-a544-5b9e83bda042" containerName="istio-proxy" Nov 01 21:04:49.454081 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:49.454060 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98" Nov 01 21:04:49.539052 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:49.539018 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-istio-envoy\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98\" (UID: \"7403ccd9-e6f4-4ceb-9d97-22e08152f92e\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98" Nov 01 21:04:49.539259 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:49.539088 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-credential-socket\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98\" (UID: \"7403ccd9-e6f4-4ceb-9d97-22e08152f92e\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98" Nov 01 21:04:49.539347 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:49.539257 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-istio-podinfo\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98\" (UID: \"7403ccd9-e6f4-4ceb-9d97-22e08152f92e\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98" Nov 01 21:04:49.539347 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:49.539296 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-istio-token\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98\" (UID: \"7403ccd9-e6f4-4ceb-9d97-22e08152f92e\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98" Nov 01 21:04:49.539479 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:49.539436 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-workload-socket\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98\" (UID: \"7403ccd9-e6f4-4ceb-9d97-22e08152f92e\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98" Nov 01 21:04:49.539526 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:49.539504 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8c7t\" (UniqueName: \"kubernetes.io/projected/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-kube-api-access-c8c7t\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98\" (UID: \"7403ccd9-e6f4-4ceb-9d97-22e08152f92e\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98" Nov 01 21:04:49.539569 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:49.539536 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-istiod-ca-cert\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98\" (UID: \"7403ccd9-e6f4-4ceb-9d97-22e08152f92e\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98" Nov 01 21:04:49.539569 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:49.539558 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-istio-data\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98\" (UID: \"7403ccd9-e6f4-4ceb-9d97-22e08152f92e\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98" Nov 01 21:04:49.539692 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:49.539672 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-workload-certs\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98\" (UID: \"7403ccd9-e6f4-4ceb-9d97-22e08152f92e\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98" Nov 01 21:04:49.561639 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:49.555916 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98"] Nov 01 21:04:49.640810 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:49.640766 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-istio-data\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98\" (UID: \"7403ccd9-e6f4-4ceb-9d97-22e08152f92e\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98" Nov 01 21:04:49.640987 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:49.640861 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-workload-certs\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98\" (UID: \"7403ccd9-e6f4-4ceb-9d97-22e08152f92e\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98" Nov 01 21:04:49.640987 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:49.640909 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-istio-envoy\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98\" (UID: \"7403ccd9-e6f4-4ceb-9d97-22e08152f92e\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98" Nov 01 21:04:49.641134 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:49.640985 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-credential-socket\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98\" (UID: \"7403ccd9-e6f4-4ceb-9d97-22e08152f92e\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98" Nov 01 21:04:49.641191 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:49.641138 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-istio-podinfo\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98\" (UID: \"7403ccd9-e6f4-4ceb-9d97-22e08152f92e\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98" Nov 01 21:04:49.641191 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:49.641166 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-istio-token\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98\" (UID: \"7403ccd9-e6f4-4ceb-9d97-22e08152f92e\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98" Nov 01 21:04:49.641296 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:49.641266 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-istio-data\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98\" (UID: \"7403ccd9-e6f4-4ceb-9d97-22e08152f92e\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98" Nov 01 21:04:49.641296 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:49.641277 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-workload-socket\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98\" (UID: \"7403ccd9-e6f4-4ceb-9d97-22e08152f92e\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98" Nov 01 21:04:49.641405 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:49.641317 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-workload-certs\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98\" (UID: \"7403ccd9-e6f4-4ceb-9d97-22e08152f92e\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98" Nov 01 21:04:49.641405 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:49.641354 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8c7t\" (UniqueName: \"kubernetes.io/projected/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-kube-api-access-c8c7t\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98\" (UID: \"7403ccd9-e6f4-4ceb-9d97-22e08152f92e\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98" Nov 01 21:04:49.641405 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:49.641394 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-istiod-ca-cert\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98\" (UID: \"7403ccd9-e6f4-4ceb-9d97-22e08152f92e\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98" Nov 01 21:04:49.641611 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:49.641472 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-credential-socket\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98\" (UID: \"7403ccd9-e6f4-4ceb-9d97-22e08152f92e\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98" Nov 01 21:04:49.641611 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:49.641526 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-workload-socket\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98\" (UID: \"7403ccd9-e6f4-4ceb-9d97-22e08152f92e\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98" Nov 01 21:04:49.641989 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:49.641962 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-istiod-ca-cert\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98\" (UID: \"7403ccd9-e6f4-4ceb-9d97-22e08152f92e\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98" Nov 01 21:04:49.643738 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:49.643711 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-istio-envoy\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98\" (UID: \"7403ccd9-e6f4-4ceb-9d97-22e08152f92e\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98" Nov 01 21:04:49.643933 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:49.643907 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-istio-podinfo\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98\" (UID: \"7403ccd9-e6f4-4ceb-9d97-22e08152f92e\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98" Nov 01 21:04:49.700641 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:49.700565 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-istio-token\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98\" (UID: \"7403ccd9-e6f4-4ceb-9d97-22e08152f92e\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98" Nov 01 21:04:49.703459 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:49.703433 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8c7t\" (UniqueName: \"kubernetes.io/projected/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-kube-api-access-c8c7t\") pod \"e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98\" (UID: \"7403ccd9-e6f4-4ceb-9d97-22e08152f92e\") " pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98" Nov 01 21:04:49.770684 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:49.770645 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98" Nov 01 21:04:50.207189 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:50.207155 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98"] Nov 01 21:04:50.207598 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:04:50.207567 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7403ccd9_e6f4_4ceb_9d97_22e08152f92e.slice/crio-d3c205ed8af7603c1f1264e9ad7b95defc80e4ff3a282c1449c226d1ecf807ff WatchSource:0}: Error finding container d3c205ed8af7603c1f1264e9ad7b95defc80e4ff3a282c1449c226d1ecf807ff: Status 404 returned error can't find the container with id d3c205ed8af7603c1f1264e9ad7b95defc80e4ff3a282c1449c226d1ecf807ff Nov 01 21:04:50.211237 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:50.211205 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:04:50.211317 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:50.211274 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:04:50.211317 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:50.211300 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:04:50.306647 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:50.306609 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:50.331014 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:50.329479 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98" event={"ID":"7403ccd9-e6f4-4ceb-9d97-22e08152f92e","Type":"ContainerStarted","Data":"e6d50989d5066e8d0f1d39a5579b51762c212c9e0d638f2d4f1316a9a8105208"} Nov 01 21:04:50.331014 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:50.329549 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98" event={"ID":"7403ccd9-e6f4-4ceb-9d97-22e08152f92e","Type":"ContainerStarted","Data":"d3c205ed8af7603c1f1264e9ad7b95defc80e4ff3a282c1449c226d1ecf807ff"} Nov 01 21:04:50.452538 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:50.452481 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98" podStartSLOduration=1.452462708 podStartE2EDuration="1.452462708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:04:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:04:50.452023436 +0000 UTC m=+2041.229663563" watchObservedRunningTime="2026-11-01 21:04:50.452462708 +0000 UTC m=+2041.230102899" Nov 01 21:04:50.515217 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:50.515175 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:50.560119 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:50.560080 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m"] Nov 01 21:04:50.560402 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:50.560371 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m" podUID="7ecec0b0-7e22-4803-891c-0a75173445ad" containerName="istio-proxy" containerID="cri-o://2f7560be7d7ec4d112867c55ef155c749075b51ed446557b8d4fba8334ff3e6d" gracePeriod=30 Nov 01 21:04:50.711019 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:50.693404 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:50.771073 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:50.770875 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98" Nov 01 21:04:51.468368 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:51.468332 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-dab736-545mp"] Nov 01 21:04:51.469162 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:51.469146 4484 cpu_manager.go:414] "containerMap: removing container" podUID="5caf509e-0540-4ff6-a544-5b9e83bda042" containerName="istio-proxy" Nov 01 21:04:51.469219 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:51.469162 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5caf509e-0540-4ff6-a544-5b9e83bda042" containerName="istio-proxy" Nov 01 21:04:51.473198 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:51.473182 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-dab736-545mp" Nov 01 21:04:51.529357 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:51.529311 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:04:51.580332 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:51.580295 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:04:51.580540 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:51.580461 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:04:51.580615 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:51.580578 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:04:51.587106 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:51.587082 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-dab736-545mp"] Nov 01 21:04:51.589414 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:51.589393 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92642861-003f-49a3-9df0-7a2913316106-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-dab736-545mp\" (UID: \"92642861-003f-49a3-9df0-7a2913316106\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-dab736-545mp" Nov 01 21:04:51.589532 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:51.589519 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nlvp\" (UniqueName: \"kubernetes.io/projected/92642861-003f-49a3-9df0-7a2913316106-kube-api-access-6nlvp\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-dab736-545mp\" (UID: \"92642861-003f-49a3-9df0-7a2913316106\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-dab736-545mp" Nov 01 21:04:51.589627 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:51.589606 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92642861-003f-49a3-9df0-7a2913316106-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-dab736-545mp\" (UID: \"92642861-003f-49a3-9df0-7a2913316106\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-dab736-545mp" Nov 01 21:04:51.690898 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:51.690864 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nlvp\" (UniqueName: \"kubernetes.io/projected/92642861-003f-49a3-9df0-7a2913316106-kube-api-access-6nlvp\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-dab736-545mp\" (UID: \"92642861-003f-49a3-9df0-7a2913316106\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-dab736-545mp" Nov 01 21:04:51.691100 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:51.690934 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92642861-003f-49a3-9df0-7a2913316106-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-dab736-545mp\" (UID: \"92642861-003f-49a3-9df0-7a2913316106\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-dab736-545mp" Nov 01 21:04:51.691215 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:51.691192 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92642861-003f-49a3-9df0-7a2913316106-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-dab736-545mp\" (UID: \"92642861-003f-49a3-9df0-7a2913316106\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-dab736-545mp" Nov 01 21:04:51.691713 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:51.691690 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92642861-003f-49a3-9df0-7a2913316106-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-dab736-545mp\" (UID: \"92642861-003f-49a3-9df0-7a2913316106\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-dab736-545mp" Nov 01 21:04:51.693513 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:51.693489 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92642861-003f-49a3-9df0-7a2913316106-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-dab736-545mp\" (UID: \"92642861-003f-49a3-9df0-7a2913316106\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-dab736-545mp" Nov 01 21:04:51.748759 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:51.748721 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nlvp\" (UniqueName: \"kubernetes.io/projected/92642861-003f-49a3-9df0-7a2913316106-kube-api-access-6nlvp\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-dab736-545mp\" (UID: \"92642861-003f-49a3-9df0-7a2913316106\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-dab736-545mp" Nov 01 21:04:51.772768 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:51.772736 4484 patch_prober.go:29] interesting pod/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Nov 01 21:04:51.772924 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:51.772788 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98" podUID="7403ccd9-e6f4-4ceb-9d97-22e08152f92e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Nov 01 21:04:51.788884 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:51.788857 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-dab736-545mp" Nov 01 21:04:52.197962 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:52.197926 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-dab736-545mp"] Nov 01 21:04:52.198677 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:04:52.198649 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92642861_003f_49a3_9df0_7a2913316106.slice/crio-0d89d79c04f9fa65b78ca71b76cee0d10fef5f715eee552e475ccf98f4d7578b WatchSource:0}: Error finding container 0d89d79c04f9fa65b78ca71b76cee0d10fef5f715eee552e475ccf98f4d7578b: Status 404 returned error can't find the container with id 0d89d79c04f9fa65b78ca71b76cee0d10fef5f715eee552e475ccf98f4d7578b Nov 01 21:04:52.349389 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:52.349352 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-dab736-545mp" event={"ID":"92642861-003f-49a3-9df0-7a2913316106","Type":"ContainerStarted","Data":"0d89d79c04f9fa65b78ca71b76cee0d10fef5f715eee552e475ccf98f4d7578b"} Nov 01 21:04:52.771605 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:52.771574 4484 patch_prober.go:29] interesting pod/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Nov 01 21:04:52.771969 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:52.771623 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98" podUID="7403ccd9-e6f4-4ceb-9d97-22e08152f92e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Nov 01 21:04:53.063701 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:53.063655 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:53.285583 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:53.285546 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:53.359277 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:53.359189 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-dab736-545mp" event={"ID":"92642861-003f-49a3-9df0-7a2913316106","Type":"ContainerStarted","Data":"134b52c382ff6c88f8c8b84568cb9cef76e0c4e878e6c7e02e62681a00c2422d"} Nov 01 21:04:53.455751 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:53.455711 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:53.772653 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:53.772615 4484 patch_prober.go:29] interesting pod/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Nov 01 21:04:53.773123 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:53.772684 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98" podUID="7403ccd9-e6f4-4ceb-9d97-22e08152f92e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Nov 01 21:04:54.368288 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:54.368256 4484 generic.go:356] "Generic (PLEG): container finished" podID="92642861-003f-49a3-9df0-7a2913316106" containerID="134b52c382ff6c88f8c8b84568cb9cef76e0c4e878e6c7e02e62681a00c2422d" exitCode=0 Nov 01 21:04:54.368508 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:54.368340 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-dab736-545mp" event={"ID":"92642861-003f-49a3-9df0-7a2913316106","Type":"ContainerDied","Data":"134b52c382ff6c88f8c8b84568cb9cef76e0c4e878e6c7e02e62681a00c2422d"} Nov 01 21:04:54.771680 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:54.771642 4484 patch_prober.go:29] interesting pod/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Nov 01 21:04:54.771870 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:54.771710 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98" podUID="7403ccd9-e6f4-4ceb-9d97-22e08152f92e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Nov 01 21:04:55.034231 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:55.034140 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-22ed7b-5768c96547-5qpms"] Nov 01 21:04:55.131779 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:55.131738 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-22ed7b-57f689dcd7-8vspb"] Nov 01 21:04:55.132560 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:55.132528 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:55.433169 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:55.433133 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:55.706711 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:55.693050 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:55.771466 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:55.771324 4484 patch_prober.go:29] interesting pod/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Nov 01 21:04:55.771466 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:55.771386 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98" podUID="7403ccd9-e6f4-4ceb-9d97-22e08152f92e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Nov 01 21:04:55.857640 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:04:55.857602 4484 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ecec0b0_7e22_4803_891c_0a75173445ad.slice/crio-2f7560be7d7ec4d112867c55ef155c749075b51ed446557b8d4fba8334ff3e6d.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ecec0b0_7e22_4803_891c_0a75173445ad.slice/crio-conmon-2f7560be7d7ec4d112867c55ef155c749075b51ed446557b8d4fba8334ff3e6d.scope\": RecentStats: unable to find data in memory cache]" Nov 01 21:04:55.877459 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:55.877431 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-dab736-545mp" Nov 01 21:04:55.963502 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:55.963400 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92642861-003f-49a3-9df0-7a2913316106-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92642861-003f-49a3-9df0-7a2913316106-maas-api-openshift-ca\") pod \"92642861-003f-49a3-9df0-7a2913316106\" (UID: \"92642861-003f-49a3-9df0-7a2913316106\") " Nov 01 21:04:55.963686 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:55.963514 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92642861-003f-49a3-9df0-7a2913316106-kube-api-access-6nlvp\" (UniqueName: \"kubernetes.io/projected/92642861-003f-49a3-9df0-7a2913316106-kube-api-access-6nlvp\") pod \"92642861-003f-49a3-9df0-7a2913316106\" (UID: \"92642861-003f-49a3-9df0-7a2913316106\") " Nov 01 21:04:55.963686 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:55.963589 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92642861-003f-49a3-9df0-7a2913316106-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92642861-003f-49a3-9df0-7a2913316106-maas-api-xks-ca\") pod \"92642861-003f-49a3-9df0-7a2913316106\" (UID: \"92642861-003f-49a3-9df0-7a2913316106\") " Nov 01 21:04:55.964260 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:55.964230 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92642861-003f-49a3-9df0-7a2913316106-maas-api-openshift-ca" pod "92642861-003f-49a3-9df0-7a2913316106" (UID: "92642861-003f-49a3-9df0-7a2913316106"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:04:55.966492 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:55.966464 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92642861-003f-49a3-9df0-7a2913316106-maas-api-xks-ca" pod "92642861-003f-49a3-9df0-7a2913316106" (UID: "92642861-003f-49a3-9df0-7a2913316106"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:04:55.967508 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:55.967480 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92642861-003f-49a3-9df0-7a2913316106-kube-api-access-6nlvp" pod "92642861-003f-49a3-9df0-7a2913316106" (UID: "92642861-003f-49a3-9df0-7a2913316106"). InnerVolumeSpecName "kube-api-access-6nlvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:04:56.032212 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:56.032183 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m" Nov 01 21:04:56.066320 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:56.066284 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92642861-003f-49a3-9df0-7a2913316106-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:56.066320 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:56.066316 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nlvp\" (UniqueName: \"kubernetes.io/projected/92642861-003f-49a3-9df0-7a2913316106-kube-api-access-6nlvp\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:56.066769 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:56.066331 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92642861-003f-49a3-9df0-7a2913316106-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:56.167124 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:56.167087 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ecec0b0-7e22-4803-891c-0a75173445ad-kube-api-access-vl6b6\" (UniqueName: \"kubernetes.io/projected/7ecec0b0-7e22-4803-891c-0a75173445ad-kube-api-access-vl6b6\") pod \"7ecec0b0-7e22-4803-891c-0a75173445ad\" (UID: \"7ecec0b0-7e22-4803-891c-0a75173445ad\") " Nov 01 21:04:56.167313 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:56.167293 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ecec0b0-7e22-4803-891c-0a75173445ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ecec0b0-7e22-4803-891c-0a75173445ad-istio-envoy\") pod \"7ecec0b0-7e22-4803-891c-0a75173445ad\" (UID: \"7ecec0b0-7e22-4803-891c-0a75173445ad\") " Nov 01 21:04:56.167410 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:56.167394 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ecec0b0-7e22-4803-891c-0a75173445ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ecec0b0-7e22-4803-891c-0a75173445ad-istiod-ca-cert\") pod \"7ecec0b0-7e22-4803-891c-0a75173445ad\" (UID: \"7ecec0b0-7e22-4803-891c-0a75173445ad\") " Nov 01 21:04:56.167495 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:56.167474 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ecec0b0-7e22-4803-891c-0a75173445ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ecec0b0-7e22-4803-891c-0a75173445ad-istio-data\") pod \"7ecec0b0-7e22-4803-891c-0a75173445ad\" (UID: \"7ecec0b0-7e22-4803-891c-0a75173445ad\") " Nov 01 21:04:56.167578 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:56.167519 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ecec0b0-7e22-4803-891c-0a75173445ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ecec0b0-7e22-4803-891c-0a75173445ad-credential-socket\") pod \"7ecec0b0-7e22-4803-891c-0a75173445ad\" (UID: \"7ecec0b0-7e22-4803-891c-0a75173445ad\") " Nov 01 21:04:56.167578 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:56.167551 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ecec0b0-7e22-4803-891c-0a75173445ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ecec0b0-7e22-4803-891c-0a75173445ad-workload-socket\") pod \"7ecec0b0-7e22-4803-891c-0a75173445ad\" (UID: \"7ecec0b0-7e22-4803-891c-0a75173445ad\") " Nov 01 21:04:56.167698 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:56.167585 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ecec0b0-7e22-4803-891c-0a75173445ad-istio-token\" (UniqueName: \"kubernetes.io/projected/7ecec0b0-7e22-4803-891c-0a75173445ad-istio-token\") pod \"7ecec0b0-7e22-4803-891c-0a75173445ad\" (UID: \"7ecec0b0-7e22-4803-891c-0a75173445ad\") " Nov 01 21:04:56.167698 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:56.167644 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ecec0b0-7e22-4803-891c-0a75173445ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ecec0b0-7e22-4803-891c-0a75173445ad-istio-podinfo\") pod \"7ecec0b0-7e22-4803-891c-0a75173445ad\" (UID: \"7ecec0b0-7e22-4803-891c-0a75173445ad\") " Nov 01 21:04:56.167803 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:56.167766 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ecec0b0-7e22-4803-891c-0a75173445ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ecec0b0-7e22-4803-891c-0a75173445ad-workload-certs\") pod \"7ecec0b0-7e22-4803-891c-0a75173445ad\" (UID: \"7ecec0b0-7e22-4803-891c-0a75173445ad\") " Nov 01 21:04:56.167859 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:56.167788 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ecec0b0-7e22-4803-891c-0a75173445ad-istio-data" pod "7ecec0b0-7e22-4803-891c-0a75173445ad" (UID: "7ecec0b0-7e22-4803-891c-0a75173445ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:04:56.167917 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:56.167864 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ecec0b0-7e22-4803-891c-0a75173445ad-istiod-ca-cert" pod "7ecec0b0-7e22-4803-891c-0a75173445ad" (UID: "7ecec0b0-7e22-4803-891c-0a75173445ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:04:56.168064 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:56.168043 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ecec0b0-7e22-4803-891c-0a75173445ad-workload-socket" pod "7ecec0b0-7e22-4803-891c-0a75173445ad" (UID: "7ecec0b0-7e22-4803-891c-0a75173445ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:04:56.168598 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:56.168543 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ecec0b0-7e22-4803-891c-0a75173445ad-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:56.168598 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:56.168570 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ecec0b0-7e22-4803-891c-0a75173445ad-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:56.168598 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:56.168585 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ecec0b0-7e22-4803-891c-0a75173445ad-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:56.168823 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:56.168645 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ecec0b0-7e22-4803-891c-0a75173445ad-workload-certs" pod "7ecec0b0-7e22-4803-891c-0a75173445ad" (UID: "7ecec0b0-7e22-4803-891c-0a75173445ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:04:56.168823 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:56.168704 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ecec0b0-7e22-4803-891c-0a75173445ad-credential-socket" pod "7ecec0b0-7e22-4803-891c-0a75173445ad" (UID: "7ecec0b0-7e22-4803-891c-0a75173445ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:04:56.170487 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:56.170461 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ecec0b0-7e22-4803-891c-0a75173445ad-kube-api-access-vl6b6" pod "7ecec0b0-7e22-4803-891c-0a75173445ad" (UID: "7ecec0b0-7e22-4803-891c-0a75173445ad"). InnerVolumeSpecName "kube-api-access-vl6b6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:04:56.170832 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:56.170810 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ecec0b0-7e22-4803-891c-0a75173445ad-istio-token" pod "7ecec0b0-7e22-4803-891c-0a75173445ad" (UID: "7ecec0b0-7e22-4803-891c-0a75173445ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:04:56.171133 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:56.171112 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ecec0b0-7e22-4803-891c-0a75173445ad-istio-envoy" pod "7ecec0b0-7e22-4803-891c-0a75173445ad" (UID: "7ecec0b0-7e22-4803-891c-0a75173445ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:04:56.171238 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:56.171180 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ecec0b0-7e22-4803-891c-0a75173445ad-istio-podinfo" pod "7ecec0b0-7e22-4803-891c-0a75173445ad" (UID: "7ecec0b0-7e22-4803-891c-0a75173445ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:04:56.269615 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:56.269581 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ecec0b0-7e22-4803-891c-0a75173445ad-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:56.269615 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:56.269610 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ecec0b0-7e22-4803-891c-0a75173445ad-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:56.269615 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:56.269621 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ecec0b0-7e22-4803-891c-0a75173445ad-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:56.269901 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:56.269630 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ecec0b0-7e22-4803-891c-0a75173445ad-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:56.269901 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:56.269639 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vl6b6\" (UniqueName: \"kubernetes.io/projected/7ecec0b0-7e22-4803-891c-0a75173445ad-kube-api-access-vl6b6\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:56.269901 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:56.269649 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ecec0b0-7e22-4803-891c-0a75173445ad-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:56.383879 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:56.383848 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-dab736-545mp" Nov 01 21:04:56.384079 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:56.383874 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-dab736-545mp" event={"ID":"92642861-003f-49a3-9df0-7a2913316106","Type":"ContainerDied","Data":"0d89d79c04f9fa65b78ca71b76cee0d10fef5f715eee552e475ccf98f4d7578b"} Nov 01 21:04:56.384079 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:56.383922 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d89d79c04f9fa65b78ca71b76cee0d10fef5f715eee552e475ccf98f4d7578b" Nov 01 21:04:56.385822 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:56.385798 4484 generic.go:356] "Generic (PLEG): container finished" podID="7ecec0b0-7e22-4803-891c-0a75173445ad" containerID="2f7560be7d7ec4d112867c55ef155c749075b51ed446557b8d4fba8334ff3e6d" exitCode=0 Nov 01 21:04:56.385966 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:56.385854 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m" Nov 01 21:04:56.385966 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:56.385882 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m" event={"ID":"7ecec0b0-7e22-4803-891c-0a75173445ad","Type":"ContainerDied","Data":"2f7560be7d7ec4d112867c55ef155c749075b51ed446557b8d4fba8334ff3e6d"} Nov 01 21:04:56.385966 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:56.385931 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m" event={"ID":"7ecec0b0-7e22-4803-891c-0a75173445ad","Type":"ContainerDied","Data":"16965be258b9ccf09afc7526afa6cb0b2c8034eea207516e2ddfe21ab1a06695"} Nov 01 21:04:56.385966 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:56.385957 4484 scope.go:122] "RemoveContainer" containerID="2f7560be7d7ec4d112867c55ef155c749075b51ed446557b8d4fba8334ff3e6d" Nov 01 21:04:56.398901 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:56.398881 4484 scope.go:122] "RemoveContainer" containerID="205e260be2d92a416298051fc1b803f8de077fbe104b1b53c437f032521d2608" Nov 01 21:04:56.434283 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:56.434253 4484 scope.go:122] "RemoveContainer" containerID="2f7560be7d7ec4d112867c55ef155c749075b51ed446557b8d4fba8334ff3e6d" Nov 01 21:04:56.434665 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:04:56.434645 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f7560be7d7ec4d112867c55ef155c749075b51ed446557b8d4fba8334ff3e6d\": container with ID starting with 2f7560be7d7ec4d112867c55ef155c749075b51ed446557b8d4fba8334ff3e6d not found: ID does not exist" containerID="2f7560be7d7ec4d112867c55ef155c749075b51ed446557b8d4fba8334ff3e6d" Nov 01 21:04:56.434754 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:56.434675 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f7560be7d7ec4d112867c55ef155c749075b51ed446557b8d4fba8334ff3e6d"} err="failed to get container status \"2f7560be7d7ec4d112867c55ef155c749075b51ed446557b8d4fba8334ff3e6d\": rpc error: code = NotFound desc = could not find container \"2f7560be7d7ec4d112867c55ef155c749075b51ed446557b8d4fba8334ff3e6d\": container with ID starting with 2f7560be7d7ec4d112867c55ef155c749075b51ed446557b8d4fba8334ff3e6d not found: ID does not exist" Nov 01 21:04:56.434754 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:56.434696 4484 scope.go:122] "RemoveContainer" containerID="205e260be2d92a416298051fc1b803f8de077fbe104b1b53c437f032521d2608" Nov 01 21:04:56.434952 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:04:56.434933 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"205e260be2d92a416298051fc1b803f8de077fbe104b1b53c437f032521d2608\": container with ID starting with 205e260be2d92a416298051fc1b803f8de077fbe104b1b53c437f032521d2608 not found: ID does not exist" containerID="205e260be2d92a416298051fc1b803f8de077fbe104b1b53c437f032521d2608" Nov 01 21:04:56.435039 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:56.434957 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"205e260be2d92a416298051fc1b803f8de077fbe104b1b53c437f032521d2608"} err="failed to get container status \"205e260be2d92a416298051fc1b803f8de077fbe104b1b53c437f032521d2608\": rpc error: code = NotFound desc = could not find container \"205e260be2d92a416298051fc1b803f8de077fbe104b1b53c437f032521d2608\": container with ID starting with 205e260be2d92a416298051fc1b803f8de077fbe104b1b53c437f032521d2608 not found: ID does not exist" Nov 01 21:04:56.629494 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:56.625056 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m"] Nov 01 21:04:56.657093 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:56.653068 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-a9b439-openshift-default-77b865d7d7-cfn9m"] Nov 01 21:04:56.773729 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:56.773694 4484 patch_prober.go:29] interesting pod/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Nov 01 21:04:56.775553 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:56.775510 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98" podUID="7403ccd9-e6f4-4ceb-9d97-22e08152f92e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Nov 01 21:04:57.375189 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:57.375160 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ecec0b0-7e22-4803-891c-0a75173445ad" path="/var/lib/kubelet/pods/7ecec0b0-7e22-4803-891c-0a75173445ad/volumes" Nov 01 21:04:57.535809 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:57.535768 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-29baa8-6bbff57b65-6djp8"] Nov 01 21:04:57.654562 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:57.654442 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-29baa8-6c565db695-5m42b"] Nov 01 21:04:57.776392 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:57.776361 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98" Nov 01 21:04:57.776832 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:57.776818 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98" Nov 01 21:04:57.777523 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:57.777501 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98" Nov 01 21:04:57.931794 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:57.931707 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:58.091325 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:04:58.091289 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:00.035412 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:00.035367 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-22ed7b-5768c96547-5qpms" podUID="430400df-fb2d-4390-9208-fc312d705254" containerName="payload-processing" containerID="cri-o://696d1ee186f7008f36a3641a228eebb9ebbe5076949519e17d2bd78ca87dcc77" gracePeriod=25 Nov 01 21:05:00.132412 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:00.132374 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-22ed7b-57f689dcd7-8vspb" podUID="7f54111b-de15-4afa-a188-c8c5f7d14a2d" containerName="payload-pre-processing" containerID="cri-o://a3a76e29696289c13b08abce97d542366503051b07d36d1d06e19e19bc4b225d" gracePeriod=25 Nov 01 21:05:00.273438 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:00.273402 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:00.469938 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:00.469910 4484 generic.go:356] "Generic (PLEG): container finished" podID="7f54111b-de15-4afa-a188-c8c5f7d14a2d" containerID="a3a76e29696289c13b08abce97d542366503051b07d36d1d06e19e19bc4b225d" exitCode=0 Nov 01 21:05:00.470415 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:00.470124 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-22ed7b-57f689dcd7-8vspb" event={"ID":"7f54111b-de15-4afa-a188-c8c5f7d14a2d","Type":"ContainerDied","Data":"a3a76e29696289c13b08abce97d542366503051b07d36d1d06e19e19bc4b225d"} Nov 01 21:05:00.471775 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:00.471750 4484 generic.go:356] "Generic (PLEG): container finished" podID="430400df-fb2d-4390-9208-fc312d705254" containerID="696d1ee186f7008f36a3641a228eebb9ebbe5076949519e17d2bd78ca87dcc77" exitCode=0 Nov 01 21:05:00.471876 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:00.471784 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-22ed7b-5768c96547-5qpms" event={"ID":"430400df-fb2d-4390-9208-fc312d705254","Type":"ContainerDied","Data":"696d1ee186f7008f36a3641a228eebb9ebbe5076949519e17d2bd78ca87dcc77"} Nov 01 21:05:00.471876 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:00.471824 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-22ed7b-5768c96547-5qpms" event={"ID":"430400df-fb2d-4390-9208-fc312d705254","Type":"ContainerDied","Data":"9e69767f0b6b251c7ae1ee09ff78fc8a9f89c0f32f23034298f587df08cfe5a5"} Nov 01 21:05:00.471876 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:00.471847 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e69767f0b6b251c7ae1ee09ff78fc8a9f89c0f32f23034298f587df08cfe5a5" Nov 01 21:05:00.497065 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:00.497037 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-22ed7b-5768c96547-5qpms" Nov 01 21:05:00.520018 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:00.519951 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/430400df-fb2d-4390-9208-fc312d705254-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/430400df-fb2d-4390-9208-fc312d705254-plugins-config-volume\") pod \"430400df-fb2d-4390-9208-fc312d705254\" (UID: \"430400df-fb2d-4390-9208-fc312d705254\") " Nov 01 21:05:00.520288 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:00.520268 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/430400df-fb2d-4390-9208-fc312d705254-kube-api-access-8twj8\" (UniqueName: \"kubernetes.io/projected/430400df-fb2d-4390-9208-fc312d705254-kube-api-access-8twj8\") pod \"430400df-fb2d-4390-9208-fc312d705254\" (UID: \"430400df-fb2d-4390-9208-fc312d705254\") " Nov 01 21:05:00.520327 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:00.520295 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/430400df-fb2d-4390-9208-fc312d705254-plugins-config-volume" pod "430400df-fb2d-4390-9208-fc312d705254" (UID: "430400df-fb2d-4390-9208-fc312d705254"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:05:00.521052 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:00.521028 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/430400df-fb2d-4390-9208-fc312d705254-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:00.522406 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:00.522380 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/430400df-fb2d-4390-9208-fc312d705254-kube-api-access-8twj8" pod "430400df-fb2d-4390-9208-fc312d705254" (UID: "430400df-fb2d-4390-9208-fc312d705254"). InnerVolumeSpecName "kube-api-access-8twj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:05:00.559461 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:00.559438 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-22ed7b-57f689dcd7-8vspb" Nov 01 21:05:00.621548 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:00.621456 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f54111b-de15-4afa-a188-c8c5f7d14a2d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f54111b-de15-4afa-a188-c8c5f7d14a2d-plugins-config-volume\") pod \"7f54111b-de15-4afa-a188-c8c5f7d14a2d\" (UID: \"7f54111b-de15-4afa-a188-c8c5f7d14a2d\") " Nov 01 21:05:00.621741 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:00.621722 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f54111b-de15-4afa-a188-c8c5f7d14a2d-kube-api-access-mj4nw\" (UniqueName: \"kubernetes.io/projected/7f54111b-de15-4afa-a188-c8c5f7d14a2d-kube-api-access-mj4nw\") pod \"7f54111b-de15-4afa-a188-c8c5f7d14a2d\" (UID: \"7f54111b-de15-4afa-a188-c8c5f7d14a2d\") " Nov 01 21:05:00.621869 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:00.621846 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f54111b-de15-4afa-a188-c8c5f7d14a2d-plugins-config-volume" pod "7f54111b-de15-4afa-a188-c8c5f7d14a2d" (UID: "7f54111b-de15-4afa-a188-c8c5f7d14a2d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:05:00.622307 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:00.622281 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8twj8\" (UniqueName: \"kubernetes.io/projected/430400df-fb2d-4390-9208-fc312d705254-kube-api-access-8twj8\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:00.622307 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:00.622305 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f54111b-de15-4afa-a188-c8c5f7d14a2d-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:00.623882 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:00.623852 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f54111b-de15-4afa-a188-c8c5f7d14a2d-kube-api-access-mj4nw" pod "7f54111b-de15-4afa-a188-c8c5f7d14a2d" (UID: "7f54111b-de15-4afa-a188-c8c5f7d14a2d"). InnerVolumeSpecName "kube-api-access-mj4nw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:05:00.723245 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:00.723215 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mj4nw\" (UniqueName: \"kubernetes.io/projected/7f54111b-de15-4afa-a188-c8c5f7d14a2d-kube-api-access-mj4nw\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:01.480546 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:01.480514 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-22ed7b-57f689dcd7-8vspb" Nov 01 21:05:01.481071 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:01.480544 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-22ed7b-57f689dcd7-8vspb" event={"ID":"7f54111b-de15-4afa-a188-c8c5f7d14a2d","Type":"ContainerDied","Data":"207f48415871ff6308a4d9040fe270c26ace95771cbff011fcfd789b11526607"} Nov 01 21:05:01.481071 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:01.480623 4484 scope.go:122] "RemoveContainer" containerID="a3a76e29696289c13b08abce97d542366503051b07d36d1d06e19e19bc4b225d" Nov 01 21:05:01.481071 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:01.480795 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-22ed7b-5768c96547-5qpms" Nov 01 21:05:01.704896 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:01.704811 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-22ed7b-57f689dcd7-8vspb"] Nov 01 21:05:01.755198 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:01.755159 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-22ed7b-57f689dcd7-8vspb"] Nov 01 21:05:01.823450 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:01.823413 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-22ed7b-5768c96547-5qpms"] Nov 01 21:05:01.827755 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:01.827725 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-22ed7b-5768c96547-5qpms"] Nov 01 21:05:02.345186 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:02.345139 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:02.536591 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:02.536547 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-29baa8-6bbff57b65-6djp8" podUID="4048d711-2455-4907-b85f-0baa9602e224" containerName="payload-processing" containerID="cri-o://c7141cbaa17e41d34f189140fef8f80314dc87f7456324ee34bfe162d9632ce3" gracePeriod=25 Nov 01 21:05:02.643882 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:02.643795 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:02.655336 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:02.655270 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-29baa8-6c565db695-5m42b" podUID="64a2f0cb-f518-45ff-9f7c-69cd38eacdeb" containerName="payload-pre-processing" containerID="cri-o://3a748d11adb52251c619a14166fc56369c6912b6ef19b0adc16c9885f56e1adc" gracePeriod=25 Nov 01 21:05:02.891454 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:02.890435 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-29baa8-6bbff57b65-6djp8" Nov 01 21:05:02.935714 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:02.935613 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-081c8e-4j2qn"] Nov 01 21:05:02.952393 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:02.952364 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4048d711-2455-4907-b85f-0baa9602e224-kube-api-access-tnp8q\" (UniqueName: \"kubernetes.io/projected/4048d711-2455-4907-b85f-0baa9602e224-kube-api-access-tnp8q\") pod \"4048d711-2455-4907-b85f-0baa9602e224\" (UID: \"4048d711-2455-4907-b85f-0baa9602e224\") " Nov 01 21:05:02.952570 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:02.952554 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4048d711-2455-4907-b85f-0baa9602e224-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4048d711-2455-4907-b85f-0baa9602e224-plugins-config-volume\") pod \"4048d711-2455-4907-b85f-0baa9602e224\" (UID: \"4048d711-2455-4907-b85f-0baa9602e224\") " Nov 01 21:05:02.952891 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:02.952867 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4048d711-2455-4907-b85f-0baa9602e224-plugins-config-volume" pod "4048d711-2455-4907-b85f-0baa9602e224" (UID: "4048d711-2455-4907-b85f-0baa9602e224"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:05:02.953115 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:02.953101 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4048d711-2455-4907-b85f-0baa9602e224-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:02.954505 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:02.954480 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4048d711-2455-4907-b85f-0baa9602e224-kube-api-access-tnp8q" pod "4048d711-2455-4907-b85f-0baa9602e224" (UID: "4048d711-2455-4907-b85f-0baa9602e224"). InnerVolumeSpecName "kube-api-access-tnp8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:05:02.970411 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:02.970384 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-29baa8-6c565db695-5m42b" Nov 01 21:05:02.986416 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:02.986378 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-081c8e-4j2qn"] Nov 01 21:05:03.053534 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:03.053499 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64a2f0cb-f518-45ff-9f7c-69cd38eacdeb-kube-api-access-hh8bf\" (UniqueName: \"kubernetes.io/projected/64a2f0cb-f518-45ff-9f7c-69cd38eacdeb-kube-api-access-hh8bf\") pod \"64a2f0cb-f518-45ff-9f7c-69cd38eacdeb\" (UID: \"64a2f0cb-f518-45ff-9f7c-69cd38eacdeb\") " Nov 01 21:05:03.053739 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:03.053682 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64a2f0cb-f518-45ff-9f7c-69cd38eacdeb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64a2f0cb-f518-45ff-9f7c-69cd38eacdeb-plugins-config-volume\") pod \"64a2f0cb-f518-45ff-9f7c-69cd38eacdeb\" (UID: \"64a2f0cb-f518-45ff-9f7c-69cd38eacdeb\") " Nov 01 21:05:03.054112 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:03.054084 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64a2f0cb-f518-45ff-9f7c-69cd38eacdeb-plugins-config-volume" pod "64a2f0cb-f518-45ff-9f7c-69cd38eacdeb" (UID: "64a2f0cb-f518-45ff-9f7c-69cd38eacdeb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:05:03.054391 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:03.054371 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64a2f0cb-f518-45ff-9f7c-69cd38eacdeb-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:03.054391 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:03.054391 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnp8q\" (UniqueName: \"kubernetes.io/projected/4048d711-2455-4907-b85f-0baa9602e224-kube-api-access-tnp8q\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:03.055867 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:03.055834 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64a2f0cb-f518-45ff-9f7c-69cd38eacdeb-kube-api-access-hh8bf" pod "64a2f0cb-f518-45ff-9f7c-69cd38eacdeb" (UID: "64a2f0cb-f518-45ff-9f7c-69cd38eacdeb"). InnerVolumeSpecName "kube-api-access-hh8bf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:05:03.083299 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:03.083262 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-dab736-5b68f8cd7c-947ht"] Nov 01 21:05:03.155540 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:03.155504 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hh8bf\" (UniqueName: \"kubernetes.io/projected/64a2f0cb-f518-45ff-9f7c-69cd38eacdeb-kube-api-access-hh8bf\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:03.183906 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:03.183869 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-dab736-7b55d47c59-lrvjr"] Nov 01 21:05:03.303767 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:03.303727 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:03.375120 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:03.375089 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="430400df-fb2d-4390-9208-fc312d705254" path="/var/lib/kubelet/pods/430400df-fb2d-4390-9208-fc312d705254/volumes" Nov 01 21:05:03.375544 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:03.375529 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f54111b-de15-4afa-a188-c8c5f7d14a2d" path="/var/lib/kubelet/pods/7f54111b-de15-4afa-a188-c8c5f7d14a2d/volumes" Nov 01 21:05:03.375931 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:03.375918 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a79544f1-af43-4f27-a247-9da722d1508e" path="/var/lib/kubelet/pods/a79544f1-af43-4f27-a247-9da722d1508e/volumes" Nov 01 21:05:03.474033 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:03.473982 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:03.578127 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:03.577193 4484 generic.go:356] "Generic (PLEG): container finished" podID="64a2f0cb-f518-45ff-9f7c-69cd38eacdeb" containerID="3a748d11adb52251c619a14166fc56369c6912b6ef19b0adc16c9885f56e1adc" exitCode=0 Nov 01 21:05:03.578127 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:03.577393 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-29baa8-6c565db695-5m42b" event={"ID":"64a2f0cb-f518-45ff-9f7c-69cd38eacdeb","Type":"ContainerDied","Data":"3a748d11adb52251c619a14166fc56369c6912b6ef19b0adc16c9885f56e1adc"} Nov 01 21:05:03.578127 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:03.577446 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-29baa8-6c565db695-5m42b" event={"ID":"64a2f0cb-f518-45ff-9f7c-69cd38eacdeb","Type":"ContainerDied","Data":"a1bed4d63ab788808d647befe9ba773d22fd3b875f744ca6e2d732966fcfb6c9"} Nov 01 21:05:03.578127 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:03.577476 4484 scope.go:122] "RemoveContainer" containerID="3a748d11adb52251c619a14166fc56369c6912b6ef19b0adc16c9885f56e1adc" Nov 01 21:05:03.578127 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:03.577628 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-29baa8-6c565db695-5m42b" Nov 01 21:05:03.597817 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:03.597783 4484 generic.go:356] "Generic (PLEG): container finished" podID="4048d711-2455-4907-b85f-0baa9602e224" containerID="c7141cbaa17e41d34f189140fef8f80314dc87f7456324ee34bfe162d9632ce3" exitCode=0 Nov 01 21:05:03.597963 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:03.597938 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-29baa8-6bbff57b65-6djp8" event={"ID":"4048d711-2455-4907-b85f-0baa9602e224","Type":"ContainerDied","Data":"c7141cbaa17e41d34f189140fef8f80314dc87f7456324ee34bfe162d9632ce3"} Nov 01 21:05:03.598088 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:03.598007 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-29baa8-6bbff57b65-6djp8" event={"ID":"4048d711-2455-4907-b85f-0baa9602e224","Type":"ContainerDied","Data":"4ad6c742592f9892764831f6382bbe393aa43fa2622966149cad423b9875ed40"} Nov 01 21:05:03.598088 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:03.598085 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-29baa8-6bbff57b65-6djp8" Nov 01 21:05:03.614245 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:03.614191 4484 scope.go:122] "RemoveContainer" containerID="3a748d11adb52251c619a14166fc56369c6912b6ef19b0adc16c9885f56e1adc" Nov 01 21:05:03.617337 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:05:03.615896 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a748d11adb52251c619a14166fc56369c6912b6ef19b0adc16c9885f56e1adc\": container with ID starting with 3a748d11adb52251c619a14166fc56369c6912b6ef19b0adc16c9885f56e1adc not found: ID does not exist" containerID="3a748d11adb52251c619a14166fc56369c6912b6ef19b0adc16c9885f56e1adc" Nov 01 21:05:03.617337 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:03.615942 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a748d11adb52251c619a14166fc56369c6912b6ef19b0adc16c9885f56e1adc"} err="failed to get container status \"3a748d11adb52251c619a14166fc56369c6912b6ef19b0adc16c9885f56e1adc\": rpc error: code = NotFound desc = could not find container \"3a748d11adb52251c619a14166fc56369c6912b6ef19b0adc16c9885f56e1adc\": container with ID starting with 3a748d11adb52251c619a14166fc56369c6912b6ef19b0adc16c9885f56e1adc not found: ID does not exist" Nov 01 21:05:03.617337 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:03.615970 4484 scope.go:122] "RemoveContainer" containerID="c7141cbaa17e41d34f189140fef8f80314dc87f7456324ee34bfe162d9632ce3" Nov 01 21:05:03.640666 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:03.640629 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:03.641982 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:03.641952 4484 scope.go:122] "RemoveContainer" containerID="c7141cbaa17e41d34f189140fef8f80314dc87f7456324ee34bfe162d9632ce3" Nov 01 21:05:03.642416 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:05:03.642393 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7141cbaa17e41d34f189140fef8f80314dc87f7456324ee34bfe162d9632ce3\": container with ID starting with c7141cbaa17e41d34f189140fef8f80314dc87f7456324ee34bfe162d9632ce3 not found: ID does not exist" containerID="c7141cbaa17e41d34f189140fef8f80314dc87f7456324ee34bfe162d9632ce3" Nov 01 21:05:03.642503 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:03.642428 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7141cbaa17e41d34f189140fef8f80314dc87f7456324ee34bfe162d9632ce3"} err="failed to get container status \"c7141cbaa17e41d34f189140fef8f80314dc87f7456324ee34bfe162d9632ce3\": rpc error: code = NotFound desc = could not find container \"c7141cbaa17e41d34f189140fef8f80314dc87f7456324ee34bfe162d9632ce3\": container with ID starting with c7141cbaa17e41d34f189140fef8f80314dc87f7456324ee34bfe162d9632ce3 not found: ID does not exist" Nov 01 21:05:03.760085 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:03.760050 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-29baa8-6c565db695-5m42b"] Nov 01 21:05:03.764594 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:03.764556 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-29baa8-6c565db695-5m42b"] Nov 01 21:05:03.875577 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:03.875489 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-29baa8-6bbff57b65-6djp8"] Nov 01 21:05:03.879623 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:03.879592 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-29baa8-6bbff57b65-6djp8"] Nov 01 21:05:05.373750 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:05.373716 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4048d711-2455-4907-b85f-0baa9602e224" path="/var/lib/kubelet/pods/4048d711-2455-4907-b85f-0baa9602e224/volumes" Nov 01 21:05:05.374428 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:05.374406 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64a2f0cb-f518-45ff-9f7c-69cd38eacdeb" path="/var/lib/kubelet/pods/64a2f0cb-f518-45ff-9f7c-69cd38eacdeb/volumes" Nov 01 21:05:05.564011 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:05.563971 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:08.084500 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:08.084454 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-dab736-5b68f8cd7c-947ht" podUID="a1da92eb-3960-46cf-836d-2b45d4aa2497" containerName="payload-processing" containerID="cri-o://27e31e8a844e74f4ea8db2c23f6fdad7b74b7d4c9829daf2b898f54c5e4c1705" gracePeriod=25 Nov 01 21:05:08.184899 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:08.184849 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-dab736-7b55d47c59-lrvjr" podUID="749e3d84-a7c5-4340-bda6-d16818decb9b" containerName="payload-pre-processing" containerID="cri-o://04e1aa87c0bc3128bb580e445d6244d06b1d0e4c7f10eb5ad73137312b623123" gracePeriod=25 Nov 01 21:05:08.386293 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:08.386259 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-dab736-5b68f8cd7c-947ht" Nov 01 21:05:08.471264 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:08.471233 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-dab736-7b55d47c59-lrvjr" Nov 01 21:05:08.483597 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:08.483564 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1da92eb-3960-46cf-836d-2b45d4aa2497-kube-api-access-ltvfn\" (UniqueName: \"kubernetes.io/projected/a1da92eb-3960-46cf-836d-2b45d4aa2497-kube-api-access-ltvfn\") pod \"a1da92eb-3960-46cf-836d-2b45d4aa2497\" (UID: \"a1da92eb-3960-46cf-836d-2b45d4aa2497\") " Nov 01 21:05:08.483773 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:08.483604 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1da92eb-3960-46cf-836d-2b45d4aa2497-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1da92eb-3960-46cf-836d-2b45d4aa2497-plugins-config-volume\") pod \"a1da92eb-3960-46cf-836d-2b45d4aa2497\" (UID: \"a1da92eb-3960-46cf-836d-2b45d4aa2497\") " Nov 01 21:05:08.484103 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:08.484079 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1da92eb-3960-46cf-836d-2b45d4aa2497-plugins-config-volume" pod "a1da92eb-3960-46cf-836d-2b45d4aa2497" (UID: "a1da92eb-3960-46cf-836d-2b45d4aa2497"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:05:08.485787 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:08.485765 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1da92eb-3960-46cf-836d-2b45d4aa2497-kube-api-access-ltvfn" pod "a1da92eb-3960-46cf-836d-2b45d4aa2497" (UID: "a1da92eb-3960-46cf-836d-2b45d4aa2497"). InnerVolumeSpecName "kube-api-access-ltvfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:05:08.585498 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:08.585466 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/749e3d84-a7c5-4340-bda6-d16818decb9b-kube-api-access-zswjn\" (UniqueName: \"kubernetes.io/projected/749e3d84-a7c5-4340-bda6-d16818decb9b-kube-api-access-zswjn\") pod \"749e3d84-a7c5-4340-bda6-d16818decb9b\" (UID: \"749e3d84-a7c5-4340-bda6-d16818decb9b\") " Nov 01 21:05:08.585676 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:08.585545 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/749e3d84-a7c5-4340-bda6-d16818decb9b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/749e3d84-a7c5-4340-bda6-d16818decb9b-plugins-config-volume\") pod \"749e3d84-a7c5-4340-bda6-d16818decb9b\" (UID: \"749e3d84-a7c5-4340-bda6-d16818decb9b\") " Nov 01 21:05:08.585922 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:08.585893 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/749e3d84-a7c5-4340-bda6-d16818decb9b-plugins-config-volume" pod "749e3d84-a7c5-4340-bda6-d16818decb9b" (UID: "749e3d84-a7c5-4340-bda6-d16818decb9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:05:08.586136 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:08.586121 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltvfn\" (UniqueName: \"kubernetes.io/projected/a1da92eb-3960-46cf-836d-2b45d4aa2497-kube-api-access-ltvfn\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:08.586201 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:08.586139 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1da92eb-3960-46cf-836d-2b45d4aa2497-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:08.586201 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:08.586152 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/749e3d84-a7c5-4340-bda6-d16818decb9b-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:08.587578 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:08.587558 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/749e3d84-a7c5-4340-bda6-d16818decb9b-kube-api-access-zswjn" pod "749e3d84-a7c5-4340-bda6-d16818decb9b" (UID: "749e3d84-a7c5-4340-bda6-d16818decb9b"). InnerVolumeSpecName "kube-api-access-zswjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:05:08.687623 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:08.687538 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zswjn\" (UniqueName: \"kubernetes.io/projected/749e3d84-a7c5-4340-bda6-d16818decb9b-kube-api-access-zswjn\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:08.728663 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:08.728626 4484 generic.go:356] "Generic (PLEG): container finished" podID="a1da92eb-3960-46cf-836d-2b45d4aa2497" containerID="27e31e8a844e74f4ea8db2c23f6fdad7b74b7d4c9829daf2b898f54c5e4c1705" exitCode=0 Nov 01 21:05:08.728876 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:08.728673 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-dab736-5b68f8cd7c-947ht" Nov 01 21:05:08.728876 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:08.728712 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-dab736-5b68f8cd7c-947ht" event={"ID":"a1da92eb-3960-46cf-836d-2b45d4aa2497","Type":"ContainerDied","Data":"27e31e8a844e74f4ea8db2c23f6fdad7b74b7d4c9829daf2b898f54c5e4c1705"} Nov 01 21:05:08.728876 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:08.728769 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-dab736-5b68f8cd7c-947ht" event={"ID":"a1da92eb-3960-46cf-836d-2b45d4aa2497","Type":"ContainerDied","Data":"b2bd753ebd68c574638ca0aabfd08669ba343cb3c83de9ef32457e0b23d2c6f0"} Nov 01 21:05:08.728876 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:08.728799 4484 scope.go:122] "RemoveContainer" containerID="27e31e8a844e74f4ea8db2c23f6fdad7b74b7d4c9829daf2b898f54c5e4c1705" Nov 01 21:05:08.730253 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:08.730230 4484 generic.go:356] "Generic (PLEG): container finished" podID="749e3d84-a7c5-4340-bda6-d16818decb9b" containerID="04e1aa87c0bc3128bb580e445d6244d06b1d0e4c7f10eb5ad73137312b623123" exitCode=0 Nov 01 21:05:08.730371 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:08.730287 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-dab736-7b55d47c59-lrvjr" Nov 01 21:05:08.730371 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:08.730287 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-dab736-7b55d47c59-lrvjr" event={"ID":"749e3d84-a7c5-4340-bda6-d16818decb9b","Type":"ContainerDied","Data":"04e1aa87c0bc3128bb580e445d6244d06b1d0e4c7f10eb5ad73137312b623123"} Nov 01 21:05:08.730457 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:08.730378 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-dab736-7b55d47c59-lrvjr" event={"ID":"749e3d84-a7c5-4340-bda6-d16818decb9b","Type":"ContainerDied","Data":"48ae3aa5cbfec8006316556bad1b24ae94d3b3632497f44e385d790f15234462"} Nov 01 21:05:08.740058 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:08.739869 4484 scope.go:122] "RemoveContainer" containerID="27e31e8a844e74f4ea8db2c23f6fdad7b74b7d4c9829daf2b898f54c5e4c1705" Nov 01 21:05:08.740350 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:05:08.740316 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27e31e8a844e74f4ea8db2c23f6fdad7b74b7d4c9829daf2b898f54c5e4c1705\": container with ID starting with 27e31e8a844e74f4ea8db2c23f6fdad7b74b7d4c9829daf2b898f54c5e4c1705 not found: ID does not exist" containerID="27e31e8a844e74f4ea8db2c23f6fdad7b74b7d4c9829daf2b898f54c5e4c1705" Nov 01 21:05:08.740455 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:08.740360 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27e31e8a844e74f4ea8db2c23f6fdad7b74b7d4c9829daf2b898f54c5e4c1705"} err="failed to get container status \"27e31e8a844e74f4ea8db2c23f6fdad7b74b7d4c9829daf2b898f54c5e4c1705\": rpc error: code = NotFound desc = could not find container \"27e31e8a844e74f4ea8db2c23f6fdad7b74b7d4c9829daf2b898f54c5e4c1705\": container with ID starting with 27e31e8a844e74f4ea8db2c23f6fdad7b74b7d4c9829daf2b898f54c5e4c1705 not found: ID does not exist" Nov 01 21:05:08.740455 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:08.740383 4484 scope.go:122] "RemoveContainer" containerID="04e1aa87c0bc3128bb580e445d6244d06b1d0e4c7f10eb5ad73137312b623123" Nov 01 21:05:08.750787 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:08.750764 4484 scope.go:122] "RemoveContainer" containerID="04e1aa87c0bc3128bb580e445d6244d06b1d0e4c7f10eb5ad73137312b623123" Nov 01 21:05:08.751123 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:05:08.751102 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04e1aa87c0bc3128bb580e445d6244d06b1d0e4c7f10eb5ad73137312b623123\": container with ID starting with 04e1aa87c0bc3128bb580e445d6244d06b1d0e4c7f10eb5ad73137312b623123 not found: ID does not exist" containerID="04e1aa87c0bc3128bb580e445d6244d06b1d0e4c7f10eb5ad73137312b623123" Nov 01 21:05:08.751197 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:08.751134 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04e1aa87c0bc3128bb580e445d6244d06b1d0e4c7f10eb5ad73137312b623123"} err="failed to get container status \"04e1aa87c0bc3128bb580e445d6244d06b1d0e4c7f10eb5ad73137312b623123\": rpc error: code = NotFound desc = could not find container \"04e1aa87c0bc3128bb580e445d6244d06b1d0e4c7f10eb5ad73137312b623123\": container with ID starting with 04e1aa87c0bc3128bb580e445d6244d06b1d0e4c7f10eb5ad73137312b623123 not found: ID does not exist" Nov 01 21:05:08.896821 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:08.896788 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-dab736-5b68f8cd7c-947ht"] Nov 01 21:05:08.900475 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:08.900453 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-dab736-5b68f8cd7c-947ht"] Nov 01 21:05:09.009747 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:09.009717 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-dab736-7b55d47c59-lrvjr"] Nov 01 21:05:09.013471 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:09.013440 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-dab736-7b55d47c59-lrvjr"] Nov 01 21:05:09.035987 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:09.035954 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:09.198440 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:09.198407 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:09.297975 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:09.294743 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m"] Nov 01 21:05:09.297975 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:09.294988 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m" podUID="3c818ecc-7536-4e57-93f6-40ba5942d318" containerName="istio-proxy" containerID="cri-o://521531af618eb362a74aed2f6b877b3991146bfe2614cb17482b05f74b816b7d" gracePeriod=30 Nov 01 21:05:09.366857 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:09.366821 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:09.383197 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:09.383157 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="749e3d84-a7c5-4340-bda6-d16818decb9b" path="/var/lib/kubelet/pods/749e3d84-a7c5-4340-bda6-d16818decb9b/volumes" Nov 01 21:05:09.401478 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:09.385597 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1da92eb-3960-46cf-836d-2b45d4aa2497" path="/var/lib/kubelet/pods/a1da92eb-3960-46cf-836d-2b45d4aa2497/volumes" Nov 01 21:05:09.994224 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:09.994188 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:11.090573 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:11.090545 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:11.719754 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:11.719720 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:12.010434 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:12.010368 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:12.161110 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:12.161069 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:12.222495 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:12.222460 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-dab736-545mp"] Nov 01 21:05:12.273185 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:12.273094 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-dab736-545mp"] Nov 01 21:05:12.629961 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:12.629867 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:12.636247 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:12.636222 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9ae21-494d5"] Nov 01 21:05:12.636921 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:12.636908 4484 cpu_manager.go:414] "containerMap: removing container" podUID="4048d711-2455-4907-b85f-0baa9602e224" containerName="payload-processing" Nov 01 21:05:12.636991 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:12.636922 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4048d711-2455-4907-b85f-0baa9602e224" containerName="payload-processing" Nov 01 21:05:12.636991 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:12.636935 4484 cpu_manager.go:414] "containerMap: removing container" podUID="7ecec0b0-7e22-4803-891c-0a75173445ad" containerName="istio-proxy" Nov 01 21:05:12.636991 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:12.636941 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ecec0b0-7e22-4803-891c-0a75173445ad" containerName="istio-proxy" Nov 01 21:05:12.636991 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:12.636961 4484 cpu_manager.go:414] "containerMap: removing container" podUID="64a2f0cb-f518-45ff-9f7c-69cd38eacdeb" containerName="payload-pre-processing" Nov 01 21:05:12.636991 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:12.636967 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64a2f0cb-f518-45ff-9f7c-69cd38eacdeb" containerName="payload-pre-processing" Nov 01 21:05:12.636991 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:12.636980 4484 cpu_manager.go:414] "containerMap: removing container" podUID="a1da92eb-3960-46cf-836d-2b45d4aa2497" containerName="payload-processing" Nov 01 21:05:12.636991 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:12.636985 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1da92eb-3960-46cf-836d-2b45d4aa2497" containerName="payload-processing" Nov 01 21:05:12.636991 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:12.637006 4484 cpu_manager.go:414] "containerMap: removing container" podUID="430400df-fb2d-4390-9208-fc312d705254" containerName="payload-processing" Nov 01 21:05:12.637428 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:12.637012 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="430400df-fb2d-4390-9208-fc312d705254" containerName="payload-processing" Nov 01 21:05:12.637428 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:12.637022 4484 cpu_manager.go:414] "containerMap: removing container" podUID="92642861-003f-49a3-9df0-7a2913316106" containerName="revoke-keys" Nov 01 21:05:12.637428 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:12.637026 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92642861-003f-49a3-9df0-7a2913316106" containerName="revoke-keys" Nov 01 21:05:12.637428 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:12.637033 4484 cpu_manager.go:414] "containerMap: removing container" podUID="7ecec0b0-7e22-4803-891c-0a75173445ad" containerName="istio-proxy" Nov 01 21:05:12.637428 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:12.637038 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ecec0b0-7e22-4803-891c-0a75173445ad" containerName="istio-proxy" Nov 01 21:05:12.637428 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:12.637049 4484 cpu_manager.go:414] "containerMap: removing container" podUID="7f54111b-de15-4afa-a188-c8c5f7d14a2d" containerName="payload-pre-processing" Nov 01 21:05:12.637428 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:12.637053 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f54111b-de15-4afa-a188-c8c5f7d14a2d" containerName="payload-pre-processing" Nov 01 21:05:12.637428 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:12.637075 4484 cpu_manager.go:414] "containerMap: removing container" podUID="749e3d84-a7c5-4340-bda6-d16818decb9b" containerName="payload-pre-processing" Nov 01 21:05:12.637428 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:12.637080 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="749e3d84-a7c5-4340-bda6-d16818decb9b" containerName="payload-pre-processing" Nov 01 21:05:12.637428 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:12.637173 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="64a2f0cb-f518-45ff-9f7c-69cd38eacdeb" containerName="payload-pre-processing" Nov 01 21:05:12.637428 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:12.637182 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1da92eb-3960-46cf-836d-2b45d4aa2497" containerName="payload-processing" Nov 01 21:05:12.637428 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:12.637188 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="430400df-fb2d-4390-9208-fc312d705254" containerName="payload-processing" Nov 01 21:05:12.637428 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:12.637193 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="92642861-003f-49a3-9df0-7a2913316106" containerName="revoke-keys" Nov 01 21:05:12.637428 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:12.637199 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ecec0b0-7e22-4803-891c-0a75173445ad" containerName="istio-proxy" Nov 01 21:05:12.637428 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:12.637205 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f54111b-de15-4afa-a188-c8c5f7d14a2d" containerName="payload-pre-processing" Nov 01 21:05:12.637428 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:12.637215 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="749e3d84-a7c5-4340-bda6-d16818decb9b" containerName="payload-pre-processing" Nov 01 21:05:12.637428 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:12.637224 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="4048d711-2455-4907-b85f-0baa9602e224" containerName="payload-processing" Nov 01 21:05:12.637428 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:12.637228 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ecec0b0-7e22-4803-891c-0a75173445ad" containerName="istio-proxy" Nov 01 21:05:12.642482 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:12.642463 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9ae21-494d5" Nov 01 21:05:12.696711 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:12.696677 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9ae21-494d5"] Nov 01 21:05:12.716723 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:12.716687 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qbzp\" (UniqueName: \"kubernetes.io/projected/8b20f2ca-5b06-4e4e-a5ef-f68b8414f2ac-kube-api-access-4qbzp\") pod \"maas-api-revoke-keys-e2e-worker-w5-c9ae21-494d5\" (UID: \"8b20f2ca-5b06-4e4e-a5ef-f68b8414f2ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9ae21-494d5" Nov 01 21:05:12.716916 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:12.716768 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b20f2ca-5b06-4e4e-a5ef-f68b8414f2ac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-c9ae21-494d5\" (UID: \"8b20f2ca-5b06-4e4e-a5ef-f68b8414f2ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9ae21-494d5" Nov 01 21:05:12.717040 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:12.716903 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b20f2ca-5b06-4e4e-a5ef-f68b8414f2ac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-c9ae21-494d5\" (UID: \"8b20f2ca-5b06-4e4e-a5ef-f68b8414f2ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9ae21-494d5" Nov 01 21:05:12.742798 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:12.742755 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:05:12.742798 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:12.742755 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:05:12.743022 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:12.742799 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:05:12.743022 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:12.742830 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:05:12.853021 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:12.851558 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:12.857027 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:12.854665 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qbzp\" (UniqueName: \"kubernetes.io/projected/8b20f2ca-5b06-4e4e-a5ef-f68b8414f2ac-kube-api-access-4qbzp\") pod \"maas-api-revoke-keys-e2e-worker-w5-c9ae21-494d5\" (UID: \"8b20f2ca-5b06-4e4e-a5ef-f68b8414f2ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9ae21-494d5" Nov 01 21:05:12.857027 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:12.854740 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b20f2ca-5b06-4e4e-a5ef-f68b8414f2ac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-c9ae21-494d5\" (UID: \"8b20f2ca-5b06-4e4e-a5ef-f68b8414f2ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9ae21-494d5" Nov 01 21:05:12.857027 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:12.854848 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b20f2ca-5b06-4e4e-a5ef-f68b8414f2ac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-c9ae21-494d5\" (UID: \"8b20f2ca-5b06-4e4e-a5ef-f68b8414f2ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9ae21-494d5" Nov 01 21:05:12.857027 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:12.856469 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b20f2ca-5b06-4e4e-a5ef-f68b8414f2ac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-c9ae21-494d5\" (UID: \"8b20f2ca-5b06-4e4e-a5ef-f68b8414f2ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9ae21-494d5" Nov 01 21:05:12.926099 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:12.919582 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b20f2ca-5b06-4e4e-a5ef-f68b8414f2ac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-c9ae21-494d5\" (UID: \"8b20f2ca-5b06-4e4e-a5ef-f68b8414f2ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9ae21-494d5" Nov 01 21:05:12.926099 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:12.925545 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qbzp\" (UniqueName: \"kubernetes.io/projected/8b20f2ca-5b06-4e4e-a5ef-f68b8414f2ac-kube-api-access-4qbzp\") pod \"maas-api-revoke-keys-e2e-worker-w5-c9ae21-494d5\" (UID: \"8b20f2ca-5b06-4e4e-a5ef-f68b8414f2ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9ae21-494d5" Nov 01 21:05:12.964673 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:12.964429 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9ae21-494d5" Nov 01 21:05:13.354488 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:05:13.354459 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b20f2ca_5b06_4e4e_a5ef_f68b8414f2ac.slice/crio-3af143248dd45aa48c0e70d2d230e29f789bd6c4fed2e2d54357fbec3eca9a49 WatchSource:0}: Error finding container 3af143248dd45aa48c0e70d2d230e29f789bd6c4fed2e2d54357fbec3eca9a49: Status 404 returned error can't find the container with id 3af143248dd45aa48c0e70d2d230e29f789bd6c4fed2e2d54357fbec3eca9a49 Nov 01 21:05:13.354840 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:13.354722 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9ae21-494d5"] Nov 01 21:05:13.375581 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:13.375552 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92642861-003f-49a3-9df0-7a2913316106" path="/var/lib/kubelet/pods/92642861-003f-49a3-9df0-7a2913316106/volumes" Nov 01 21:05:13.776672 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:13.776625 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9ae21-494d5" event={"ID":"8b20f2ca-5b06-4e4e-a5ef-f68b8414f2ac","Type":"ContainerStarted","Data":"ea01f477ba1394ecc23d38e1cc46a4f862614977326f4f219c5069088449f225"} Nov 01 21:05:13.776856 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:13.776695 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9ae21-494d5" event={"ID":"8b20f2ca-5b06-4e4e-a5ef-f68b8414f2ac","Type":"ContainerStarted","Data":"3af143248dd45aa48c0e70d2d230e29f789bd6c4fed2e2d54357fbec3eca9a49"} Nov 01 21:05:13.896393 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:13.896300 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9ae21-494d5" podStartSLOduration=1.8962863570000001 podStartE2EDuration="1.896286357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:05:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:05:13.893293844 +0000 UTC m=+2064.670933979" watchObservedRunningTime="2026-11-01 21:05:13.896286357 +0000 UTC m=+2064.673926492" Nov 01 21:05:14.596202 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:14.596170 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m" Nov 01 21:05:14.682242 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:14.682208 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c818ecc-7536-4e57-93f6-40ba5942d318-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c818ecc-7536-4e57-93f6-40ba5942d318-workload-certs\") pod \"3c818ecc-7536-4e57-93f6-40ba5942d318\" (UID: \"3c818ecc-7536-4e57-93f6-40ba5942d318\") " Nov 01 21:05:14.682452 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:14.682252 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c818ecc-7536-4e57-93f6-40ba5942d318-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c818ecc-7536-4e57-93f6-40ba5942d318-credential-socket\") pod \"3c818ecc-7536-4e57-93f6-40ba5942d318\" (UID: \"3c818ecc-7536-4e57-93f6-40ba5942d318\") " Nov 01 21:05:14.682452 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:14.682281 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c818ecc-7536-4e57-93f6-40ba5942d318-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c818ecc-7536-4e57-93f6-40ba5942d318-istio-data\") pod \"3c818ecc-7536-4e57-93f6-40ba5942d318\" (UID: \"3c818ecc-7536-4e57-93f6-40ba5942d318\") " Nov 01 21:05:14.682452 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:14.682403 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c818ecc-7536-4e57-93f6-40ba5942d318-kube-api-access-r9c4k\" (UniqueName: \"kubernetes.io/projected/3c818ecc-7536-4e57-93f6-40ba5942d318-kube-api-access-r9c4k\") pod \"3c818ecc-7536-4e57-93f6-40ba5942d318\" (UID: \"3c818ecc-7536-4e57-93f6-40ba5942d318\") " Nov 01 21:05:14.682646 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:14.682454 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c818ecc-7536-4e57-93f6-40ba5942d318-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c818ecc-7536-4e57-93f6-40ba5942d318-workload-socket\") pod \"3c818ecc-7536-4e57-93f6-40ba5942d318\" (UID: \"3c818ecc-7536-4e57-93f6-40ba5942d318\") " Nov 01 21:05:14.682646 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:14.682553 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c818ecc-7536-4e57-93f6-40ba5942d318-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c818ecc-7536-4e57-93f6-40ba5942d318-istiod-ca-cert\") pod \"3c818ecc-7536-4e57-93f6-40ba5942d318\" (UID: \"3c818ecc-7536-4e57-93f6-40ba5942d318\") " Nov 01 21:05:14.682646 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:14.682578 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c818ecc-7536-4e57-93f6-40ba5942d318-workload-certs" pod "3c818ecc-7536-4e57-93f6-40ba5942d318" (UID: "3c818ecc-7536-4e57-93f6-40ba5942d318"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:05:14.682646 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:14.682589 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c818ecc-7536-4e57-93f6-40ba5942d318-credential-socket" pod "3c818ecc-7536-4e57-93f6-40ba5942d318" (UID: "3c818ecc-7536-4e57-93f6-40ba5942d318"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:05:14.682646 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:14.682614 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c818ecc-7536-4e57-93f6-40ba5942d318-istio-data" pod "3c818ecc-7536-4e57-93f6-40ba5942d318" (UID: "3c818ecc-7536-4e57-93f6-40ba5942d318"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:05:14.682934 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:14.682700 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c818ecc-7536-4e57-93f6-40ba5942d318-istio-token\" (UniqueName: \"kubernetes.io/projected/3c818ecc-7536-4e57-93f6-40ba5942d318-istio-token\") pod \"3c818ecc-7536-4e57-93f6-40ba5942d318\" (UID: \"3c818ecc-7536-4e57-93f6-40ba5942d318\") " Nov 01 21:05:14.682934 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:14.682705 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c818ecc-7536-4e57-93f6-40ba5942d318-workload-socket" pod "3c818ecc-7536-4e57-93f6-40ba5942d318" (UID: "3c818ecc-7536-4e57-93f6-40ba5942d318"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:05:14.682934 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:14.682858 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c818ecc-7536-4e57-93f6-40ba5942d318-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c818ecc-7536-4e57-93f6-40ba5942d318-istio-envoy\") pod \"3c818ecc-7536-4e57-93f6-40ba5942d318\" (UID: \"3c818ecc-7536-4e57-93f6-40ba5942d318\") " Nov 01 21:05:14.682934 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:14.682883 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c818ecc-7536-4e57-93f6-40ba5942d318-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c818ecc-7536-4e57-93f6-40ba5942d318-istio-podinfo\") pod \"3c818ecc-7536-4e57-93f6-40ba5942d318\" (UID: \"3c818ecc-7536-4e57-93f6-40ba5942d318\") " Nov 01 21:05:14.682934 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:14.682908 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c818ecc-7536-4e57-93f6-40ba5942d318-istiod-ca-cert" pod "3c818ecc-7536-4e57-93f6-40ba5942d318" (UID: "3c818ecc-7536-4e57-93f6-40ba5942d318"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:05:14.683502 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:14.683480 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c818ecc-7536-4e57-93f6-40ba5942d318-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:14.683502 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:14.683503 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c818ecc-7536-4e57-93f6-40ba5942d318-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:14.683679 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:14.683513 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c818ecc-7536-4e57-93f6-40ba5942d318-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:14.683679 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:14.683522 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c818ecc-7536-4e57-93f6-40ba5942d318-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:14.683679 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:14.683531 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c818ecc-7536-4e57-93f6-40ba5942d318-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:14.684707 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:14.684683 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c818ecc-7536-4e57-93f6-40ba5942d318-kube-api-access-r9c4k" pod "3c818ecc-7536-4e57-93f6-40ba5942d318" (UID: "3c818ecc-7536-4e57-93f6-40ba5942d318"). InnerVolumeSpecName "kube-api-access-r9c4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:05:14.684940 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:14.684914 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c818ecc-7536-4e57-93f6-40ba5942d318-istio-podinfo" pod "3c818ecc-7536-4e57-93f6-40ba5942d318" (UID: "3c818ecc-7536-4e57-93f6-40ba5942d318"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:05:14.685180 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:14.685163 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c818ecc-7536-4e57-93f6-40ba5942d318-istio-envoy" pod "3c818ecc-7536-4e57-93f6-40ba5942d318" (UID: "3c818ecc-7536-4e57-93f6-40ba5942d318"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:05:14.685226 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:14.685183 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c818ecc-7536-4e57-93f6-40ba5942d318-istio-token" pod "3c818ecc-7536-4e57-93f6-40ba5942d318" (UID: "3c818ecc-7536-4e57-93f6-40ba5942d318"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:05:14.785847 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:14.785798 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c818ecc-7536-4e57-93f6-40ba5942d318-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:14.785847 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:14.785843 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c818ecc-7536-4e57-93f6-40ba5942d318-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:14.786180 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:14.785859 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9c4k\" (UniqueName: \"kubernetes.io/projected/3c818ecc-7536-4e57-93f6-40ba5942d318-kube-api-access-r9c4k\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:14.786180 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:14.785877 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c818ecc-7536-4e57-93f6-40ba5942d318-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:14.790883 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:14.790844 4484 generic.go:356] "Generic (PLEG): container finished" podID="8b20f2ca-5b06-4e4e-a5ef-f68b8414f2ac" containerID="ea01f477ba1394ecc23d38e1cc46a4f862614977326f4f219c5069088449f225" exitCode=0 Nov 01 21:05:14.791514 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:14.791205 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9ae21-494d5" event={"ID":"8b20f2ca-5b06-4e4e-a5ef-f68b8414f2ac","Type":"ContainerDied","Data":"ea01f477ba1394ecc23d38e1cc46a4f862614977326f4f219c5069088449f225"} Nov 01 21:05:14.793494 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:14.793466 4484 generic.go:356] "Generic (PLEG): container finished" podID="3c818ecc-7536-4e57-93f6-40ba5942d318" containerID="521531af618eb362a74aed2f6b877b3991146bfe2614cb17482b05f74b816b7d" exitCode=0 Nov 01 21:05:14.793632 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:14.793518 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m" Nov 01 21:05:14.793794 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:14.793517 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m" event={"ID":"3c818ecc-7536-4e57-93f6-40ba5942d318","Type":"ContainerDied","Data":"521531af618eb362a74aed2f6b877b3991146bfe2614cb17482b05f74b816b7d"} Nov 01 21:05:14.793952 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:14.793930 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m" event={"ID":"3c818ecc-7536-4e57-93f6-40ba5942d318","Type":"ContainerDied","Data":"95615dbdcc1e42084f8f337d0836d355df22245924453a562c2b173af5c6e884"} Nov 01 21:05:14.794124 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:14.793956 4484 scope.go:122] "RemoveContainer" containerID="521531af618eb362a74aed2f6b877b3991146bfe2614cb17482b05f74b816b7d" Nov 01 21:05:14.808603 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:14.808500 4484 scope.go:122] "RemoveContainer" containerID="e1ec29d393c8e11d13cb254f36e949be173583f41b56f7ffab9cbe9783651dee" Nov 01 21:05:14.822040 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:14.822015 4484 scope.go:122] "RemoveContainer" containerID="521531af618eb362a74aed2f6b877b3991146bfe2614cb17482b05f74b816b7d" Nov 01 21:05:14.822373 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:05:14.822353 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"521531af618eb362a74aed2f6b877b3991146bfe2614cb17482b05f74b816b7d\": container with ID starting with 521531af618eb362a74aed2f6b877b3991146bfe2614cb17482b05f74b816b7d not found: ID does not exist" containerID="521531af618eb362a74aed2f6b877b3991146bfe2614cb17482b05f74b816b7d" Nov 01 21:05:14.822431 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:14.822383 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"521531af618eb362a74aed2f6b877b3991146bfe2614cb17482b05f74b816b7d"} err="failed to get container status \"521531af618eb362a74aed2f6b877b3991146bfe2614cb17482b05f74b816b7d\": rpc error: code = NotFound desc = could not find container \"521531af618eb362a74aed2f6b877b3991146bfe2614cb17482b05f74b816b7d\": container with ID starting with 521531af618eb362a74aed2f6b877b3991146bfe2614cb17482b05f74b816b7d not found: ID does not exist" Nov 01 21:05:14.822431 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:14.822405 4484 scope.go:122] "RemoveContainer" containerID="e1ec29d393c8e11d13cb254f36e949be173583f41b56f7ffab9cbe9783651dee" Nov 01 21:05:14.822707 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:05:14.822689 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1ec29d393c8e11d13cb254f36e949be173583f41b56f7ffab9cbe9783651dee\": container with ID starting with e1ec29d393c8e11d13cb254f36e949be173583f41b56f7ffab9cbe9783651dee not found: ID does not exist" containerID="e1ec29d393c8e11d13cb254f36e949be173583f41b56f7ffab9cbe9783651dee" Nov 01 21:05:14.822769 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:14.822713 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1ec29d393c8e11d13cb254f36e949be173583f41b56f7ffab9cbe9783651dee"} err="failed to get container status \"e1ec29d393c8e11d13cb254f36e949be173583f41b56f7ffab9cbe9783651dee\": rpc error: code = NotFound desc = could not find container \"e1ec29d393c8e11d13cb254f36e949be173583f41b56f7ffab9cbe9783651dee\": container with ID starting with e1ec29d393c8e11d13cb254f36e949be173583f41b56f7ffab9cbe9783651dee not found: ID does not exist" Nov 01 21:05:15.021598 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:15.021559 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m"] Nov 01 21:05:15.027820 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:15.027788 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-081c8e-openshift-default-8cd89ddf-xx79m"] Nov 01 21:05:15.373815 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:15.373781 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c818ecc-7536-4e57-93f6-40ba5942d318" path="/var/lib/kubelet/pods/3c818ecc-7536-4e57-93f6-40ba5942d318/volumes" Nov 01 21:05:15.769111 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:05:15.769077 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:05:15.968459 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:15.968429 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9ae21-494d5" Nov 01 21:05:16.078359 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:16.078277 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg"] Nov 01 21:05:16.103800 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:16.103765 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b20f2ca-5b06-4e4e-a5ef-f68b8414f2ac-kube-api-access-4qbzp\" (UniqueName: \"kubernetes.io/projected/8b20f2ca-5b06-4e4e-a5ef-f68b8414f2ac-kube-api-access-4qbzp\") pod \"8b20f2ca-5b06-4e4e-a5ef-f68b8414f2ac\" (UID: \"8b20f2ca-5b06-4e4e-a5ef-f68b8414f2ac\") " Nov 01 21:05:16.104025 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:16.103833 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b20f2ca-5b06-4e4e-a5ef-f68b8414f2ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b20f2ca-5b06-4e4e-a5ef-f68b8414f2ac-maas-api-xks-ca\") pod \"8b20f2ca-5b06-4e4e-a5ef-f68b8414f2ac\" (UID: \"8b20f2ca-5b06-4e4e-a5ef-f68b8414f2ac\") " Nov 01 21:05:16.104025 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:16.103895 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b20f2ca-5b06-4e4e-a5ef-f68b8414f2ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b20f2ca-5b06-4e4e-a5ef-f68b8414f2ac-maas-api-openshift-ca\") pod \"8b20f2ca-5b06-4e4e-a5ef-f68b8414f2ac\" (UID: \"8b20f2ca-5b06-4e4e-a5ef-f68b8414f2ac\") " Nov 01 21:05:16.104319 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:16.104291 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b20f2ca-5b06-4e4e-a5ef-f68b8414f2ac-maas-api-openshift-ca" pod "8b20f2ca-5b06-4e4e-a5ef-f68b8414f2ac" (UID: "8b20f2ca-5b06-4e4e-a5ef-f68b8414f2ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:05:16.104600 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:16.104573 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b20f2ca-5b06-4e4e-a5ef-f68b8414f2ac-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:16.105944 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:16.105916 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b20f2ca-5b06-4e4e-a5ef-f68b8414f2ac-kube-api-access-4qbzp" pod "8b20f2ca-5b06-4e4e-a5ef-f68b8414f2ac" (UID: "8b20f2ca-5b06-4e4e-a5ef-f68b8414f2ac"). InnerVolumeSpecName "kube-api-access-4qbzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:05:16.106055 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:16.105954 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b20f2ca-5b06-4e4e-a5ef-f68b8414f2ac-maas-api-xks-ca" pod "8b20f2ca-5b06-4e4e-a5ef-f68b8414f2ac" (UID: "8b20f2ca-5b06-4e4e-a5ef-f68b8414f2ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:05:16.184644 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:16.184564 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:16.206127 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:16.206095 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qbzp\" (UniqueName: \"kubernetes.io/projected/8b20f2ca-5b06-4e4e-a5ef-f68b8414f2ac-kube-api-access-4qbzp\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:16.206127 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:16.206122 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b20f2ca-5b06-4e4e-a5ef-f68b8414f2ac-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:16.812301 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:16.812263 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9ae21-494d5" event={"ID":"8b20f2ca-5b06-4e4e-a5ef-f68b8414f2ac","Type":"ContainerDied","Data":"3af143248dd45aa48c0e70d2d230e29f789bd6c4fed2e2d54357fbec3eca9a49"} Nov 01 21:05:16.812701 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:16.812314 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9ae21-494d5" Nov 01 21:05:16.812701 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:16.812339 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3af143248dd45aa48c0e70d2d230e29f789bd6c4fed2e2d54357fbec3eca9a49" Nov 01 21:05:16.830926 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:16.830894 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-11a069b1-fq8q6"] Nov 01 21:05:16.831602 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:16.831588 4484 cpu_manager.go:414] "containerMap: removing container" podUID="8b20f2ca-5b06-4e4e-a5ef-f68b8414f2ac" containerName="revoke-keys" Nov 01 21:05:16.831655 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:16.831604 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b20f2ca-5b06-4e4e-a5ef-f68b8414f2ac" containerName="revoke-keys" Nov 01 21:05:16.831692 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:16.831655 4484 cpu_manager.go:414] "containerMap: removing container" podUID="3c818ecc-7536-4e57-93f6-40ba5942d318" containerName="istio-proxy" Nov 01 21:05:16.831692 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:16.831660 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c818ecc-7536-4e57-93f6-40ba5942d318" containerName="istio-proxy" Nov 01 21:05:16.831692 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:16.831668 4484 cpu_manager.go:414] "containerMap: removing container" podUID="3c818ecc-7536-4e57-93f6-40ba5942d318" containerName="istio-proxy" Nov 01 21:05:16.831692 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:16.831674 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c818ecc-7536-4e57-93f6-40ba5942d318" containerName="istio-proxy" Nov 01 21:05:16.831827 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:16.831777 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c818ecc-7536-4e57-93f6-40ba5942d318" containerName="istio-proxy" Nov 01 21:05:16.831827 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:16.831786 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c818ecc-7536-4e57-93f6-40ba5942d318" containerName="istio-proxy" Nov 01 21:05:16.831827 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:16.831795 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b20f2ca-5b06-4e4e-a5ef-f68b8414f2ac" containerName="revoke-keys" Nov 01 21:05:16.835529 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:16.835512 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-11a069b1-fq8q6" Nov 01 21:05:16.888541 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:16.888509 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:05:16.888759 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:16.888725 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:05:16.888948 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:16.888560 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:05:16.890434 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:16.890407 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-11a069b1-fq8q6"] Nov 01 21:05:16.917975 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:16.917939 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/638b3ec7-e860-4c27-9a43-765f613a7fe7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-11a069b1-fq8q6\" (UID: \"638b3ec7-e860-4c27-9a43-765f613a7fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-11a069b1-fq8q6" Nov 01 21:05:16.918298 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:16.918272 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/638b3ec7-e860-4c27-9a43-765f613a7fe7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-11a069b1-fq8q6\" (UID: \"638b3ec7-e860-4c27-9a43-765f613a7fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-11a069b1-fq8q6" Nov 01 21:05:16.918401 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:16.918386 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn4hj\" (UniqueName: \"kubernetes.io/projected/638b3ec7-e860-4c27-9a43-765f613a7fe7-kube-api-access-vn4hj\") pod \"maas-api-revoke-keys-e2e-mt-11a069b1-fq8q6\" (UID: \"638b3ec7-e860-4c27-9a43-765f613a7fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-11a069b1-fq8q6" Nov 01 21:05:16.925909 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:16.925870 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:05:17.019573 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:17.019540 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/638b3ec7-e860-4c27-9a43-765f613a7fe7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-11a069b1-fq8q6\" (UID: \"638b3ec7-e860-4c27-9a43-765f613a7fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-11a069b1-fq8q6" Nov 01 21:05:17.019737 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:17.019614 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn4hj\" (UniqueName: \"kubernetes.io/projected/638b3ec7-e860-4c27-9a43-765f613a7fe7-kube-api-access-vn4hj\") pod \"maas-api-revoke-keys-e2e-mt-11a069b1-fq8q6\" (UID: \"638b3ec7-e860-4c27-9a43-765f613a7fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-11a069b1-fq8q6" Nov 01 21:05:17.019737 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:17.019658 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/638b3ec7-e860-4c27-9a43-765f613a7fe7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-11a069b1-fq8q6\" (UID: \"638b3ec7-e860-4c27-9a43-765f613a7fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-11a069b1-fq8q6" Nov 01 21:05:17.020285 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:17.020255 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/638b3ec7-e860-4c27-9a43-765f613a7fe7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-11a069b1-fq8q6\" (UID: \"638b3ec7-e860-4c27-9a43-765f613a7fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-11a069b1-fq8q6" Nov 01 21:05:17.021972 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:17.021948 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/638b3ec7-e860-4c27-9a43-765f613a7fe7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-11a069b1-fq8q6\" (UID: \"638b3ec7-e860-4c27-9a43-765f613a7fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-11a069b1-fq8q6" Nov 01 21:05:17.077497 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:17.077425 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn4hj\" (UniqueName: \"kubernetes.io/projected/638b3ec7-e860-4c27-9a43-765f613a7fe7-kube-api-access-vn4hj\") pod \"maas-api-revoke-keys-e2e-mt-11a069b1-fq8q6\" (UID: \"638b3ec7-e860-4c27-9a43-765f613a7fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-11a069b1-fq8q6" Nov 01 21:05:17.154098 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:17.154046 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-11a069b1-fq8q6" Nov 01 21:05:17.187197 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:17.187127 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:17.594503 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:17.594471 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-11a069b1-fq8q6"] Nov 01 21:05:17.594766 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:05:17.594745 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod638b3ec7_e860_4c27_9a43_765f613a7fe7.slice/crio-e6fd6586de612ea53ae5c547b9208584dbdc1d6c4baf73e435bf875d5d243271 WatchSource:0}: Error finding container e6fd6586de612ea53ae5c547b9208584dbdc1d6c4baf73e435bf875d5d243271: Status 404 returned error can't find the container with id e6fd6586de612ea53ae5c547b9208584dbdc1d6c4baf73e435bf875d5d243271 Nov 01 21:05:17.821191 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:17.821160 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-11a069b1-fq8q6" event={"ID":"638b3ec7-e860-4c27-9a43-765f613a7fe7","Type":"ContainerStarted","Data":"e6fd6586de612ea53ae5c547b9208584dbdc1d6c4baf73e435bf875d5d243271"} Nov 01 21:05:18.336126 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:18.336090 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:18.507602 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:18.507564 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:18.620621 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:18.620528 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq"] Nov 01 21:05:18.620789 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:18.620756 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq" podUID="6c3c6fec-0841-4a0c-b761-626d8c53a1cb" containerName="istio-proxy" containerID="cri-o://8d9ae7629a83e488649a1e3de0872ded59eb6ec2a1317076573ca545c7d5abcd" gracePeriod=30 Nov 01 21:05:18.667513 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:18.667469 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:18.897411 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:18.897316 4484 generic.go:356] "Generic (PLEG): container finished" podID="638b3ec7-e860-4c27-9a43-765f613a7fe7" containerID="c492d331e125a38faf3d6f62d30cb040649d013ff87b2b5002ed81e25dab6623" exitCode=0 Nov 01 21:05:18.897882 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:18.897497 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-11a069b1-fq8q6" event={"ID":"638b3ec7-e860-4c27-9a43-765f613a7fe7","Type":"ContainerDied","Data":"c492d331e125a38faf3d6f62d30cb040649d013ff87b2b5002ed81e25dab6623"} Nov 01 21:05:19.390613 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:19.390578 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:19.878508 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:19.878466 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc"] Nov 01 21:05:19.884164 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:19.884121 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc" Nov 01 21:05:19.912020 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:19.898040 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-istio-envoy\") pod \"e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc\" (UID: \"8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff\") " pod="openshift-ingress/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc" Nov 01 21:05:19.912020 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:19.898158 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-workload-certs\") pod \"e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc\" (UID: \"8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff\") " pod="openshift-ingress/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc" Nov 01 21:05:19.912020 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:19.898188 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlngd\" (UniqueName: \"kubernetes.io/projected/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-kube-api-access-wlngd\") pod \"e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc\" (UID: \"8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff\") " pod="openshift-ingress/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc" Nov 01 21:05:19.912020 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:19.898278 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-istio-token\") pod \"e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc\" (UID: \"8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff\") " pod="openshift-ingress/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc" Nov 01 21:05:19.912020 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:19.898340 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-workload-socket\") pod \"e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc\" (UID: \"8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff\") " pod="openshift-ingress/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc" Nov 01 21:05:19.912020 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:19.898376 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-istiod-ca-cert\") pod \"e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc\" (UID: \"8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff\") " pod="openshift-ingress/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc" Nov 01 21:05:19.912020 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:19.898438 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-istio-podinfo\") pod \"e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc\" (UID: \"8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff\") " pod="openshift-ingress/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc" Nov 01 21:05:19.912020 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:19.898484 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-istio-data\") pod \"e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc\" (UID: \"8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff\") " pod="openshift-ingress/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc" Nov 01 21:05:19.912020 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:19.898541 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-credential-socket\") pod \"e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc\" (UID: \"8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff\") " pod="openshift-ingress/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc" Nov 01 21:05:20.035021 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:20.028740 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-e0a6f5-openshift-default-dockercfg-rb7qx\"" Nov 01 21:05:20.206650 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:20.192506 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-istio-envoy\") pod \"e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc\" (UID: \"8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff\") " pod="openshift-ingress/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc" Nov 01 21:05:20.206650 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:20.192598 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-istio-envoy\") pod \"e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc\" (UID: \"8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff\") " pod="openshift-ingress/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc" Nov 01 21:05:20.206650 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:20.192778 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-workload-certs\") pod \"e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc\" (UID: \"8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff\") " pod="openshift-ingress/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc" Nov 01 21:05:20.206650 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:20.192812 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlngd\" (UniqueName: \"kubernetes.io/projected/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-kube-api-access-wlngd\") pod \"e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc\" (UID: \"8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff\") " pod="openshift-ingress/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc" Nov 01 21:05:20.206650 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:20.192918 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-istio-token\") pod \"e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc\" (UID: \"8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff\") " pod="openshift-ingress/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc" Nov 01 21:05:20.206650 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:20.193019 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-workload-socket\") pod \"e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc\" (UID: \"8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff\") " pod="openshift-ingress/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc" Nov 01 21:05:20.206650 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:20.193066 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-istiod-ca-cert\") pod \"e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc\" (UID: \"8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff\") " pod="openshift-ingress/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc" Nov 01 21:05:20.206650 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:20.193159 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-istio-podinfo\") pod \"e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc\" (UID: \"8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff\") " pod="openshift-ingress/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc" Nov 01 21:05:20.206650 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:20.193219 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-istio-data\") pod \"e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc\" (UID: \"8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff\") " pod="openshift-ingress/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc" Nov 01 21:05:20.206650 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:20.193291 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-credential-socket\") pod \"e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc\" (UID: \"8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff\") " pod="openshift-ingress/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc" Nov 01 21:05:20.206650 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:20.194046 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-credential-socket\") pod \"e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc\" (UID: \"8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff\") " pod="openshift-ingress/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc" Nov 01 21:05:20.206650 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:20.194520 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-workload-certs\") pod \"e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc\" (UID: \"8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff\") " pod="openshift-ingress/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc" Nov 01 21:05:20.224054 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:20.223446 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-workload-socket\") pod \"e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc\" (UID: \"8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff\") " pod="openshift-ingress/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc" Nov 01 21:05:20.226046 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:20.224523 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-istiod-ca-cert\") pod \"e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc\" (UID: \"8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff\") " pod="openshift-ingress/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc" Nov 01 21:05:20.240392 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:20.227939 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-istio-podinfo\") pod \"e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc\" (UID: \"8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff\") " pod="openshift-ingress/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc" Nov 01 21:05:20.240392 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:20.228339 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-istio-data\") pod \"e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc\" (UID: \"8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff\") " pod="openshift-ingress/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc" Nov 01 21:05:20.240392 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:20.231333 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:20.240392 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:20.236142 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc"] Nov 01 21:05:20.288512 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:20.288345 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlngd\" (UniqueName: \"kubernetes.io/projected/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-kube-api-access-wlngd\") pod \"e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc\" (UID: \"8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff\") " pod="openshift-ingress/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc" Nov 01 21:05:20.289334 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:20.289304 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-istio-token\") pod \"e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc\" (UID: \"8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff\") " pod="openshift-ingress/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc" Nov 01 21:05:20.369827 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:20.369803 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-11a069b1-fq8q6" Nov 01 21:05:20.463124 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:20.460843 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:20.497773 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:20.497741 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/638b3ec7-e860-4c27-9a43-765f613a7fe7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/638b3ec7-e860-4c27-9a43-765f613a7fe7-maas-api-xks-ca\") pod \"638b3ec7-e860-4c27-9a43-765f613a7fe7\" (UID: \"638b3ec7-e860-4c27-9a43-765f613a7fe7\") " Nov 01 21:05:20.497939 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:20.497928 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/638b3ec7-e860-4c27-9a43-765f613a7fe7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/638b3ec7-e860-4c27-9a43-765f613a7fe7-maas-api-openshift-ca\") pod \"638b3ec7-e860-4c27-9a43-765f613a7fe7\" (UID: \"638b3ec7-e860-4c27-9a43-765f613a7fe7\") " Nov 01 21:05:20.498028 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:20.498013 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/638b3ec7-e860-4c27-9a43-765f613a7fe7-kube-api-access-vn4hj\" (UniqueName: \"kubernetes.io/projected/638b3ec7-e860-4c27-9a43-765f613a7fe7-kube-api-access-vn4hj\") pod \"638b3ec7-e860-4c27-9a43-765f613a7fe7\" (UID: \"638b3ec7-e860-4c27-9a43-765f613a7fe7\") " Nov 01 21:05:20.498328 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:20.498288 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/638b3ec7-e860-4c27-9a43-765f613a7fe7-maas-api-openshift-ca" pod "638b3ec7-e860-4c27-9a43-765f613a7fe7" (UID: "638b3ec7-e860-4c27-9a43-765f613a7fe7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:05:20.498840 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:20.498805 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/638b3ec7-e860-4c27-9a43-765f613a7fe7-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:20.500538 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:20.500515 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/638b3ec7-e860-4c27-9a43-765f613a7fe7-maas-api-xks-ca" pod "638b3ec7-e860-4c27-9a43-765f613a7fe7" (UID: "638b3ec7-e860-4c27-9a43-765f613a7fe7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:05:20.500646 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:20.500534 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/638b3ec7-e860-4c27-9a43-765f613a7fe7-kube-api-access-vn4hj" pod "638b3ec7-e860-4c27-9a43-765f613a7fe7" (UID: "638b3ec7-e860-4c27-9a43-765f613a7fe7"). InnerVolumeSpecName "kube-api-access-vn4hj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:05:20.551901 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:20.551861 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc" Nov 01 21:05:20.600114 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:20.600071 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vn4hj\" (UniqueName: \"kubernetes.io/projected/638b3ec7-e860-4c27-9a43-765f613a7fe7-kube-api-access-vn4hj\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:20.600362 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:20.600338 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/638b3ec7-e860-4c27-9a43-765f613a7fe7-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:20.938646 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:20.938614 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-11a069b1-fq8q6" Nov 01 21:05:20.938646 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:20.938643 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-11a069b1-fq8q6" event={"ID":"638b3ec7-e860-4c27-9a43-765f613a7fe7","Type":"ContainerDied","Data":"e6fd6586de612ea53ae5c547b9208584dbdc1d6c4baf73e435bf875d5d243271"} Nov 01 21:05:20.939098 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:20.938688 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6fd6586de612ea53ae5c547b9208584dbdc1d6c4baf73e435bf875d5d243271" Nov 01 21:05:20.986575 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:05:20.986544 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ef2ec69_fb02_409c_94cb_4e1cfa9d06ff.slice/crio-c4697355f311056e7b68aef7b018752058a79def0a72c449a1cbb61d8571093f WatchSource:0}: Error finding container c4697355f311056e7b68aef7b018752058a79def0a72c449a1cbb61d8571093f: Status 404 returned error can't find the container with id c4697355f311056e7b68aef7b018752058a79def0a72c449a1cbb61d8571093f Nov 01 21:05:20.986693 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:20.986616 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc"] Nov 01 21:05:20.989527 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:20.989493 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:05:20.989633 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:20.989561 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:05:20.989633 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:20.989588 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:05:21.948737 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:21.948697 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc" event={"ID":"8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff","Type":"ContainerStarted","Data":"093879cd3f9e7ce1ed44b391516fca5207ccb2ff263dfbb60a9a50f436e9d87c"} Nov 01 21:05:21.949186 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:21.948760 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc" event={"ID":"8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff","Type":"ContainerStarted","Data":"c4697355f311056e7b68aef7b018752058a79def0a72c449a1cbb61d8571093f"} Nov 01 21:05:22.119604 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:22.119550 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc" podStartSLOduration=3.11953334 podStartE2EDuration="3.11953334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:05:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:05:22.117117016 +0000 UTC m=+2072.894757356" watchObservedRunningTime="2026-11-01 21:05:22.11953334 +0000 UTC m=+2072.897173475" Nov 01 21:05:22.437434 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:22.437399 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:22.552798 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:22.552764 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc" Nov 01 21:05:22.558470 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:22.558443 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc" Nov 01 21:05:22.956742 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:22.956712 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc" Nov 01 21:05:22.957852 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:22.957831 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc" Nov 01 21:05:23.914859 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:23.914837 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq" Nov 01 21:05:23.965257 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:23.965224 4484 generic.go:356] "Generic (PLEG): container finished" podID="6c3c6fec-0841-4a0c-b761-626d8c53a1cb" containerID="8d9ae7629a83e488649a1e3de0872ded59eb6ec2a1317076573ca545c7d5abcd" exitCode=0 Nov 01 21:05:23.965732 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:23.965292 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq" Nov 01 21:05:23.965732 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:23.965319 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq" event={"ID":"6c3c6fec-0841-4a0c-b761-626d8c53a1cb","Type":"ContainerDied","Data":"8d9ae7629a83e488649a1e3de0872ded59eb6ec2a1317076573ca545c7d5abcd"} Nov 01 21:05:23.965732 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:23.965384 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq" event={"ID":"6c3c6fec-0841-4a0c-b761-626d8c53a1cb","Type":"ContainerDied","Data":"d6f5356e786c5ba83b5215c9fcb613b3ff089d2b2912b975bb5a078a47ff6361"} Nov 01 21:05:23.965732 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:23.965423 4484 scope.go:122] "RemoveContainer" containerID="8d9ae7629a83e488649a1e3de0872ded59eb6ec2a1317076573ca545c7d5abcd" Nov 01 21:05:23.976587 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:23.976568 4484 scope.go:122] "RemoveContainer" containerID="a5d91c1d8490cb6b27129606cf5d3c39f95dedafea0ed794a5cd959283a6e608" Nov 01 21:05:23.986818 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:23.986798 4484 scope.go:122] "RemoveContainer" containerID="8d9ae7629a83e488649a1e3de0872ded59eb6ec2a1317076573ca545c7d5abcd" Nov 01 21:05:23.987254 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:05:23.987232 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d9ae7629a83e488649a1e3de0872ded59eb6ec2a1317076573ca545c7d5abcd\": container with ID starting with 8d9ae7629a83e488649a1e3de0872ded59eb6ec2a1317076573ca545c7d5abcd not found: ID does not exist" containerID="8d9ae7629a83e488649a1e3de0872ded59eb6ec2a1317076573ca545c7d5abcd" Nov 01 21:05:23.987340 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:23.987263 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d9ae7629a83e488649a1e3de0872ded59eb6ec2a1317076573ca545c7d5abcd"} err="failed to get container status \"8d9ae7629a83e488649a1e3de0872ded59eb6ec2a1317076573ca545c7d5abcd\": rpc error: code = NotFound desc = could not find container \"8d9ae7629a83e488649a1e3de0872ded59eb6ec2a1317076573ca545c7d5abcd\": container with ID starting with 8d9ae7629a83e488649a1e3de0872ded59eb6ec2a1317076573ca545c7d5abcd not found: ID does not exist" Nov 01 21:05:23.987340 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:23.987281 4484 scope.go:122] "RemoveContainer" containerID="a5d91c1d8490cb6b27129606cf5d3c39f95dedafea0ed794a5cd959283a6e608" Nov 01 21:05:23.987554 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:05:23.987524 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5d91c1d8490cb6b27129606cf5d3c39f95dedafea0ed794a5cd959283a6e608\": container with ID starting with a5d91c1d8490cb6b27129606cf5d3c39f95dedafea0ed794a5cd959283a6e608 not found: ID does not exist" containerID="a5d91c1d8490cb6b27129606cf5d3c39f95dedafea0ed794a5cd959283a6e608" Nov 01 21:05:23.987622 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:23.987565 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5d91c1d8490cb6b27129606cf5d3c39f95dedafea0ed794a5cd959283a6e608"} err="failed to get container status \"a5d91c1d8490cb6b27129606cf5d3c39f95dedafea0ed794a5cd959283a6e608\": rpc error: code = NotFound desc = could not find container \"a5d91c1d8490cb6b27129606cf5d3c39f95dedafea0ed794a5cd959283a6e608\": container with ID starting with a5d91c1d8490cb6b27129606cf5d3c39f95dedafea0ed794a5cd959283a6e608 not found: ID does not exist" Nov 01 21:05:24.051550 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:24.051516 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-workload-certs\") pod \"6c3c6fec-0841-4a0c-b761-626d8c53a1cb\" (UID: \"6c3c6fec-0841-4a0c-b761-626d8c53a1cb\") " Nov 01 21:05:24.051727 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:24.051565 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-istio-token\" (UniqueName: \"kubernetes.io/projected/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-istio-token\") pod \"6c3c6fec-0841-4a0c-b761-626d8c53a1cb\" (UID: \"6c3c6fec-0841-4a0c-b761-626d8c53a1cb\") " Nov 01 21:05:24.051727 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:24.051637 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-credential-socket\") pod \"6c3c6fec-0841-4a0c-b761-626d8c53a1cb\" (UID: \"6c3c6fec-0841-4a0c-b761-626d8c53a1cb\") " Nov 01 21:05:24.051727 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:24.051673 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-istio-data\") pod \"6c3c6fec-0841-4a0c-b761-626d8c53a1cb\" (UID: \"6c3c6fec-0841-4a0c-b761-626d8c53a1cb\") " Nov 01 21:05:24.051923 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:24.051778 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-istio-envoy\") pod \"6c3c6fec-0841-4a0c-b761-626d8c53a1cb\" (UID: \"6c3c6fec-0841-4a0c-b761-626d8c53a1cb\") " Nov 01 21:05:24.051923 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:24.051809 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-workload-certs" pod "6c3c6fec-0841-4a0c-b761-626d8c53a1cb" (UID: "6c3c6fec-0841-4a0c-b761-626d8c53a1cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:05:24.051923 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:24.051835 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-workload-socket\") pod \"6c3c6fec-0841-4a0c-b761-626d8c53a1cb\" (UID: \"6c3c6fec-0841-4a0c-b761-626d8c53a1cb\") " Nov 01 21:05:24.052195 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:24.052172 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-workload-socket" pod "6c3c6fec-0841-4a0c-b761-626d8c53a1cb" (UID: "6c3c6fec-0841-4a0c-b761-626d8c53a1cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:05:24.052195 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:24.052187 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-credential-socket" pod "6c3c6fec-0841-4a0c-b761-626d8c53a1cb" (UID: "6c3c6fec-0841-4a0c-b761-626d8c53a1cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:05:24.052310 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:24.052236 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-istio-data" pod "6c3c6fec-0841-4a0c-b761-626d8c53a1cb" (UID: "6c3c6fec-0841-4a0c-b761-626d8c53a1cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:05:24.052370 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:24.052327 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-kube-api-access-xb8bm\" (UniqueName: \"kubernetes.io/projected/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-kube-api-access-xb8bm\") pod \"6c3c6fec-0841-4a0c-b761-626d8c53a1cb\" (UID: \"6c3c6fec-0841-4a0c-b761-626d8c53a1cb\") " Nov 01 21:05:24.052435 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:24.052411 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-istiod-ca-cert\") pod \"6c3c6fec-0841-4a0c-b761-626d8c53a1cb\" (UID: \"6c3c6fec-0841-4a0c-b761-626d8c53a1cb\") " Nov 01 21:05:24.052491 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:24.052455 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-istio-podinfo\") pod \"6c3c6fec-0841-4a0c-b761-626d8c53a1cb\" (UID: \"6c3c6fec-0841-4a0c-b761-626d8c53a1cb\") " Nov 01 21:05:24.052822 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:24.052796 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-istiod-ca-cert" pod "6c3c6fec-0841-4a0c-b761-626d8c53a1cb" (UID: "6c3c6fec-0841-4a0c-b761-626d8c53a1cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:05:24.053482 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:24.053454 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:24.053587 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:24.053484 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:24.053587 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:24.053500 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:24.053587 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:24.053514 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:24.053587 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:24.053528 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:24.054185 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:24.054164 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-istio-token" pod "6c3c6fec-0841-4a0c-b761-626d8c53a1cb" (UID: "6c3c6fec-0841-4a0c-b761-626d8c53a1cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:05:24.054834 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:24.054802 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-istio-envoy" pod "6c3c6fec-0841-4a0c-b761-626d8c53a1cb" (UID: "6c3c6fec-0841-4a0c-b761-626d8c53a1cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:05:24.055909 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:24.055883 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-kube-api-access-xb8bm" pod "6c3c6fec-0841-4a0c-b761-626d8c53a1cb" (UID: "6c3c6fec-0841-4a0c-b761-626d8c53a1cb"). InnerVolumeSpecName "kube-api-access-xb8bm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:05:24.056115 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:24.056091 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-istio-podinfo" pod "6c3c6fec-0841-4a0c-b761-626d8c53a1cb" (UID: "6c3c6fec-0841-4a0c-b761-626d8c53a1cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:05:24.126329 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:24.126293 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-c9ae21-6667d54cf5-twgq6"] Nov 01 21:05:24.155649 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:24.155564 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:24.155649 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:24.155596 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:24.155649 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:24.155606 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xb8bm\" (UniqueName: \"kubernetes.io/projected/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-kube-api-access-xb8bm\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:24.155649 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:24.155616 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c3c6fec-0841-4a0c-b761-626d8c53a1cb-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:24.242331 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:24.242291 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-c9ae21-7bc765fb9f-7tjsn"] Nov 01 21:05:24.476189 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:24.476100 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq"] Nov 01 21:05:24.482945 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:24.482914 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-dab736-openshift-default-5d5d779c5f-tcsxq"] Nov 01 21:05:24.517851 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:24.517813 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:25.374925 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:25.374895 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c3c6fec-0841-4a0c-b761-626d8c53a1cb" path="/var/lib/kubelet/pods/6c3c6fec-0841-4a0c-b761-626d8c53a1cb/volumes" Nov 01 21:05:27.062175 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:27.046903 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:27.749089 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:27.749057 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:28.185037 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:28.184902 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:28.355684 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:28.355641 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:28.553077 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:28.553043 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e0a6f5-68b97cb76c-sm88c"] Nov 01 21:05:28.554166 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:28.554147 4484 cpu_manager.go:414] "containerMap: removing container" podUID="638b3ec7-e860-4c27-9a43-765f613a7fe7" containerName="revoke-keys" Nov 01 21:05:28.554242 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:28.554169 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="638b3ec7-e860-4c27-9a43-765f613a7fe7" containerName="revoke-keys" Nov 01 21:05:28.554242 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:28.554193 4484 cpu_manager.go:414] "containerMap: removing container" podUID="6c3c6fec-0841-4a0c-b761-626d8c53a1cb" containerName="istio-proxy" Nov 01 21:05:28.554242 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:28.554201 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c3c6fec-0841-4a0c-b761-626d8c53a1cb" containerName="istio-proxy" Nov 01 21:05:28.554242 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:28.554219 4484 cpu_manager.go:414] "containerMap: removing container" podUID="6c3c6fec-0841-4a0c-b761-626d8c53a1cb" containerName="istio-proxy" Nov 01 21:05:28.554242 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:28.554226 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c3c6fec-0841-4a0c-b761-626d8c53a1cb" containerName="istio-proxy" Nov 01 21:05:28.554415 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:28.554404 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="638b3ec7-e860-4c27-9a43-765f613a7fe7" containerName="revoke-keys" Nov 01 21:05:28.554448 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:28.554421 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c3c6fec-0841-4a0c-b761-626d8c53a1cb" containerName="istio-proxy" Nov 01 21:05:28.554448 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:28.554433 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c3c6fec-0841-4a0c-b761-626d8c53a1cb" containerName="istio-proxy" Nov 01 21:05:28.560574 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:28.560552 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e0a6f5-68b97cb76c-sm88c" Nov 01 21:05:28.614150 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:28.614118 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e0a6f5-68b97cb76c-sm88c"] Nov 01 21:05:28.724020 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:28.711440 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs5fr\" (UniqueName: \"kubernetes.io/projected/a007bf8c-6aab-4b78-87eb-3bbd63998634-kube-api-access-cs5fr\") pod \"payload-pre-processing-e2e-api-a-e0a6f5-68b97cb76c-sm88c\" (UID: \"a007bf8c-6aab-4b78-87eb-3bbd63998634\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e0a6f5-68b97cb76c-sm88c" Nov 01 21:05:28.724020 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:28.711610 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a007bf8c-6aab-4b78-87eb-3bbd63998634-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e0a6f5-68b97cb76c-sm88c\" (UID: \"a007bf8c-6aab-4b78-87eb-3bbd63998634\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e0a6f5-68b97cb76c-sm88c" Nov 01 21:05:28.755109 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:28.755013 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-e0a6f5\"" Nov 01 21:05:28.755547 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:28.755526 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-e0a6f5-dockercfg-x8xk6\"" Nov 01 21:05:28.838334 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:28.837128 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs5fr\" (UniqueName: \"kubernetes.io/projected/a007bf8c-6aab-4b78-87eb-3bbd63998634-kube-api-access-cs5fr\") pod \"payload-pre-processing-e2e-api-a-e0a6f5-68b97cb76c-sm88c\" (UID: \"a007bf8c-6aab-4b78-87eb-3bbd63998634\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e0a6f5-68b97cb76c-sm88c" Nov 01 21:05:28.838334 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:28.837284 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a007bf8c-6aab-4b78-87eb-3bbd63998634-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e0a6f5-68b97cb76c-sm88c\" (UID: \"a007bf8c-6aab-4b78-87eb-3bbd63998634\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e0a6f5-68b97cb76c-sm88c" Nov 01 21:05:28.838334 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:28.838226 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a007bf8c-6aab-4b78-87eb-3bbd63998634-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e0a6f5-68b97cb76c-sm88c\" (UID: \"a007bf8c-6aab-4b78-87eb-3bbd63998634\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e0a6f5-68b97cb76c-sm88c" Nov 01 21:05:28.881921 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:28.870637 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e0a6f5-56bbd4f6f7-5gtlm"] Nov 01 21:05:28.881921 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:28.879990 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e0a6f5-56bbd4f6f7-5gtlm"] Nov 01 21:05:28.881921 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:28.880147 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e0a6f5-56bbd4f6f7-5gtlm" Nov 01 21:05:28.897079 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:28.897038 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs5fr\" (UniqueName: \"kubernetes.io/projected/a007bf8c-6aab-4b78-87eb-3bbd63998634-kube-api-access-cs5fr\") pod \"payload-pre-processing-e2e-api-a-e0a6f5-68b97cb76c-sm88c\" (UID: \"a007bf8c-6aab-4b78-87eb-3bbd63998634\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e0a6f5-68b97cb76c-sm88c" Nov 01 21:05:29.039351 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:29.039311 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8m5g\" (UniqueName: \"kubernetes.io/projected/6c9c289a-bbe6-4ef2-a11d-e87f0bf77480-kube-api-access-f8m5g\") pod \"payload-processing-e2e-api-a-e0a6f5-56bbd4f6f7-5gtlm\" (UID: \"6c9c289a-bbe6-4ef2-a11d-e87f0bf77480\") " pod="openshift-ingress/payload-processing-e2e-api-a-e0a6f5-56bbd4f6f7-5gtlm" Nov 01 21:05:29.039553 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:29.039363 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c9c289a-bbe6-4ef2-a11d-e87f0bf77480-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e0a6f5-56bbd4f6f7-5gtlm\" (UID: \"6c9c289a-bbe6-4ef2-a11d-e87f0bf77480\") " pod="openshift-ingress/payload-processing-e2e-api-a-e0a6f5-56bbd4f6f7-5gtlm" Nov 01 21:05:29.127331 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:29.127213 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-c9ae21-6667d54cf5-twgq6" podUID="824b88e5-a583-4489-9562-846fb877553d" containerName="payload-processing" containerID="cri-o://69879df6e5a6ec6e02d154f04409e49cdd92d7f0183c21a22ba09a3df32601a6" gracePeriod=25 Nov 01 21:05:29.141063 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:29.141028 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8m5g\" (UniqueName: \"kubernetes.io/projected/6c9c289a-bbe6-4ef2-a11d-e87f0bf77480-kube-api-access-f8m5g\") pod \"payload-processing-e2e-api-a-e0a6f5-56bbd4f6f7-5gtlm\" (UID: \"6c9c289a-bbe6-4ef2-a11d-e87f0bf77480\") " pod="openshift-ingress/payload-processing-e2e-api-a-e0a6f5-56bbd4f6f7-5gtlm" Nov 01 21:05:29.141223 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:29.141089 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c9c289a-bbe6-4ef2-a11d-e87f0bf77480-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e0a6f5-56bbd4f6f7-5gtlm\" (UID: \"6c9c289a-bbe6-4ef2-a11d-e87f0bf77480\") " pod="openshift-ingress/payload-processing-e2e-api-a-e0a6f5-56bbd4f6f7-5gtlm" Nov 01 21:05:29.141835 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:29.141809 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c9c289a-bbe6-4ef2-a11d-e87f0bf77480-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e0a6f5-56bbd4f6f7-5gtlm\" (UID: \"6c9c289a-bbe6-4ef2-a11d-e87f0bf77480\") " pod="openshift-ingress/payload-processing-e2e-api-a-e0a6f5-56bbd4f6f7-5gtlm" Nov 01 21:05:29.176226 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:29.176188 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e0a6f5-68b97cb76c-sm88c" Nov 01 21:05:29.198849 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:29.198819 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8m5g\" (UniqueName: \"kubernetes.io/projected/6c9c289a-bbe6-4ef2-a11d-e87f0bf77480-kube-api-access-f8m5g\") pod \"payload-processing-e2e-api-a-e0a6f5-56bbd4f6f7-5gtlm\" (UID: \"6c9c289a-bbe6-4ef2-a11d-e87f0bf77480\") " pod="openshift-ingress/payload-processing-e2e-api-a-e0a6f5-56bbd4f6f7-5gtlm" Nov 01 21:05:29.243390 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:29.243340 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-c9ae21-7bc765fb9f-7tjsn" podUID="7d955afb-8551-4657-9daa-c70b309757a6" containerName="payload-pre-processing" containerID="cri-o://8598b20a8496f3d79d63ab7ca83c0b69a78470e99fee48cc983fbd4f761cce7c" gracePeriod=25 Nov 01 21:05:29.497916 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:29.497882 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e0a6f5-56bbd4f6f7-5gtlm" Nov 01 21:05:29.554011 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:05:29.553967 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda007bf8c_6aab_4b78_87eb_3bbd63998634.slice/crio-71e92f302e0b0180b25871ddcdb105db6578c8e46954cdb7970c66b6a2e0755b WatchSource:0}: Error finding container 71e92f302e0b0180b25871ddcdb105db6578c8e46954cdb7970c66b6a2e0755b: Status 404 returned error can't find the container with id 71e92f302e0b0180b25871ddcdb105db6578c8e46954cdb7970c66b6a2e0755b Nov 01 21:05:29.556418 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:29.556393 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e0a6f5-68b97cb76c-sm88c"] Nov 01 21:05:29.669262 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:29.669228 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:29.850051 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:29.845807 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:29.904810 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:29.904782 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e0a6f5-56bbd4f6f7-5gtlm"] Nov 01 21:05:29.904978 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:05:29.904952 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c9c289a_bbe6_4ef2_a11d_e87f0bf77480.slice/crio-ab5ca9655fac3228cc480b0ebe6e63a5f80b36b22630b15028e22f168bcd9cda WatchSource:0}: Error finding container ab5ca9655fac3228cc480b0ebe6e63a5f80b36b22630b15028e22f168bcd9cda: Status 404 returned error can't find the container with id ab5ca9655fac3228cc480b0ebe6e63a5f80b36b22630b15028e22f168bcd9cda Nov 01 21:05:30.069882 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:30.069786 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e0a6f5-68b97cb76c-sm88c" event={"ID":"a007bf8c-6aab-4b78-87eb-3bbd63998634","Type":"ContainerStarted","Data":"d83793fddda42fd98e1a8a16a6b71e745cf300461144ae7582a42c4680b3fe13"} Nov 01 21:05:30.069882 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:30.069857 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e0a6f5-68b97cb76c-sm88c" event={"ID":"a007bf8c-6aab-4b78-87eb-3bbd63998634","Type":"ContainerStarted","Data":"71e92f302e0b0180b25871ddcdb105db6578c8e46954cdb7970c66b6a2e0755b"} Nov 01 21:05:30.070134 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:30.069892 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e0a6f5-68b97cb76c-sm88c" Nov 01 21:05:30.071584 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:30.071555 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e0a6f5-56bbd4f6f7-5gtlm" event={"ID":"6c9c289a-bbe6-4ef2-a11d-e87f0bf77480","Type":"ContainerStarted","Data":"d5df541a80bf0a13bf7e56fcee990f1f157ff469a0a25f14e468a9c0ca089688"} Nov 01 21:05:30.071699 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:30.071606 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e0a6f5-56bbd4f6f7-5gtlm" event={"ID":"6c9c289a-bbe6-4ef2-a11d-e87f0bf77480","Type":"ContainerStarted","Data":"ab5ca9655fac3228cc480b0ebe6e63a5f80b36b22630b15028e22f168bcd9cda"} Nov 01 21:05:30.071699 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:30.071626 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-e0a6f5-56bbd4f6f7-5gtlm" Nov 01 21:05:30.185253 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:30.185199 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e0a6f5-68b97cb76c-sm88c" podStartSLOduration=2.185184074 podStartE2EDuration="2.185184074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:05:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:05:30.18440103 +0000 UTC m=+2080.962041166" watchObservedRunningTime="2026-11-01 21:05:30.185184074 +0000 UTC m=+2080.962824208" Nov 01 21:05:30.311171 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:30.311102 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-e0a6f5-56bbd4f6f7-5gtlm" podStartSLOduration=2.3110781129999998 podStartE2EDuration="2.311078113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:05:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:05:30.308147515 +0000 UTC m=+2081.085787651" watchObservedRunningTime="2026-11-01 21:05:30.311078113 +0000 UTC m=+2081.088718256" Nov 01 21:05:31.130214 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:31.130170 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" podUID="17a9ac24-19cb-48c4-99b5-d74ce8aac0a1" containerName="main" containerID="cri-o://640d3d99214489846e68eedfa38ef3f6ef234f4d1c14624c85c2cf6feaee34a7" gracePeriod=45 Nov 01 21:05:31.135507 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:31.135469 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" podUID="17a9ac24-19cb-48c4-99b5-d74ce8aac0a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": dial tcp 10.134.0.53:8000: connect: connection refused" Nov 01 21:05:31.427392 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:31.427362 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" Nov 01 21:05:31.535388 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:31.535355 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17a9ac24-19cb-48c4-99b5-d74ce8aac0a1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17a9ac24-19cb-48c4-99b5-d74ce8aac0a1-tmp-dir\") pod \"17a9ac24-19cb-48c4-99b5-d74ce8aac0a1\" (UID: \"17a9ac24-19cb-48c4-99b5-d74ce8aac0a1\") " Nov 01 21:05:31.535559 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:31.535415 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17a9ac24-19cb-48c4-99b5-d74ce8aac0a1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17a9ac24-19cb-48c4-99b5-d74ce8aac0a1-model-cache\") pod \"17a9ac24-19cb-48c4-99b5-d74ce8aac0a1\" (UID: \"17a9ac24-19cb-48c4-99b5-d74ce8aac0a1\") " Nov 01 21:05:31.535700 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:31.535685 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17a9ac24-19cb-48c4-99b5-d74ce8aac0a1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/17a9ac24-19cb-48c4-99b5-d74ce8aac0a1-dshm\") pod \"17a9ac24-19cb-48c4-99b5-d74ce8aac0a1\" (UID: \"17a9ac24-19cb-48c4-99b5-d74ce8aac0a1\") " Nov 01 21:05:31.535758 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:31.535688 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17a9ac24-19cb-48c4-99b5-d74ce8aac0a1-tmp-dir" pod "17a9ac24-19cb-48c4-99b5-d74ce8aac0a1" (UID: "17a9ac24-19cb-48c4-99b5-d74ce8aac0a1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:05:31.535834 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:31.535756 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17a9ac24-19cb-48c4-99b5-d74ce8aac0a1-kube-api-access-jvttj\" (UniqueName: \"kubernetes.io/projected/17a9ac24-19cb-48c4-99b5-d74ce8aac0a1-kube-api-access-jvttj\") pod \"17a9ac24-19cb-48c4-99b5-d74ce8aac0a1\" (UID: \"17a9ac24-19cb-48c4-99b5-d74ce8aac0a1\") " Nov 01 21:05:31.535834 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:31.535824 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17a9ac24-19cb-48c4-99b5-d74ce8aac0a1-tls-certs\" (UniqueName: \"kubernetes.io/secret/17a9ac24-19cb-48c4-99b5-d74ce8aac0a1-tls-certs\") pod \"17a9ac24-19cb-48c4-99b5-d74ce8aac0a1\" (UID: \"17a9ac24-19cb-48c4-99b5-d74ce8aac0a1\") " Nov 01 21:05:31.536044 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:31.535859 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17a9ac24-19cb-48c4-99b5-d74ce8aac0a1-home\" (UniqueName: \"kubernetes.io/empty-dir/17a9ac24-19cb-48c4-99b5-d74ce8aac0a1-home\") pod \"17a9ac24-19cb-48c4-99b5-d74ce8aac0a1\" (UID: \"17a9ac24-19cb-48c4-99b5-d74ce8aac0a1\") " Nov 01 21:05:31.536044 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:31.535898 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17a9ac24-19cb-48c4-99b5-d74ce8aac0a1-model-cache" pod "17a9ac24-19cb-48c4-99b5-d74ce8aac0a1" (UID: "17a9ac24-19cb-48c4-99b5-d74ce8aac0a1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:05:31.536445 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:31.536410 4484 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17a9ac24-19cb-48c4-99b5-d74ce8aac0a1-tmp-dir\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:31.536445 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:31.536437 4484 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17a9ac24-19cb-48c4-99b5-d74ce8aac0a1-model-cache\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:31.536660 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:31.536641 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17a9ac24-19cb-48c4-99b5-d74ce8aac0a1-home" pod "17a9ac24-19cb-48c4-99b5-d74ce8aac0a1" (UID: "17a9ac24-19cb-48c4-99b5-d74ce8aac0a1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:05:31.538818 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:31.538783 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17a9ac24-19cb-48c4-99b5-d74ce8aac0a1-kube-api-access-jvttj" pod "17a9ac24-19cb-48c4-99b5-d74ce8aac0a1" (UID: "17a9ac24-19cb-48c4-99b5-d74ce8aac0a1"). InnerVolumeSpecName "kube-api-access-jvttj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:05:31.538818 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:31.538794 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17a9ac24-19cb-48c4-99b5-d74ce8aac0a1-tls-certs" pod "17a9ac24-19cb-48c4-99b5-d74ce8aac0a1" (UID: "17a9ac24-19cb-48c4-99b5-d74ce8aac0a1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:05:31.541862 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:31.541839 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17a9ac24-19cb-48c4-99b5-d74ce8aac0a1-dshm" pod "17a9ac24-19cb-48c4-99b5-d74ce8aac0a1" (UID: "17a9ac24-19cb-48c4-99b5-d74ce8aac0a1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:05:31.602054 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:31.601986 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-c9ae21-7bc765fb9f-7tjsn" podUID="7d955afb-8551-4657-9daa-c70b309757a6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 21:05:31.602252 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:31.602102 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-c9ae21-6667d54cf5-twgq6" podUID="824b88e5-a583-4489-9562-846fb877553d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 21:05:31.638108 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:31.638072 4484 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17a9ac24-19cb-48c4-99b5-d74ce8aac0a1-dshm\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:31.638312 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:31.638135 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvttj\" (UniqueName: \"kubernetes.io/projected/17a9ac24-19cb-48c4-99b5-d74ce8aac0a1-kube-api-access-jvttj\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:31.638312 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:31.638153 4484 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17a9ac24-19cb-48c4-99b5-d74ce8aac0a1-tls-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:31.638312 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:31.638167 4484 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17a9ac24-19cb-48c4-99b5-d74ce8aac0a1-home\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:32.113927 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:32.113889 4484 generic.go:356] "Generic (PLEG): container finished" podID="17a9ac24-19cb-48c4-99b5-d74ce8aac0a1" containerID="640d3d99214489846e68eedfa38ef3f6ef234f4d1c14624c85c2cf6feaee34a7" exitCode=0 Nov 01 21:05:32.114127 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:32.113962 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" Nov 01 21:05:32.114127 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:32.113970 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" event={"ID":"17a9ac24-19cb-48c4-99b5-d74ce8aac0a1","Type":"ContainerDied","Data":"640d3d99214489846e68eedfa38ef3f6ef234f4d1c14624c85c2cf6feaee34a7"} Nov 01 21:05:32.114127 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:32.114046 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg" event={"ID":"17a9ac24-19cb-48c4-99b5-d74ce8aac0a1","Type":"ContainerDied","Data":"5152d213b52aef5d73cc2fe3ed76f96a80bcc2558a4a4a9a534b8b349f44c4cf"} Nov 01 21:05:32.114127 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:32.114067 4484 scope.go:122] "RemoveContainer" containerID="640d3d99214489846e68eedfa38ef3f6ef234f4d1c14624c85c2cf6feaee34a7" Nov 01 21:05:32.127772 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:32.127754 4484 scope.go:122] "RemoveContainer" containerID="640d3d99214489846e68eedfa38ef3f6ef234f4d1c14624c85c2cf6feaee34a7" Nov 01 21:05:32.128115 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:05:32.128093 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"640d3d99214489846e68eedfa38ef3f6ef234f4d1c14624c85c2cf6feaee34a7\": container with ID starting with 640d3d99214489846e68eedfa38ef3f6ef234f4d1c14624c85c2cf6feaee34a7 not found: ID does not exist" containerID="640d3d99214489846e68eedfa38ef3f6ef234f4d1c14624c85c2cf6feaee34a7" Nov 01 21:05:32.128210 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:32.128127 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"640d3d99214489846e68eedfa38ef3f6ef234f4d1c14624c85c2cf6feaee34a7"} err="failed to get container status \"640d3d99214489846e68eedfa38ef3f6ef234f4d1c14624c85c2cf6feaee34a7\": rpc error: code = NotFound desc = could not find container \"640d3d99214489846e68eedfa38ef3f6ef234f4d1c14624c85c2cf6feaee34a7\": container with ID starting with 640d3d99214489846e68eedfa38ef3f6ef234f4d1c14624c85c2cf6feaee34a7 not found: ID does not exist" Nov 01 21:05:32.281293 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:32.281262 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg"] Nov 01 21:05:32.331871 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:32.331842 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-c9ae21/simulator-w5-c9ae21-kserve-7b8b74bc97-tx7kg"] Nov 01 21:05:32.820463 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:32.820422 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:33.373957 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:33.373925 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17a9ac24-19cb-48c4-99b5-d74ce8aac0a1" path="/var/lib/kubelet/pods/17a9ac24-19cb-48c4-99b5-d74ce8aac0a1/volumes" Nov 01 21:05:36.049348 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:36.049313 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9a7e213-42tzt"] Nov 01 21:05:36.099483 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:36.099449 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9a7e213-42tzt"] Nov 01 21:05:37.373597 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:37.373566 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="752bf457-c713-429d-a6d2-9550a30c3efd" path="/var/lib/kubelet/pods/752bf457-c713-429d-a6d2-9550a30c3efd/volumes" Nov 01 21:05:37.856956 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:37.856924 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:41.134375 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:41.134347 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-e0a6f5-56bbd4f6f7-5gtlm" Nov 01 21:05:41.134802 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:41.134430 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e0a6f5-68b97cb76c-sm88c" Nov 01 21:05:41.603177 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:41.603128 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-c9ae21-7bc765fb9f-7tjsn" podUID="7d955afb-8551-4657-9daa-c70b309757a6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 21:05:41.603366 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:41.603150 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-c9ae21-6667d54cf5-twgq6" podUID="824b88e5-a583-4489-9562-846fb877553d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 21:05:42.913312 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:42.913263 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:44.840544 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:44.840507 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:45.767857 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:05:45.767821 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:05:48.830976 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:48.830941 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:49.117373 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:49.117277 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:49.247232 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:49.247205 4484 generic.go:356] "Generic (PLEG): container finished" podID="824b88e5-a583-4489-9562-846fb877553d" containerID="69879df6e5a6ec6e02d154f04409e49cdd92d7f0183c21a22ba09a3df32601a6" exitCode=0 Nov 01 21:05:49.247349 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:49.247276 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-c9ae21-6667d54cf5-twgq6" event={"ID":"824b88e5-a583-4489-9562-846fb877553d","Type":"ContainerDied","Data":"69879df6e5a6ec6e02d154f04409e49cdd92d7f0183c21a22ba09a3df32601a6"} Nov 01 21:05:49.274315 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:49.274270 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:49.309187 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:49.309145 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd840a-4t6zl"] Nov 01 21:05:49.314476 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:49.314446 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd840a-4t6zl"] Nov 01 21:05:49.376846 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:49.376812 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d85edeb-2371-4625-97b4-b41cba9a30d5" path="/var/lib/kubelet/pods/4d85edeb-2371-4625-97b4-b41cba9a30d5/volumes" Nov 01 21:05:49.419783 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:49.419754 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-c9ae21-6667d54cf5-twgq6" Nov 01 21:05:49.501569 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:49.501534 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/824b88e5-a583-4489-9562-846fb877553d-kube-api-access-ckd7p\" (UniqueName: \"kubernetes.io/projected/824b88e5-a583-4489-9562-846fb877553d-kube-api-access-ckd7p\") pod \"824b88e5-a583-4489-9562-846fb877553d\" (UID: \"824b88e5-a583-4489-9562-846fb877553d\") " Nov 01 21:05:49.501754 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:49.501717 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/824b88e5-a583-4489-9562-846fb877553d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/824b88e5-a583-4489-9562-846fb877553d-plugins-config-volume\") pod \"824b88e5-a583-4489-9562-846fb877553d\" (UID: \"824b88e5-a583-4489-9562-846fb877553d\") " Nov 01 21:05:49.502374 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:49.502344 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/824b88e5-a583-4489-9562-846fb877553d-plugins-config-volume" pod "824b88e5-a583-4489-9562-846fb877553d" (UID: "824b88e5-a583-4489-9562-846fb877553d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:05:49.502505 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:49.502477 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/824b88e5-a583-4489-9562-846fb877553d-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:49.504588 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:49.504558 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/824b88e5-a583-4489-9562-846fb877553d-kube-api-access-ckd7p" pod "824b88e5-a583-4489-9562-846fb877553d" (UID: "824b88e5-a583-4489-9562-846fb877553d"). InnerVolumeSpecName "kube-api-access-ckd7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:05:49.533836 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:49.533813 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-c9ae21-7bc765fb9f-7tjsn" Nov 01 21:05:49.603291 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:49.603261 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d955afb-8551-4657-9daa-c70b309757a6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d955afb-8551-4657-9daa-c70b309757a6-plugins-config-volume\") pod \"7d955afb-8551-4657-9daa-c70b309757a6\" (UID: \"7d955afb-8551-4657-9daa-c70b309757a6\") " Nov 01 21:05:49.603476 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:49.603461 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d955afb-8551-4657-9daa-c70b309757a6-kube-api-access-lctpx\" (UniqueName: \"kubernetes.io/projected/7d955afb-8551-4657-9daa-c70b309757a6-kube-api-access-lctpx\") pod \"7d955afb-8551-4657-9daa-c70b309757a6\" (UID: \"7d955afb-8551-4657-9daa-c70b309757a6\") " Nov 01 21:05:49.603668 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:49.603642 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d955afb-8551-4657-9daa-c70b309757a6-plugins-config-volume" pod "7d955afb-8551-4657-9daa-c70b309757a6" (UID: "7d955afb-8551-4657-9daa-c70b309757a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:05:49.603933 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:49.603918 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d955afb-8551-4657-9daa-c70b309757a6-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:49.603978 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:49.603936 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckd7p\" (UniqueName: \"kubernetes.io/projected/824b88e5-a583-4489-9562-846fb877553d-kube-api-access-ckd7p\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:49.605514 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:49.605496 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d955afb-8551-4657-9daa-c70b309757a6-kube-api-access-lctpx" pod "7d955afb-8551-4657-9daa-c70b309757a6" (UID: "7d955afb-8551-4657-9daa-c70b309757a6"). InnerVolumeSpecName "kube-api-access-lctpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:05:49.706317 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:49.706237 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lctpx\" (UniqueName: \"kubernetes.io/projected/7d955afb-8551-4657-9daa-c70b309757a6-kube-api-access-lctpx\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:49.723669 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:49.723627 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:50.256269 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:50.256237 4484 generic.go:356] "Generic (PLEG): container finished" podID="7d955afb-8551-4657-9daa-c70b309757a6" containerID="8598b20a8496f3d79d63ab7ca83c0b69a78470e99fee48cc983fbd4f761cce7c" exitCode=0 Nov 01 21:05:50.256749 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:50.256279 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-c9ae21-7bc765fb9f-7tjsn" Nov 01 21:05:50.256749 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:50.256329 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-c9ae21-7bc765fb9f-7tjsn" event={"ID":"7d955afb-8551-4657-9daa-c70b309757a6","Type":"ContainerDied","Data":"8598b20a8496f3d79d63ab7ca83c0b69a78470e99fee48cc983fbd4f761cce7c"} Nov 01 21:05:50.256749 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:50.256387 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-c9ae21-7bc765fb9f-7tjsn" event={"ID":"7d955afb-8551-4657-9daa-c70b309757a6","Type":"ContainerDied","Data":"a7459c0b77a6d25996f67ed2e2b0f4e2e145f95c3a49a5c2ffd25b151a167773"} Nov 01 21:05:50.256749 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:50.256406 4484 scope.go:122] "RemoveContainer" containerID="8598b20a8496f3d79d63ab7ca83c0b69a78470e99fee48cc983fbd4f761cce7c" Nov 01 21:05:50.257911 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:50.257891 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-c9ae21-6667d54cf5-twgq6" Nov 01 21:05:50.257911 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:50.257899 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-c9ae21-6667d54cf5-twgq6" event={"ID":"824b88e5-a583-4489-9562-846fb877553d","Type":"ContainerDied","Data":"2b0daa3cab291b4e6ee4d095cbfc6ba1be24ed529fceca6d01dcbc7698194289"} Nov 01 21:05:50.267816 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:50.267792 4484 scope.go:122] "RemoveContainer" containerID="8598b20a8496f3d79d63ab7ca83c0b69a78470e99fee48cc983fbd4f761cce7c" Nov 01 21:05:50.277304 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:05:50.277277 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8598b20a8496f3d79d63ab7ca83c0b69a78470e99fee48cc983fbd4f761cce7c\": container with ID starting with 8598b20a8496f3d79d63ab7ca83c0b69a78470e99fee48cc983fbd4f761cce7c not found: ID does not exist" containerID="8598b20a8496f3d79d63ab7ca83c0b69a78470e99fee48cc983fbd4f761cce7c" Nov 01 21:05:50.277437 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:50.277314 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8598b20a8496f3d79d63ab7ca83c0b69a78470e99fee48cc983fbd4f761cce7c"} err="failed to get container status \"8598b20a8496f3d79d63ab7ca83c0b69a78470e99fee48cc983fbd4f761cce7c\": rpc error: code = NotFound desc = could not find container \"8598b20a8496f3d79d63ab7ca83c0b69a78470e99fee48cc983fbd4f761cce7c\": container with ID starting with 8598b20a8496f3d79d63ab7ca83c0b69a78470e99fee48cc983fbd4f761cce7c not found: ID does not exist" Nov 01 21:05:50.277437 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:50.277339 4484 scope.go:122] "RemoveContainer" containerID="69879df6e5a6ec6e02d154f04409e49cdd92d7f0183c21a22ba09a3df32601a6" Nov 01 21:05:50.424677 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:50.424595 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-c9ae21-6667d54cf5-twgq6"] Nov 01 21:05:50.475316 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:50.475288 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-c9ae21-6667d54cf5-twgq6"] Nov 01 21:05:50.538320 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:50.538278 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-c9ae21-7bc765fb9f-7tjsn"] Nov 01 21:05:50.543034 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:50.542963 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-c9ae21-7bc765fb9f-7tjsn"] Nov 01 21:05:51.374446 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:51.374411 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d955afb-8551-4657-9daa-c70b309757a6" path="/var/lib/kubelet/pods/7d955afb-8551-4657-9daa-c70b309757a6/volumes" Nov 01 21:05:51.374903 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:51.374889 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="824b88e5-a583-4489-9562-846fb877553d" path="/var/lib/kubelet/pods/824b88e5-a583-4489-9562-846fb877553d/volumes" Nov 01 21:05:52.739272 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:52.739238 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:52.902004 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:52.901957 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:53.725601 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:53.725565 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:54.335064 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:54.335028 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:55.665322 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:55.665286 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:55.891469 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:55.891426 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:56.062536 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:56.062497 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:57.975317 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:57.975283 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:58.694025 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:58.679449 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:59.338855 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:59.338817 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:59.511210 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:59.511173 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:59.999094 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:05:59.999059 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:00.166958 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:00.166921 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:03.295209 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:03.295113 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:03.462251 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:03.462214 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:03.634255 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:03.634170 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:04.341802 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:04.341764 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:06.529201 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:06.529165 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:06.749981 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:06.749935 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:07.104018 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:07.103329 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4ac1ef-5bddb89485-dpbt4"] Nov 01 21:06:07.117020 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:07.104360 4484 cpu_manager.go:414] "containerMap: removing container" podUID="824b88e5-a583-4489-9562-846fb877553d" containerName="payload-processing" Nov 01 21:06:07.117020 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:07.104385 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="824b88e5-a583-4489-9562-846fb877553d" containerName="payload-processing" Nov 01 21:06:07.117020 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:07.104441 4484 cpu_manager.go:414] "containerMap: removing container" podUID="7d955afb-8551-4657-9daa-c70b309757a6" containerName="payload-pre-processing" Nov 01 21:06:07.117020 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:07.104450 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d955afb-8551-4657-9daa-c70b309757a6" containerName="payload-pre-processing" Nov 01 21:06:07.117020 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:07.104478 4484 cpu_manager.go:414] "containerMap: removing container" podUID="17a9ac24-19cb-48c4-99b5-d74ce8aac0a1" containerName="main" Nov 01 21:06:07.117020 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:07.104485 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17a9ac24-19cb-48c4-99b5-d74ce8aac0a1" containerName="main" Nov 01 21:06:07.117020 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:07.104639 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="17a9ac24-19cb-48c4-99b5-d74ce8aac0a1" containerName="main" Nov 01 21:06:07.117020 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:07.104653 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="824b88e5-a583-4489-9562-846fb877553d" containerName="payload-processing" Nov 01 21:06:07.117020 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:07.104672 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d955afb-8551-4657-9daa-c70b309757a6" containerName="payload-pre-processing" Nov 01 21:06:07.139279 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:07.139243 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4ac1ef-5bddb89485-dpbt4" Nov 01 21:06:07.210434 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:07.210390 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4ac1ef-5bddb89485-dpbt4"] Nov 01 21:06:07.245674 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:07.238402 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nprmb\" (UniqueName: \"kubernetes.io/projected/74f275b8-4451-479f-9a16-99cb725261b3-kube-api-access-nprmb\") pod \"payload-processing-e2e-inf-a-4ac1ef-5bddb89485-dpbt4\" (UID: \"74f275b8-4451-479f-9a16-99cb725261b3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4ac1ef-5bddb89485-dpbt4" Nov 01 21:06:07.248013 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:07.247966 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74f275b8-4451-479f-9a16-99cb725261b3-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4ac1ef-5bddb89485-dpbt4\" (UID: \"74f275b8-4451-479f-9a16-99cb725261b3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4ac1ef-5bddb89485-dpbt4" Nov 01 21:06:07.285612 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:07.285579 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-4ac1ef\"" Nov 01 21:06:07.286018 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:07.285978 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-4ac1ef-dockercfg-wqd72\"" Nov 01 21:06:07.294064 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:07.294028 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4ac1ef-db4769dc-ffkcl"] Nov 01 21:06:07.301539 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:07.301511 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4ac1ef-db4769dc-ffkcl" Nov 01 21:06:07.304988 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:07.304956 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4ac1ef-db4769dc-ffkcl"] Nov 01 21:06:07.350274 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:07.350222 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dndhm\" (UniqueName: \"kubernetes.io/projected/c5429352-860e-44aa-a7c0-2ad3dcdf9267-kube-api-access-dndhm\") pod \"payload-pre-processing-e2e-inf-a-4ac1ef-db4769dc-ffkcl\" (UID: \"c5429352-860e-44aa-a7c0-2ad3dcdf9267\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4ac1ef-db4769dc-ffkcl" Nov 01 21:06:07.350474 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:07.350374 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5429352-860e-44aa-a7c0-2ad3dcdf9267-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4ac1ef-db4769dc-ffkcl\" (UID: \"c5429352-860e-44aa-a7c0-2ad3dcdf9267\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4ac1ef-db4769dc-ffkcl" Nov 01 21:06:07.350627 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:07.350599 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nprmb\" (UniqueName: \"kubernetes.io/projected/74f275b8-4451-479f-9a16-99cb725261b3-kube-api-access-nprmb\") pod \"payload-processing-e2e-inf-a-4ac1ef-5bddb89485-dpbt4\" (UID: \"74f275b8-4451-479f-9a16-99cb725261b3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4ac1ef-5bddb89485-dpbt4" Nov 01 21:06:07.350685 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:07.350649 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74f275b8-4451-479f-9a16-99cb725261b3-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4ac1ef-5bddb89485-dpbt4\" (UID: \"74f275b8-4451-479f-9a16-99cb725261b3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4ac1ef-5bddb89485-dpbt4" Nov 01 21:06:07.351429 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:07.351407 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74f275b8-4451-479f-9a16-99cb725261b3-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4ac1ef-5bddb89485-dpbt4\" (UID: \"74f275b8-4451-479f-9a16-99cb725261b3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4ac1ef-5bddb89485-dpbt4" Nov 01 21:06:07.408842 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:07.408761 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nprmb\" (UniqueName: \"kubernetes.io/projected/74f275b8-4451-479f-9a16-99cb725261b3-kube-api-access-nprmb\") pod \"payload-processing-e2e-inf-a-4ac1ef-5bddb89485-dpbt4\" (UID: \"74f275b8-4451-479f-9a16-99cb725261b3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4ac1ef-5bddb89485-dpbt4" Nov 01 21:06:07.451739 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:07.451696 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dndhm\" (UniqueName: \"kubernetes.io/projected/c5429352-860e-44aa-a7c0-2ad3dcdf9267-kube-api-access-dndhm\") pod \"payload-pre-processing-e2e-inf-a-4ac1ef-db4769dc-ffkcl\" (UID: \"c5429352-860e-44aa-a7c0-2ad3dcdf9267\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4ac1ef-db4769dc-ffkcl" Nov 01 21:06:07.451952 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:07.451810 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5429352-860e-44aa-a7c0-2ad3dcdf9267-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4ac1ef-db4769dc-ffkcl\" (UID: \"c5429352-860e-44aa-a7c0-2ad3dcdf9267\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4ac1ef-db4769dc-ffkcl" Nov 01 21:06:07.452645 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:07.452620 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5429352-860e-44aa-a7c0-2ad3dcdf9267-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4ac1ef-db4769dc-ffkcl\" (UID: \"c5429352-860e-44aa-a7c0-2ad3dcdf9267\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4ac1ef-db4769dc-ffkcl" Nov 01 21:06:07.509855 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:07.509824 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dndhm\" (UniqueName: \"kubernetes.io/projected/c5429352-860e-44aa-a7c0-2ad3dcdf9267-kube-api-access-dndhm\") pod \"payload-pre-processing-e2e-inf-a-4ac1ef-db4769dc-ffkcl\" (UID: \"c5429352-860e-44aa-a7c0-2ad3dcdf9267\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4ac1ef-db4769dc-ffkcl" Nov 01 21:06:07.571188 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:07.571158 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4ac1ef-5bddb89485-dpbt4" Nov 01 21:06:07.620617 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:07.620582 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4ac1ef-db4769dc-ffkcl" Nov 01 21:06:07.953875 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:07.953846 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4ac1ef-5bddb89485-dpbt4"] Nov 01 21:06:07.954030 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:06:07.953933 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74f275b8_4451_479f_9a16_99cb725261b3.slice/crio-4fffdcaba6b10a137f591d815dc582c12f91eb85dffb5a6b000e039e9cb73949 WatchSource:0}: Error finding container 4fffdcaba6b10a137f591d815dc582c12f91eb85dffb5a6b000e039e9cb73949: Status 404 returned error can't find the container with id 4fffdcaba6b10a137f591d815dc582c12f91eb85dffb5a6b000e039e9cb73949 Nov 01 21:06:08.062986 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:06:08.062942 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5429352_860e_44aa_a7c0_2ad3dcdf9267.slice/crio-dbff33f5f1caaafc8033e7d1ae5b2c68d6a8ec051942f88c94008891ec32f93d WatchSource:0}: Error finding container dbff33f5f1caaafc8033e7d1ae5b2c68d6a8ec051942f88c94008891ec32f93d: Status 404 returned error can't find the container with id dbff33f5f1caaafc8033e7d1ae5b2c68d6a8ec051942f88c94008891ec32f93d Nov 01 21:06:08.063186 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:08.063165 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4ac1ef-db4769dc-ffkcl"] Nov 01 21:06:08.447874 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:08.447827 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4ac1ef-5bddb89485-dpbt4" event={"ID":"74f275b8-4451-479f-9a16-99cb725261b3","Type":"ContainerStarted","Data":"3936314ce0f913719d75613f969434fefbce2855f29d5087890f7375e6a68f6d"} Nov 01 21:06:08.448057 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:08.447891 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4ac1ef-5bddb89485-dpbt4" event={"ID":"74f275b8-4451-479f-9a16-99cb725261b3","Type":"ContainerStarted","Data":"4fffdcaba6b10a137f591d815dc582c12f91eb85dffb5a6b000e039e9cb73949"} Nov 01 21:06:08.448057 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:08.447927 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-4ac1ef-5bddb89485-dpbt4" Nov 01 21:06:08.449550 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:08.449520 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4ac1ef-db4769dc-ffkcl" event={"ID":"c5429352-860e-44aa-a7c0-2ad3dcdf9267","Type":"ContainerStarted","Data":"819f27af16c160ec2178c1679cf290305cf397680cbe51a9d2ddb88dc806c669"} Nov 01 21:06:08.449667 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:08.449560 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4ac1ef-db4769dc-ffkcl" event={"ID":"c5429352-860e-44aa-a7c0-2ad3dcdf9267","Type":"ContainerStarted","Data":"dbff33f5f1caaafc8033e7d1ae5b2c68d6a8ec051942f88c94008891ec32f93d"} Nov 01 21:06:08.449667 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:08.449594 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4ac1ef-db4769dc-ffkcl" Nov 01 21:06:08.565501 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:08.565439 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-4ac1ef-5bddb89485-dpbt4" podStartSLOduration=1.565424589 podStartE2EDuration="1.565424589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:06:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:06:08.563148508 +0000 UTC m=+2119.340788644" watchObservedRunningTime="2026-11-01 21:06:08.565424589 +0000 UTC m=+2119.343064760" Nov 01 21:06:08.715073 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:08.714391 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4ac1ef-db4769dc-ffkcl" podStartSLOduration=1.7143670439999998 podStartE2EDuration="1.714367044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:06:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:06:08.698120351 +0000 UTC m=+2119.475760486" watchObservedRunningTime="2026-11-01 21:06:08.714367044 +0000 UTC m=+2119.492007181" Nov 01 21:06:08.822543 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:08.822510 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:10.399489 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:10.399453 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:10.561645 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:10.561611 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:10.725417 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:10.725327 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:10.829232 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:10.829200 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-29baa8-v55fq"] Nov 01 21:06:10.880200 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:10.880166 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-29baa8-v55fq"] Nov 01 21:06:11.246512 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:11.246477 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:11.375240 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:11.375205 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c1812a5-2e95-4ca5-8c91-b6734bbc348e" path="/var/lib/kubelet/pods/2c1812a5-2e95-4ca5-8c91-b6734bbc348e/volumes" Nov 01 21:06:15.103238 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:15.103201 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:15.767362 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:06:15.767324 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:06:16.652091 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:16.651991 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:16.852523 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:16.852482 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:17.006626 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:17.006583 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:17.037300 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:17.037263 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq"] Nov 01 21:06:17.037513 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:17.037476 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq" podUID="50906dd8-8d76-4fcb-8415-f8f32cd47ffc" containerName="istio-proxy" containerID="cri-o://7129fcd315cf108e1128365007816b2d9879a2a0ee5c21d8a4794338bc86c9ee" gracePeriod=30 Nov 01 21:06:17.154690 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:17.154643 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-22ed7b-tkt8f"] Nov 01 21:06:17.159057 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:17.158981 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-22ed7b-tkt8f"] Nov 01 21:06:17.373086 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:17.372985 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa2af658-15d9-48a7-8f65-36f359fee14c" path="/var/lib/kubelet/pods/fa2af658-15d9-48a7-8f65-36f359fee14c/volumes" Nov 01 21:06:17.479867 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:17.479827 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5"] Nov 01 21:06:17.480615 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:17.480582 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5" podUID="ba0637f3-6fc9-4052-9808-ffc561da21ca" containerName="istio-proxy" containerID="cri-o://364e65f65e27277e3131247864654cd2e076a2ebeb6e6f655f30ba7f54da675c" gracePeriod=30 Nov 01 21:06:17.495719 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:17.495682 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:18.352911 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:18.352875 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:18.517018 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:18.516016 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:18.653411 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:18.646508 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h"] Nov 01 21:06:18.653411 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:18.647107 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h" podUID="c2fa50d2-8fa0-4997-9fdf-354318d94561" containerName="istio-proxy" containerID="cri-o://7b41bb1933550373133d88f2be5aef3a2731f762e951e4c83f680ced0208eb22" gracePeriod=30 Nov 01 21:06:18.681261 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:18.681223 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:19.460007 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:19.459961 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-4ac1ef-5bddb89485-dpbt4" Nov 01 21:06:19.460571 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:19.460550 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4ac1ef-db4769dc-ffkcl" Nov 01 21:06:19.477552 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:19.477518 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:19.937961 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:19.937921 4484 patch_prober.go:29] interesting pod/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 21:06:19.938149 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:19.937984 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq" podUID="50906dd8-8d76-4fcb-8415-f8f32cd47ffc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 21:06:20.133785 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:20.133745 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:20.541526 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:20.541489 4484 patch_prober.go:29] interesting pod/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 21:06:20.541925 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:20.541561 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5" podUID="ba0637f3-6fc9-4052-9808-ffc561da21ca" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 21:06:20.618446 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:20.618414 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:21.934739 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:21.934704 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:22.317372 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.317346 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq" Nov 01 21:06:22.474806 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.474716 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-istio-podinfo\") pod \"50906dd8-8d76-4fcb-8415-f8f32cd47ffc\" (UID: \"50906dd8-8d76-4fcb-8415-f8f32cd47ffc\") " Nov 01 21:06:22.474806 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.474776 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-workload-socket\") pod \"50906dd8-8d76-4fcb-8415-f8f32cd47ffc\" (UID: \"50906dd8-8d76-4fcb-8415-f8f32cd47ffc\") " Nov 01 21:06:22.475024 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.474818 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-credential-socket\") pod \"50906dd8-8d76-4fcb-8415-f8f32cd47ffc\" (UID: \"50906dd8-8d76-4fcb-8415-f8f32cd47ffc\") " Nov 01 21:06:22.475024 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.474855 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-workload-certs\") pod \"50906dd8-8d76-4fcb-8415-f8f32cd47ffc\" (UID: \"50906dd8-8d76-4fcb-8415-f8f32cd47ffc\") " Nov 01 21:06:22.475024 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.474958 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-kube-api-access-nv9mq\" (UniqueName: \"kubernetes.io/projected/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-kube-api-access-nv9mq\") pod \"50906dd8-8d76-4fcb-8415-f8f32cd47ffc\" (UID: \"50906dd8-8d76-4fcb-8415-f8f32cd47ffc\") " Nov 01 21:06:22.475188 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.475067 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-istio-data\") pod \"50906dd8-8d76-4fcb-8415-f8f32cd47ffc\" (UID: \"50906dd8-8d76-4fcb-8415-f8f32cd47ffc\") " Nov 01 21:06:22.475188 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.475104 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-istio-envoy\") pod \"50906dd8-8d76-4fcb-8415-f8f32cd47ffc\" (UID: \"50906dd8-8d76-4fcb-8415-f8f32cd47ffc\") " Nov 01 21:06:22.475309 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.475201 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-credential-socket" pod "50906dd8-8d76-4fcb-8415-f8f32cd47ffc" (UID: "50906dd8-8d76-4fcb-8415-f8f32cd47ffc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:06:22.475309 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.475207 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-workload-socket" pod "50906dd8-8d76-4fcb-8415-f8f32cd47ffc" (UID: "50906dd8-8d76-4fcb-8415-f8f32cd47ffc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:06:22.475309 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.475233 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-istio-token\" (UniqueName: \"kubernetes.io/projected/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-istio-token\") pod \"50906dd8-8d76-4fcb-8415-f8f32cd47ffc\" (UID: \"50906dd8-8d76-4fcb-8415-f8f32cd47ffc\") " Nov 01 21:06:22.475309 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.475274 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-workload-certs" pod "50906dd8-8d76-4fcb-8415-f8f32cd47ffc" (UID: "50906dd8-8d76-4fcb-8415-f8f32cd47ffc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:06:22.475309 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.475292 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-istiod-ca-cert\") pod \"50906dd8-8d76-4fcb-8415-f8f32cd47ffc\" (UID: \"50906dd8-8d76-4fcb-8415-f8f32cd47ffc\") " Nov 01 21:06:22.475598 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.475306 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-istio-data" pod "50906dd8-8d76-4fcb-8415-f8f32cd47ffc" (UID: "50906dd8-8d76-4fcb-8415-f8f32cd47ffc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:06:22.475667 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.475591 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-istiod-ca-cert" pod "50906dd8-8d76-4fcb-8415-f8f32cd47ffc" (UID: "50906dd8-8d76-4fcb-8415-f8f32cd47ffc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:06:22.476193 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.476167 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:22.476343 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.476196 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:22.476343 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.476211 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:22.476343 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.476224 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:22.476343 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.476238 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:22.477358 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.477333 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-istio-podinfo" pod "50906dd8-8d76-4fcb-8415-f8f32cd47ffc" (UID: "50906dd8-8d76-4fcb-8415-f8f32cd47ffc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:06:22.477463 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.477409 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-istio-envoy" pod "50906dd8-8d76-4fcb-8415-f8f32cd47ffc" (UID: "50906dd8-8d76-4fcb-8415-f8f32cd47ffc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:06:22.477535 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.477472 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-istio-token" pod "50906dd8-8d76-4fcb-8415-f8f32cd47ffc" (UID: "50906dd8-8d76-4fcb-8415-f8f32cd47ffc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:06:22.477535 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.477517 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-kube-api-access-nv9mq" pod "50906dd8-8d76-4fcb-8415-f8f32cd47ffc" (UID: "50906dd8-8d76-4fcb-8415-f8f32cd47ffc"). InnerVolumeSpecName "kube-api-access-nv9mq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:06:22.570807 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.570773 4484 generic.go:356] "Generic (PLEG): container finished" podID="50906dd8-8d76-4fcb-8415-f8f32cd47ffc" containerID="7129fcd315cf108e1128365007816b2d9879a2a0ee5c21d8a4794338bc86c9ee" exitCode=0 Nov 01 21:06:22.570976 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.570822 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq" Nov 01 21:06:22.570976 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.570870 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq" event={"ID":"50906dd8-8d76-4fcb-8415-f8f32cd47ffc","Type":"ContainerDied","Data":"7129fcd315cf108e1128365007816b2d9879a2a0ee5c21d8a4794338bc86c9ee"} Nov 01 21:06:22.570976 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.570925 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq" event={"ID":"50906dd8-8d76-4fcb-8415-f8f32cd47ffc","Type":"ContainerDied","Data":"933787bfbcd161ad0a94fd956a8729306dd6b6ae5e8523e40f4e85aec6a16a16"} Nov 01 21:06:22.570976 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.570964 4484 scope.go:122] "RemoveContainer" containerID="7129fcd315cf108e1128365007816b2d9879a2a0ee5c21d8a4794338bc86c9ee" Nov 01 21:06:22.577118 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.577095 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:22.577221 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.577121 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:22.577221 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.577135 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:22.577221 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.577150 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv9mq\" (UniqueName: \"kubernetes.io/projected/50906dd8-8d76-4fcb-8415-f8f32cd47ffc-kube-api-access-nv9mq\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:22.595057 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.595039 4484 scope.go:122] "RemoveContainer" containerID="7129fcd315cf108e1128365007816b2d9879a2a0ee5c21d8a4794338bc86c9ee" Nov 01 21:06:22.595338 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:06:22.595317 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7129fcd315cf108e1128365007816b2d9879a2a0ee5c21d8a4794338bc86c9ee\": container with ID starting with 7129fcd315cf108e1128365007816b2d9879a2a0ee5c21d8a4794338bc86c9ee not found: ID does not exist" containerID="7129fcd315cf108e1128365007816b2d9879a2a0ee5c21d8a4794338bc86c9ee" Nov 01 21:06:22.595417 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.595345 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7129fcd315cf108e1128365007816b2d9879a2a0ee5c21d8a4794338bc86c9ee"} err="failed to get container status \"7129fcd315cf108e1128365007816b2d9879a2a0ee5c21d8a4794338bc86c9ee\": rpc error: code = NotFound desc = could not find container \"7129fcd315cf108e1128365007816b2d9879a2a0ee5c21d8a4794338bc86c9ee\": container with ID starting with 7129fcd315cf108e1128365007816b2d9879a2a0ee5c21d8a4794338bc86c9ee not found: ID does not exist" Nov 01 21:06:22.742094 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.742063 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq"] Nov 01 21:06:22.745921 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.745894 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-9nlfq"] Nov 01 21:06:22.746035 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.745923 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5" Nov 01 21:06:22.883895 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.883863 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba0637f3-6fc9-4052-9808-ffc561da21ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba0637f3-6fc9-4052-9808-ffc561da21ca-istio-envoy\") pod \"ba0637f3-6fc9-4052-9808-ffc561da21ca\" (UID: \"ba0637f3-6fc9-4052-9808-ffc561da21ca\") " Nov 01 21:06:22.884121 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.884013 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba0637f3-6fc9-4052-9808-ffc561da21ca-istio-token\" (UniqueName: \"kubernetes.io/projected/ba0637f3-6fc9-4052-9808-ffc561da21ca-istio-token\") pod \"ba0637f3-6fc9-4052-9808-ffc561da21ca\" (UID: \"ba0637f3-6fc9-4052-9808-ffc561da21ca\") " Nov 01 21:06:22.884121 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.884077 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba0637f3-6fc9-4052-9808-ffc561da21ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba0637f3-6fc9-4052-9808-ffc561da21ca-workload-certs\") pod \"ba0637f3-6fc9-4052-9808-ffc561da21ca\" (UID: \"ba0637f3-6fc9-4052-9808-ffc561da21ca\") " Nov 01 21:06:22.884266 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.884146 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba0637f3-6fc9-4052-9808-ffc561da21ca-kube-api-access-qgdq5\" (UniqueName: \"kubernetes.io/projected/ba0637f3-6fc9-4052-9808-ffc561da21ca-kube-api-access-qgdq5\") pod \"ba0637f3-6fc9-4052-9808-ffc561da21ca\" (UID: \"ba0637f3-6fc9-4052-9808-ffc561da21ca\") " Nov 01 21:06:22.884266 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.884181 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba0637f3-6fc9-4052-9808-ffc561da21ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba0637f3-6fc9-4052-9808-ffc561da21ca-istiod-ca-cert\") pod \"ba0637f3-6fc9-4052-9808-ffc561da21ca\" (UID: \"ba0637f3-6fc9-4052-9808-ffc561da21ca\") " Nov 01 21:06:22.884266 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.884231 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba0637f3-6fc9-4052-9808-ffc561da21ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba0637f3-6fc9-4052-9808-ffc561da21ca-workload-socket\") pod \"ba0637f3-6fc9-4052-9808-ffc561da21ca\" (UID: \"ba0637f3-6fc9-4052-9808-ffc561da21ca\") " Nov 01 21:06:22.884443 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.884310 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba0637f3-6fc9-4052-9808-ffc561da21ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba0637f3-6fc9-4052-9808-ffc561da21ca-istio-data\") pod \"ba0637f3-6fc9-4052-9808-ffc561da21ca\" (UID: \"ba0637f3-6fc9-4052-9808-ffc561da21ca\") " Nov 01 21:06:22.884443 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.884381 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba0637f3-6fc9-4052-9808-ffc561da21ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba0637f3-6fc9-4052-9808-ffc561da21ca-istio-podinfo\") pod \"ba0637f3-6fc9-4052-9808-ffc561da21ca\" (UID: \"ba0637f3-6fc9-4052-9808-ffc561da21ca\") " Nov 01 21:06:22.884559 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.884486 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba0637f3-6fc9-4052-9808-ffc561da21ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba0637f3-6fc9-4052-9808-ffc561da21ca-credential-socket\") pod \"ba0637f3-6fc9-4052-9808-ffc561da21ca\" (UID: \"ba0637f3-6fc9-4052-9808-ffc561da21ca\") " Nov 01 21:06:22.884559 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.884506 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba0637f3-6fc9-4052-9808-ffc561da21ca-workload-certs" pod "ba0637f3-6fc9-4052-9808-ffc561da21ca" (UID: "ba0637f3-6fc9-4052-9808-ffc561da21ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:06:22.885225 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.885198 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba0637f3-6fc9-4052-9808-ffc561da21ca-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:22.886939 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.886546 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba0637f3-6fc9-4052-9808-ffc561da21ca-istiod-ca-cert" pod "ba0637f3-6fc9-4052-9808-ffc561da21ca" (UID: "ba0637f3-6fc9-4052-9808-ffc561da21ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:06:22.887126 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.886716 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba0637f3-6fc9-4052-9808-ffc561da21ca-istio-envoy" pod "ba0637f3-6fc9-4052-9808-ffc561da21ca" (UID: "ba0637f3-6fc9-4052-9808-ffc561da21ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:06:22.887126 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.886813 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba0637f3-6fc9-4052-9808-ffc561da21ca-istio-data" pod "ba0637f3-6fc9-4052-9808-ffc561da21ca" (UID: "ba0637f3-6fc9-4052-9808-ffc561da21ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:06:22.887126 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.886857 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba0637f3-6fc9-4052-9808-ffc561da21ca-credential-socket" pod "ba0637f3-6fc9-4052-9808-ffc561da21ca" (UID: "ba0637f3-6fc9-4052-9808-ffc561da21ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:06:22.887126 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.886899 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba0637f3-6fc9-4052-9808-ffc561da21ca-workload-socket" pod "ba0637f3-6fc9-4052-9808-ffc561da21ca" (UID: "ba0637f3-6fc9-4052-9808-ffc561da21ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:06:22.887126 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.886988 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba0637f3-6fc9-4052-9808-ffc561da21ca-istio-token" pod "ba0637f3-6fc9-4052-9808-ffc561da21ca" (UID: "ba0637f3-6fc9-4052-9808-ffc561da21ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:06:22.888031 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.887972 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba0637f3-6fc9-4052-9808-ffc561da21ca-kube-api-access-qgdq5" pod "ba0637f3-6fc9-4052-9808-ffc561da21ca" (UID: "ba0637f3-6fc9-4052-9808-ffc561da21ca"). InnerVolumeSpecName "kube-api-access-qgdq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:06:22.888709 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.888683 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba0637f3-6fc9-4052-9808-ffc561da21ca-istio-podinfo" pod "ba0637f3-6fc9-4052-9808-ffc561da21ca" (UID: "ba0637f3-6fc9-4052-9808-ffc561da21ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:06:22.988259 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.988233 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba0637f3-6fc9-4052-9808-ffc561da21ca-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:22.988643 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.988262 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba0637f3-6fc9-4052-9808-ffc561da21ca-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:22.988643 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.988272 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba0637f3-6fc9-4052-9808-ffc561da21ca-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:22.988643 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.988283 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba0637f3-6fc9-4052-9808-ffc561da21ca-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:22.988643 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.988292 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba0637f3-6fc9-4052-9808-ffc561da21ca-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:22.988643 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.988300 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba0637f3-6fc9-4052-9808-ffc561da21ca-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:22.988643 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.988308 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba0637f3-6fc9-4052-9808-ffc561da21ca-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:22.988643 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:22.988316 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgdq5\" (UniqueName: \"kubernetes.io/projected/ba0637f3-6fc9-4052-9808-ffc561da21ca-kube-api-access-qgdq5\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:23.373198 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:23.373120 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50906dd8-8d76-4fcb-8415-f8f32cd47ffc" path="/var/lib/kubelet/pods/50906dd8-8d76-4fcb-8415-f8f32cd47ffc/volumes" Nov 01 21:06:23.415656 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:23.415619 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:23.574581 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:23.574545 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:23.580518 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:23.580488 4484 generic.go:356] "Generic (PLEG): container finished" podID="ba0637f3-6fc9-4052-9808-ffc561da21ca" containerID="364e65f65e27277e3131247864654cd2e076a2ebeb6e6f655f30ba7f54da675c" exitCode=0 Nov 01 21:06:23.580699 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:23.580547 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5" Nov 01 21:06:23.580699 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:23.580572 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5" event={"ID":"ba0637f3-6fc9-4052-9808-ffc561da21ca","Type":"ContainerDied","Data":"364e65f65e27277e3131247864654cd2e076a2ebeb6e6f655f30ba7f54da675c"} Nov 01 21:06:23.580699 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:23.580628 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5" event={"ID":"ba0637f3-6fc9-4052-9808-ffc561da21ca","Type":"ContainerDied","Data":"716d2c8de7d4b1849b2373f06181ef426480dd1616012c8e087f7eade84a4abe"} Nov 01 21:06:23.580699 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:23.580651 4484 scope.go:122] "RemoveContainer" containerID="364e65f65e27277e3131247864654cd2e076a2ebeb6e6f655f30ba7f54da675c" Nov 01 21:06:23.592112 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:23.592092 4484 scope.go:122] "RemoveContainer" containerID="364e65f65e27277e3131247864654cd2e076a2ebeb6e6f655f30ba7f54da675c" Nov 01 21:06:23.592441 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:06:23.592420 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"364e65f65e27277e3131247864654cd2e076a2ebeb6e6f655f30ba7f54da675c\": container with ID starting with 364e65f65e27277e3131247864654cd2e076a2ebeb6e6f655f30ba7f54da675c not found: ID does not exist" containerID="364e65f65e27277e3131247864654cd2e076a2ebeb6e6f655f30ba7f54da675c" Nov 01 21:06:23.592532 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:23.592448 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"364e65f65e27277e3131247864654cd2e076a2ebeb6e6f655f30ba7f54da675c"} err="failed to get container status \"364e65f65e27277e3131247864654cd2e076a2ebeb6e6f655f30ba7f54da675c\": rpc error: code = NotFound desc = could not find container \"364e65f65e27277e3131247864654cd2e076a2ebeb6e6f655f30ba7f54da675c\": container with ID starting with 364e65f65e27277e3131247864654cd2e076a2ebeb6e6f655f30ba7f54da675c not found: ID does not exist" Nov 01 21:06:23.890621 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:23.884069 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:23.920553 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:23.920515 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5"] Nov 01 21:06:23.929927 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:23.929841 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9ae21-openshift-default-5b788bdc65-8xtk5"] Nov 01 21:06:24.094575 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:24.094551 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h" Nov 01 21:06:24.184043 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:24.183954 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2fa50d2-8fa0-4997-9fdf-354318d94561-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2fa50d2-8fa0-4997-9fdf-354318d94561-istio-envoy\") pod \"c2fa50d2-8fa0-4997-9fdf-354318d94561\" (UID: \"c2fa50d2-8fa0-4997-9fdf-354318d94561\") " Nov 01 21:06:24.184190 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:24.184100 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2fa50d2-8fa0-4997-9fdf-354318d94561-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2fa50d2-8fa0-4997-9fdf-354318d94561-istiod-ca-cert\") pod \"c2fa50d2-8fa0-4997-9fdf-354318d94561\" (UID: \"c2fa50d2-8fa0-4997-9fdf-354318d94561\") " Nov 01 21:06:24.184190 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:24.184131 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2fa50d2-8fa0-4997-9fdf-354318d94561-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2fa50d2-8fa0-4997-9fdf-354318d94561-istio-data\") pod \"c2fa50d2-8fa0-4997-9fdf-354318d94561\" (UID: \"c2fa50d2-8fa0-4997-9fdf-354318d94561\") " Nov 01 21:06:24.184761 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:24.184199 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2fa50d2-8fa0-4997-9fdf-354318d94561-kube-api-access-zthvd\" (UniqueName: \"kubernetes.io/projected/c2fa50d2-8fa0-4997-9fdf-354318d94561-kube-api-access-zthvd\") pod \"c2fa50d2-8fa0-4997-9fdf-354318d94561\" (UID: \"c2fa50d2-8fa0-4997-9fdf-354318d94561\") " Nov 01 21:06:24.184761 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:24.184223 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2fa50d2-8fa0-4997-9fdf-354318d94561-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2fa50d2-8fa0-4997-9fdf-354318d94561-workload-certs\") pod \"c2fa50d2-8fa0-4997-9fdf-354318d94561\" (UID: \"c2fa50d2-8fa0-4997-9fdf-354318d94561\") " Nov 01 21:06:24.184761 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:24.184291 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2fa50d2-8fa0-4997-9fdf-354318d94561-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2fa50d2-8fa0-4997-9fdf-354318d94561-workload-socket\") pod \"c2fa50d2-8fa0-4997-9fdf-354318d94561\" (UID: \"c2fa50d2-8fa0-4997-9fdf-354318d94561\") " Nov 01 21:06:24.184761 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:24.184316 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2fa50d2-8fa0-4997-9fdf-354318d94561-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2fa50d2-8fa0-4997-9fdf-354318d94561-credential-socket\") pod \"c2fa50d2-8fa0-4997-9fdf-354318d94561\" (UID: \"c2fa50d2-8fa0-4997-9fdf-354318d94561\") " Nov 01 21:06:24.184761 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:24.184400 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2fa50d2-8fa0-4997-9fdf-354318d94561-istio-token\" (UniqueName: \"kubernetes.io/projected/c2fa50d2-8fa0-4997-9fdf-354318d94561-istio-token\") pod \"c2fa50d2-8fa0-4997-9fdf-354318d94561\" (UID: \"c2fa50d2-8fa0-4997-9fdf-354318d94561\") " Nov 01 21:06:24.184761 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:24.184444 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c2fa50d2-8fa0-4997-9fdf-354318d94561-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2fa50d2-8fa0-4997-9fdf-354318d94561-istio-podinfo\") pod \"c2fa50d2-8fa0-4997-9fdf-354318d94561\" (UID: \"c2fa50d2-8fa0-4997-9fdf-354318d94561\") " Nov 01 21:06:24.184761 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:24.184536 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2fa50d2-8fa0-4997-9fdf-354318d94561-istiod-ca-cert" pod "c2fa50d2-8fa0-4997-9fdf-354318d94561" (UID: "c2fa50d2-8fa0-4997-9fdf-354318d94561"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:06:24.184761 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:24.184591 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2fa50d2-8fa0-4997-9fdf-354318d94561-istio-data" pod "c2fa50d2-8fa0-4997-9fdf-354318d94561" (UID: "c2fa50d2-8fa0-4997-9fdf-354318d94561"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:06:24.184761 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:24.184707 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2fa50d2-8fa0-4997-9fdf-354318d94561-workload-certs" pod "c2fa50d2-8fa0-4997-9fdf-354318d94561" (UID: "c2fa50d2-8fa0-4997-9fdf-354318d94561"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:06:24.184761 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:24.184747 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2fa50d2-8fa0-4997-9fdf-354318d94561-credential-socket" pod "c2fa50d2-8fa0-4997-9fdf-354318d94561" (UID: "c2fa50d2-8fa0-4997-9fdf-354318d94561"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:06:24.185368 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:24.185211 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2fa50d2-8fa0-4997-9fdf-354318d94561-workload-socket" pod "c2fa50d2-8fa0-4997-9fdf-354318d94561" (UID: "c2fa50d2-8fa0-4997-9fdf-354318d94561"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:06:24.185368 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:24.185273 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2fa50d2-8fa0-4997-9fdf-354318d94561-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:24.185368 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:24.185290 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2fa50d2-8fa0-4997-9fdf-354318d94561-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:24.185368 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:24.185298 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2fa50d2-8fa0-4997-9fdf-354318d94561-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:24.185368 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:24.185308 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2fa50d2-8fa0-4997-9fdf-354318d94561-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:24.186619 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:24.186579 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2fa50d2-8fa0-4997-9fdf-354318d94561-kube-api-access-zthvd" pod "c2fa50d2-8fa0-4997-9fdf-354318d94561" (UID: "c2fa50d2-8fa0-4997-9fdf-354318d94561"). InnerVolumeSpecName "kube-api-access-zthvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:06:24.186727 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:24.186708 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2fa50d2-8fa0-4997-9fdf-354318d94561-istio-token" pod "c2fa50d2-8fa0-4997-9fdf-354318d94561" (UID: "c2fa50d2-8fa0-4997-9fdf-354318d94561"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:06:24.186771 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:24.186722 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2fa50d2-8fa0-4997-9fdf-354318d94561-istio-envoy" pod "c2fa50d2-8fa0-4997-9fdf-354318d94561" (UID: "c2fa50d2-8fa0-4997-9fdf-354318d94561"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:06:24.186921 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:24.186905 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2fa50d2-8fa0-4997-9fdf-354318d94561-istio-podinfo" pod "c2fa50d2-8fa0-4997-9fdf-354318d94561" (UID: "c2fa50d2-8fa0-4997-9fdf-354318d94561"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:06:24.286517 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:24.286484 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2fa50d2-8fa0-4997-9fdf-354318d94561-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:24.286517 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:24.286515 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zthvd\" (UniqueName: \"kubernetes.io/projected/c2fa50d2-8fa0-4997-9fdf-354318d94561-kube-api-access-zthvd\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:24.286720 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:24.286526 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2fa50d2-8fa0-4997-9fdf-354318d94561-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:24.286720 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:24.286535 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2fa50d2-8fa0-4997-9fdf-354318d94561-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:24.286720 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:24.286544 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2fa50d2-8fa0-4997-9fdf-354318d94561-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:24.545288 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:24.545254 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:24.590684 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:24.590650 4484 generic.go:356] "Generic (PLEG): container finished" podID="c2fa50d2-8fa0-4997-9fdf-354318d94561" containerID="7b41bb1933550373133d88f2be5aef3a2731f762e951e4c83f680ced0208eb22" exitCode=0 Nov 01 21:06:24.590897 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:24.590703 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h" Nov 01 21:06:24.590897 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:24.590723 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h" event={"ID":"c2fa50d2-8fa0-4997-9fdf-354318d94561","Type":"ContainerDied","Data":"7b41bb1933550373133d88f2be5aef3a2731f762e951e4c83f680ced0208eb22"} Nov 01 21:06:24.590897 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:24.590776 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h" event={"ID":"c2fa50d2-8fa0-4997-9fdf-354318d94561","Type":"ContainerDied","Data":"401ed49514df8e798455704fc65da17495321ec2f7974781f7c83e78fe59db9b"} Nov 01 21:06:24.590897 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:24.590804 4484 scope.go:122] "RemoveContainer" containerID="7b41bb1933550373133d88f2be5aef3a2731f762e951e4c83f680ced0208eb22" Nov 01 21:06:24.602063 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:24.602045 4484 scope.go:122] "RemoveContainer" containerID="208dd4fa036e63c78be6383d20c5d27a5acb11fe9023605beafda5d7a153a014" Nov 01 21:06:24.611501 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:24.611482 4484 scope.go:122] "RemoveContainer" containerID="7b41bb1933550373133d88f2be5aef3a2731f762e951e4c83f680ced0208eb22" Nov 01 21:06:24.611770 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:06:24.611754 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b41bb1933550373133d88f2be5aef3a2731f762e951e4c83f680ced0208eb22\": container with ID starting with 7b41bb1933550373133d88f2be5aef3a2731f762e951e4c83f680ced0208eb22 not found: ID does not exist" containerID="7b41bb1933550373133d88f2be5aef3a2731f762e951e4c83f680ced0208eb22" Nov 01 21:06:24.611840 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:24.611779 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b41bb1933550373133d88f2be5aef3a2731f762e951e4c83f680ced0208eb22"} err="failed to get container status \"7b41bb1933550373133d88f2be5aef3a2731f762e951e4c83f680ced0208eb22\": rpc error: code = NotFound desc = could not find container \"7b41bb1933550373133d88f2be5aef3a2731f762e951e4c83f680ced0208eb22\": container with ID starting with 7b41bb1933550373133d88f2be5aef3a2731f762e951e4c83f680ced0208eb22 not found: ID does not exist" Nov 01 21:06:24.611840 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:24.611799 4484 scope.go:122] "RemoveContainer" containerID="208dd4fa036e63c78be6383d20c5d27a5acb11fe9023605beafda5d7a153a014" Nov 01 21:06:24.612097 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:06:24.612076 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"208dd4fa036e63c78be6383d20c5d27a5acb11fe9023605beafda5d7a153a014\": container with ID starting with 208dd4fa036e63c78be6383d20c5d27a5acb11fe9023605beafda5d7a153a014 not found: ID does not exist" containerID="208dd4fa036e63c78be6383d20c5d27a5acb11fe9023605beafda5d7a153a014" Nov 01 21:06:24.612207 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:24.612099 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"208dd4fa036e63c78be6383d20c5d27a5acb11fe9023605beafda5d7a153a014"} err="failed to get container status \"208dd4fa036e63c78be6383d20c5d27a5acb11fe9023605beafda5d7a153a014\": rpc error: code = NotFound desc = could not find container \"208dd4fa036e63c78be6383d20c5d27a5acb11fe9023605beafda5d7a153a014\": container with ID starting with 208dd4fa036e63c78be6383d20c5d27a5acb11fe9023605beafda5d7a153a014 not found: ID does not exist" Nov 01 21:06:24.712190 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:24.712155 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:24.761733 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:24.761700 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h"] Nov 01 21:06:24.812592 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:24.812509 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-29baa8-openshift-default-84786bd7d7-q544h"] Nov 01 21:06:24.871829 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:24.871796 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:25.373533 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:25.373500 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba0637f3-6fc9-4052-9808-ffc561da21ca" path="/var/lib/kubelet/pods/ba0637f3-6fc9-4052-9808-ffc561da21ca/volumes" Nov 01 21:06:25.374063 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:25.374047 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2fa50d2-8fa0-4997-9fdf-354318d94561" path="/var/lib/kubelet/pods/c2fa50d2-8fa0-4997-9fdf-354318d94561/volumes" Nov 01 21:06:25.643447 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:25.643367 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:26.108735 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:26.108688 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26"] Nov 01 21:06:26.109620 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:26.109592 4484 cpu_manager.go:414] "containerMap: removing container" podUID="c2fa50d2-8fa0-4997-9fdf-354318d94561" containerName="istio-proxy" Nov 01 21:06:26.109620 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:26.109617 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2fa50d2-8fa0-4997-9fdf-354318d94561" containerName="istio-proxy" Nov 01 21:06:26.109836 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:26.109659 4484 cpu_manager.go:414] "containerMap: removing container" podUID="c2fa50d2-8fa0-4997-9fdf-354318d94561" containerName="istio-proxy" Nov 01 21:06:26.109836 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:26.109667 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2fa50d2-8fa0-4997-9fdf-354318d94561" containerName="istio-proxy" Nov 01 21:06:26.109836 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:26.109693 4484 cpu_manager.go:414] "containerMap: removing container" podUID="ba0637f3-6fc9-4052-9808-ffc561da21ca" containerName="istio-proxy" Nov 01 21:06:26.109836 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:26.109700 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba0637f3-6fc9-4052-9808-ffc561da21ca" containerName="istio-proxy" Nov 01 21:06:26.109836 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:26.109728 4484 cpu_manager.go:414] "containerMap: removing container" podUID="50906dd8-8d76-4fcb-8415-f8f32cd47ffc" containerName="istio-proxy" Nov 01 21:06:26.109836 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:26.109736 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50906dd8-8d76-4fcb-8415-f8f32cd47ffc" containerName="istio-proxy" Nov 01 21:06:26.110241 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:26.109882 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2fa50d2-8fa0-4997-9fdf-354318d94561" containerName="istio-proxy" Nov 01 21:06:26.110241 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:26.109897 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba0637f3-6fc9-4052-9808-ffc561da21ca" containerName="istio-proxy" Nov 01 21:06:26.110241 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:26.109911 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="50906dd8-8d76-4fcb-8415-f8f32cd47ffc" containerName="istio-proxy" Nov 01 21:06:26.110241 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:26.109924 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2fa50d2-8fa0-4997-9fdf-354318d94561" containerName="istio-proxy" Nov 01 21:06:26.123101 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:26.117370 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26" Nov 01 21:06:26.195333 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:26.195301 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-6b9305-openshift-default-dockercfg-w9nk5\"" Nov 01 21:06:26.229097 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:26.229046 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69124be6-394b-45bd-9d52-6de87dfc5648-istio-data\") pod \"e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26\" (UID: \"69124be6-394b-45bd-9d52-6de87dfc5648\") " pod="openshift-ingress/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26" Nov 01 21:06:26.237021 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:26.229433 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69124be6-394b-45bd-9d52-6de87dfc5648-credential-socket\") pod \"e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26\" (UID: \"69124be6-394b-45bd-9d52-6de87dfc5648\") " pod="openshift-ingress/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26" Nov 01 21:06:26.237436 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:26.237392 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69124be6-394b-45bd-9d52-6de87dfc5648-istio-token\") pod \"e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26\" (UID: \"69124be6-394b-45bd-9d52-6de87dfc5648\") " pod="openshift-ingress/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26" Nov 01 21:06:26.237688 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:26.237671 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69124be6-394b-45bd-9d52-6de87dfc5648-istiod-ca-cert\") pod \"e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26\" (UID: \"69124be6-394b-45bd-9d52-6de87dfc5648\") " pod="openshift-ingress/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26" Nov 01 21:06:26.237848 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:26.237834 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69124be6-394b-45bd-9d52-6de87dfc5648-workload-certs\") pod \"e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26\" (UID: \"69124be6-394b-45bd-9d52-6de87dfc5648\") " pod="openshift-ingress/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26" Nov 01 21:06:26.243279 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:26.243250 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69124be6-394b-45bd-9d52-6de87dfc5648-workload-socket\") pod \"e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26\" (UID: \"69124be6-394b-45bd-9d52-6de87dfc5648\") " pod="openshift-ingress/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26" Nov 01 21:06:26.243512 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:26.243495 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69124be6-394b-45bd-9d52-6de87dfc5648-istio-envoy\") pod \"e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26\" (UID: \"69124be6-394b-45bd-9d52-6de87dfc5648\") " pod="openshift-ingress/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26" Nov 01 21:06:26.250159 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:26.249573 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:26.250461 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:26.250441 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69124be6-394b-45bd-9d52-6de87dfc5648-istio-podinfo\") pod \"e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26\" (UID: \"69124be6-394b-45bd-9d52-6de87dfc5648\") " pod="openshift-ingress/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26" Nov 01 21:06:26.250701 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:26.250681 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqfh8\" (UniqueName: \"kubernetes.io/projected/69124be6-394b-45bd-9d52-6de87dfc5648-kube-api-access-kqfh8\") pod \"e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26\" (UID: \"69124be6-394b-45bd-9d52-6de87dfc5648\") " pod="openshift-ingress/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26" Nov 01 21:06:26.376035 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:26.375926 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26"] Nov 01 21:06:26.476851 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:26.476808 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69124be6-394b-45bd-9d52-6de87dfc5648-workload-socket\") pod \"e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26\" (UID: \"69124be6-394b-45bd-9d52-6de87dfc5648\") " pod="openshift-ingress/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26" Nov 01 21:06:26.477074 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:26.476856 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69124be6-394b-45bd-9d52-6de87dfc5648-istio-envoy\") pod \"e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26\" (UID: \"69124be6-394b-45bd-9d52-6de87dfc5648\") " pod="openshift-ingress/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26" Nov 01 21:06:26.477074 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:26.476890 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69124be6-394b-45bd-9d52-6de87dfc5648-istio-podinfo\") pod \"e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26\" (UID: \"69124be6-394b-45bd-9d52-6de87dfc5648\") " pod="openshift-ingress/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26" Nov 01 21:06:26.477074 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:26.476933 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqfh8\" (UniqueName: \"kubernetes.io/projected/69124be6-394b-45bd-9d52-6de87dfc5648-kube-api-access-kqfh8\") pod \"e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26\" (UID: \"69124be6-394b-45bd-9d52-6de87dfc5648\") " pod="openshift-ingress/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26" Nov 01 21:06:26.477074 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:26.477055 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69124be6-394b-45bd-9d52-6de87dfc5648-istio-data\") pod \"e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26\" (UID: \"69124be6-394b-45bd-9d52-6de87dfc5648\") " pod="openshift-ingress/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26" Nov 01 21:06:26.477333 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:26.477111 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69124be6-394b-45bd-9d52-6de87dfc5648-credential-socket\") pod \"e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26\" (UID: \"69124be6-394b-45bd-9d52-6de87dfc5648\") " pod="openshift-ingress/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26" Nov 01 21:06:26.477333 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:26.477178 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69124be6-394b-45bd-9d52-6de87dfc5648-istio-token\") pod \"e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26\" (UID: \"69124be6-394b-45bd-9d52-6de87dfc5648\") " pod="openshift-ingress/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26" Nov 01 21:06:26.477333 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:26.477240 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69124be6-394b-45bd-9d52-6de87dfc5648-istiod-ca-cert\") pod \"e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26\" (UID: \"69124be6-394b-45bd-9d52-6de87dfc5648\") " pod="openshift-ingress/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26" Nov 01 21:06:26.477333 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:26.477248 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69124be6-394b-45bd-9d52-6de87dfc5648-workload-socket\") pod \"e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26\" (UID: \"69124be6-394b-45bd-9d52-6de87dfc5648\") " pod="openshift-ingress/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26" Nov 01 21:06:26.477675 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:26.477437 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69124be6-394b-45bd-9d52-6de87dfc5648-workload-certs\") pod \"e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26\" (UID: \"69124be6-394b-45bd-9d52-6de87dfc5648\") " pod="openshift-ingress/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26" Nov 01 21:06:26.477675 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:26.477532 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69124be6-394b-45bd-9d52-6de87dfc5648-credential-socket\") pod \"e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26\" (UID: \"69124be6-394b-45bd-9d52-6de87dfc5648\") " pod="openshift-ingress/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26" Nov 01 21:06:26.477796 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:26.477753 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69124be6-394b-45bd-9d52-6de87dfc5648-istio-data\") pod \"e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26\" (UID: \"69124be6-394b-45bd-9d52-6de87dfc5648\") " pod="openshift-ingress/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26" Nov 01 21:06:26.478343 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:26.477804 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69124be6-394b-45bd-9d52-6de87dfc5648-workload-certs\") pod \"e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26\" (UID: \"69124be6-394b-45bd-9d52-6de87dfc5648\") " pod="openshift-ingress/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26" Nov 01 21:06:26.478343 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:26.478234 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69124be6-394b-45bd-9d52-6de87dfc5648-istiod-ca-cert\") pod \"e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26\" (UID: \"69124be6-394b-45bd-9d52-6de87dfc5648\") " pod="openshift-ingress/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26" Nov 01 21:06:26.521934 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:26.518671 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69124be6-394b-45bd-9d52-6de87dfc5648-istio-envoy\") pod \"e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26\" (UID: \"69124be6-394b-45bd-9d52-6de87dfc5648\") " pod="openshift-ingress/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26" Nov 01 21:06:26.522220 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:26.522177 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69124be6-394b-45bd-9d52-6de87dfc5648-istio-podinfo\") pod \"e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26\" (UID: \"69124be6-394b-45bd-9d52-6de87dfc5648\") " pod="openshift-ingress/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26" Nov 01 21:06:26.535693 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:26.535667 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69124be6-394b-45bd-9d52-6de87dfc5648-istio-token\") pod \"e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26\" (UID: \"69124be6-394b-45bd-9d52-6de87dfc5648\") " pod="openshift-ingress/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26" Nov 01 21:06:26.535856 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:26.535837 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqfh8\" (UniqueName: \"kubernetes.io/projected/69124be6-394b-45bd-9d52-6de87dfc5648-kube-api-access-kqfh8\") pod \"e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26\" (UID: \"69124be6-394b-45bd-9d52-6de87dfc5648\") " pod="openshift-ingress/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26" Nov 01 21:06:26.538499 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:26.538474 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:26.766856 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:26.766820 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26" Nov 01 21:06:27.110068 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:27.104593 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:27.204039 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:27.204007 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26"] Nov 01 21:06:27.204406 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:06:27.204380 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69124be6_394b_45bd_9d52_6de87dfc5648.slice/crio-bd6991dfef6e77e33fb61901d7da4ca76aef504ef7fc039f9b362075b840480b WatchSource:0}: Error finding container bd6991dfef6e77e33fb61901d7da4ca76aef504ef7fc039f9b362075b840480b: Status 404 returned error can't find the container with id bd6991dfef6e77e33fb61901d7da4ca76aef504ef7fc039f9b362075b840480b Nov 01 21:06:27.207029 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:27.206980 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:06:27.207212 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:27.207078 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:06:27.207212 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:27.207125 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:06:27.614876 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:27.614831 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26" event={"ID":"69124be6-394b-45bd-9d52-6de87dfc5648","Type":"ContainerStarted","Data":"7d182619c2a5d6c4a794f450da9d548e759cabb34a4972bf9152cc237cc01aab"} Nov 01 21:06:27.615304 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:27.614899 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26" event={"ID":"69124be6-394b-45bd-9d52-6de87dfc5648","Type":"ContainerStarted","Data":"bd6991dfef6e77e33fb61901d7da4ca76aef504ef7fc039f9b362075b840480b"} Nov 01 21:06:27.734712 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:27.734648 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26" podStartSLOduration=1.734633699 podStartE2EDuration="1.734633699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:06:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:06:27.733268412 +0000 UTC m=+2138.510908548" watchObservedRunningTime="2026-11-01 21:06:27.734633699 +0000 UTC m=+2138.512273834" Nov 01 21:06:27.767360 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:27.767319 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26" Nov 01 21:06:27.771175 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:27.771147 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:27.966617 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:27.962859 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:28.171038 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:28.167819 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-deff5e-77db945ff9-jtzp7"] Nov 01 21:06:28.175039 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:28.172835 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-deff5e-77db945ff9-jtzp7" Nov 01 21:06:28.205871 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:28.205839 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d19b68e-4561-480b-bcd4-3cc376211ff3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-deff5e-77db945ff9-jtzp7\" (UID: \"8d19b68e-4561-480b-bcd4-3cc376211ff3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-deff5e-77db945ff9-jtzp7" Nov 01 21:06:28.206103 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:28.206082 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75nbc\" (UniqueName: \"kubernetes.io/projected/8d19b68e-4561-480b-bcd4-3cc376211ff3-kube-api-access-75nbc\") pod \"payload-pre-processing-e2e-api-a-deff5e-77db945ff9-jtzp7\" (UID: \"8d19b68e-4561-480b-bcd4-3cc376211ff3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-deff5e-77db945ff9-jtzp7" Nov 01 21:06:28.217703 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:28.217631 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-deff5e-77db945ff9-jtzp7"] Nov 01 21:06:28.225959 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:28.225930 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-deff5e\"" Nov 01 21:06:28.226203 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:28.226052 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-deff5e-dockercfg-x84rj\"" Nov 01 21:06:28.315019 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:28.308907 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75nbc\" (UniqueName: \"kubernetes.io/projected/8d19b68e-4561-480b-bcd4-3cc376211ff3-kube-api-access-75nbc\") pod \"payload-pre-processing-e2e-api-a-deff5e-77db945ff9-jtzp7\" (UID: \"8d19b68e-4561-480b-bcd4-3cc376211ff3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-deff5e-77db945ff9-jtzp7" Nov 01 21:06:28.315019 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:28.309050 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d19b68e-4561-480b-bcd4-3cc376211ff3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-deff5e-77db945ff9-jtzp7\" (UID: \"8d19b68e-4561-480b-bcd4-3cc376211ff3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-deff5e-77db945ff9-jtzp7" Nov 01 21:06:28.315019 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:28.310254 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d19b68e-4561-480b-bcd4-3cc376211ff3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-deff5e-77db945ff9-jtzp7\" (UID: \"8d19b68e-4561-480b-bcd4-3cc376211ff3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-deff5e-77db945ff9-jtzp7" Nov 01 21:06:28.362953 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:28.362890 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-deff5e-bd664cbf7-wvr5w"] Nov 01 21:06:28.369366 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:28.369328 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-deff5e-bd664cbf7-wvr5w" Nov 01 21:06:28.413029 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:28.410199 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-deff5e-bd664cbf7-wvr5w"] Nov 01 21:06:28.425041 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:28.418071 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75nbc\" (UniqueName: \"kubernetes.io/projected/8d19b68e-4561-480b-bcd4-3cc376211ff3-kube-api-access-75nbc\") pod \"payload-pre-processing-e2e-api-a-deff5e-77db945ff9-jtzp7\" (UID: \"8d19b68e-4561-480b-bcd4-3cc376211ff3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-deff5e-77db945ff9-jtzp7" Nov 01 21:06:28.489405 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:28.489356 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-deff5e-77db945ff9-jtzp7" Nov 01 21:06:28.513443 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:28.513318 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ef99ac7-1947-4ffc-98ad-4a167eecba8c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-deff5e-bd664cbf7-wvr5w\" (UID: \"9ef99ac7-1947-4ffc-98ad-4a167eecba8c\") " pod="openshift-ingress/payload-processing-e2e-api-a-deff5e-bd664cbf7-wvr5w" Nov 01 21:06:28.513677 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:28.513649 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g7gz\" (UniqueName: \"kubernetes.io/projected/9ef99ac7-1947-4ffc-98ad-4a167eecba8c-kube-api-access-8g7gz\") pod \"payload-processing-e2e-api-a-deff5e-bd664cbf7-wvr5w\" (UID: \"9ef99ac7-1947-4ffc-98ad-4a167eecba8c\") " pod="openshift-ingress/payload-processing-e2e-api-a-deff5e-bd664cbf7-wvr5w" Nov 01 21:06:28.536759 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:28.536706 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:28.615660 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:28.615622 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ef99ac7-1947-4ffc-98ad-4a167eecba8c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-deff5e-bd664cbf7-wvr5w\" (UID: \"9ef99ac7-1947-4ffc-98ad-4a167eecba8c\") " pod="openshift-ingress/payload-processing-e2e-api-a-deff5e-bd664cbf7-wvr5w" Nov 01 21:06:28.616201 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:28.615699 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g7gz\" (UniqueName: \"kubernetes.io/projected/9ef99ac7-1947-4ffc-98ad-4a167eecba8c-kube-api-access-8g7gz\") pod \"payload-processing-e2e-api-a-deff5e-bd664cbf7-wvr5w\" (UID: \"9ef99ac7-1947-4ffc-98ad-4a167eecba8c\") " pod="openshift-ingress/payload-processing-e2e-api-a-deff5e-bd664cbf7-wvr5w" Nov 01 21:06:28.616568 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:28.616532 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ef99ac7-1947-4ffc-98ad-4a167eecba8c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-deff5e-bd664cbf7-wvr5w\" (UID: \"9ef99ac7-1947-4ffc-98ad-4a167eecba8c\") " pod="openshift-ingress/payload-processing-e2e-api-a-deff5e-bd664cbf7-wvr5w" Nov 01 21:06:28.674145 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:28.674115 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g7gz\" (UniqueName: \"kubernetes.io/projected/9ef99ac7-1947-4ffc-98ad-4a167eecba8c-kube-api-access-8g7gz\") pod \"payload-processing-e2e-api-a-deff5e-bd664cbf7-wvr5w\" (UID: \"9ef99ac7-1947-4ffc-98ad-4a167eecba8c\") " pod="openshift-ingress/payload-processing-e2e-api-a-deff5e-bd664cbf7-wvr5w" Nov 01 21:06:28.704173 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:28.704123 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:28.736303 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:28.736269 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-deff5e-bd664cbf7-wvr5w" Nov 01 21:06:28.775046 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:28.775015 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26" Nov 01 21:06:28.873534 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:28.873496 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-deff5e-77db945ff9-jtzp7"] Nov 01 21:06:28.873679 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:06:28.873507 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d19b68e_4561_480b_bcd4_3cc376211ff3.slice/crio-c4aa699a240fc79414edf17d046409ee3f2fc730111588579aa07e5c7428f31d WatchSource:0}: Error finding container c4aa699a240fc79414edf17d046409ee3f2fc730111588579aa07e5c7428f31d: Status 404 returned error can't find the container with id c4aa699a240fc79414edf17d046409ee3f2fc730111588579aa07e5c7428f31d Nov 01 21:06:29.120398 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:29.120368 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-deff5e-bd664cbf7-wvr5w"] Nov 01 21:06:29.121010 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:06:29.120965 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ef99ac7_1947_4ffc_98ad_4a167eecba8c.slice/crio-cc42eb0d8fdd747be28450f10f4f0f9215cb55bdf59d3abe3cff1c9e799977a0 WatchSource:0}: Error finding container cc42eb0d8fdd747be28450f10f4f0f9215cb55bdf59d3abe3cff1c9e799977a0: Status 404 returned error can't find the container with id cc42eb0d8fdd747be28450f10f4f0f9215cb55bdf59d3abe3cff1c9e799977a0 Nov 01 21:06:29.636237 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:29.636200 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-deff5e-77db945ff9-jtzp7" event={"ID":"8d19b68e-4561-480b-bcd4-3cc376211ff3","Type":"ContainerStarted","Data":"d176456610e05c06599d5c7fcf74edec940c77e8294a2b88b31f6f75b2386c84"} Nov 01 21:06:29.636775 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:29.636271 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-deff5e-77db945ff9-jtzp7" Nov 01 21:06:29.636775 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:29.636296 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-deff5e-77db945ff9-jtzp7" event={"ID":"8d19b68e-4561-480b-bcd4-3cc376211ff3","Type":"ContainerStarted","Data":"c4aa699a240fc79414edf17d046409ee3f2fc730111588579aa07e5c7428f31d"} Nov 01 21:06:29.638121 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:29.638095 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-deff5e-bd664cbf7-wvr5w" event={"ID":"9ef99ac7-1947-4ffc-98ad-4a167eecba8c","Type":"ContainerStarted","Data":"79fad4827ba26033adc4030db87771feea93adc731d7d8114624b186596d43b5"} Nov 01 21:06:29.638258 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:29.638138 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-deff5e-bd664cbf7-wvr5w" event={"ID":"9ef99ac7-1947-4ffc-98ad-4a167eecba8c","Type":"ContainerStarted","Data":"cc42eb0d8fdd747be28450f10f4f0f9215cb55bdf59d3abe3cff1c9e799977a0"} Nov 01 21:06:29.638258 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:29.638185 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-deff5e-bd664cbf7-wvr5w" Nov 01 21:06:29.638483 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:29.638472 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26" Nov 01 21:06:29.639538 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:29.639522 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26" Nov 01 21:06:29.752470 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:29.752413 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-deff5e-77db945ff9-jtzp7" podStartSLOduration=1.752400487 podStartE2EDuration="1.752400487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:06:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:06:29.750072739 +0000 UTC m=+2140.527712874" watchObservedRunningTime="2026-11-01 21:06:29.752400487 +0000 UTC m=+2140.530040621" Nov 01 21:06:29.996032 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:29.995950 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-deff5e-bd664cbf7-wvr5w" podStartSLOduration=1.995937721 podStartE2EDuration="1.995937721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:06:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:06:29.994864388 +0000 UTC m=+2140.772504524" watchObservedRunningTime="2026-11-01 21:06:29.995937721 +0000 UTC m=+2140.773577862" Nov 01 21:06:31.144608 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:31.144574 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:31.322673 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:31.322635 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:31.479796 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:31.479698 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:31.585692 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:31.585659 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44594b37-7m92d"] Nov 01 21:06:31.590300 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:31.590284 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44594b37-7m92d" Nov 01 21:06:31.647768 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:31.647733 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44594b37-7m92d"] Nov 01 21:06:31.667651 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:31.667609 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2dpz\" (UniqueName: \"kubernetes.io/projected/5126f2ca-9d9e-47c5-ac5c-917192929e84-kube-api-access-s2dpz\") pod \"maas-api-revoke-sub-44594b37-7m92d\" (UID: \"5126f2ca-9d9e-47c5-ac5c-917192929e84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44594b37-7m92d" Nov 01 21:06:31.667843 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:31.667720 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5126f2ca-9d9e-47c5-ac5c-917192929e84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44594b37-7m92d\" (UID: \"5126f2ca-9d9e-47c5-ac5c-917192929e84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44594b37-7m92d" Nov 01 21:06:31.667943 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:31.667923 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5126f2ca-9d9e-47c5-ac5c-917192929e84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44594b37-7m92d\" (UID: \"5126f2ca-9d9e-47c5-ac5c-917192929e84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44594b37-7m92d" Nov 01 21:06:31.693216 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:31.693176 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:06:31.693216 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:31.693204 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:06:31.693444 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:31.693299 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:06:31.694651 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:31.694624 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:06:31.698437 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:31.698412 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c004f1b-mfht8"] Nov 01 21:06:31.702964 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:31.702944 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c004f1b-mfht8" Nov 01 21:06:31.757587 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:31.757558 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c004f1b-mfht8"] Nov 01 21:06:31.769514 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:31.769476 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnmvq\" (UniqueName: \"kubernetes.io/projected/f40c85e9-90d4-4d9a-a2b4-fcddafc3aa43-kube-api-access-pnmvq\") pod \"maas-api-revoke-sub-9c004f1b-mfht8\" (UID: \"f40c85e9-90d4-4d9a-a2b4-fcddafc3aa43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c004f1b-mfht8" Nov 01 21:06:31.769682 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:31.769664 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2dpz\" (UniqueName: \"kubernetes.io/projected/5126f2ca-9d9e-47c5-ac5c-917192929e84-kube-api-access-s2dpz\") pod \"maas-api-revoke-sub-44594b37-7m92d\" (UID: \"5126f2ca-9d9e-47c5-ac5c-917192929e84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44594b37-7m92d" Nov 01 21:06:31.769741 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:31.769723 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f40c85e9-90d4-4d9a-a2b4-fcddafc3aa43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c004f1b-mfht8\" (UID: \"f40c85e9-90d4-4d9a-a2b4-fcddafc3aa43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c004f1b-mfht8" Nov 01 21:06:31.769805 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:31.769766 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5126f2ca-9d9e-47c5-ac5c-917192929e84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44594b37-7m92d\" (UID: \"5126f2ca-9d9e-47c5-ac5c-917192929e84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44594b37-7m92d" Nov 01 21:06:31.769971 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:31.769947 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f40c85e9-90d4-4d9a-a2b4-fcddafc3aa43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c004f1b-mfht8\" (UID: \"f40c85e9-90d4-4d9a-a2b4-fcddafc3aa43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c004f1b-mfht8" Nov 01 21:06:31.770267 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:31.770244 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5126f2ca-9d9e-47c5-ac5c-917192929e84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44594b37-7m92d\" (UID: \"5126f2ca-9d9e-47c5-ac5c-917192929e84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44594b37-7m92d" Nov 01 21:06:31.770728 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:31.770699 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5126f2ca-9d9e-47c5-ac5c-917192929e84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44594b37-7m92d\" (UID: \"5126f2ca-9d9e-47c5-ac5c-917192929e84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44594b37-7m92d" Nov 01 21:06:31.772738 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:31.772713 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5126f2ca-9d9e-47c5-ac5c-917192929e84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44594b37-7m92d\" (UID: \"5126f2ca-9d9e-47c5-ac5c-917192929e84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44594b37-7m92d" Nov 01 21:06:31.830271 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:31.830218 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2dpz\" (UniqueName: \"kubernetes.io/projected/5126f2ca-9d9e-47c5-ac5c-917192929e84-kube-api-access-s2dpz\") pod \"maas-api-revoke-sub-44594b37-7m92d\" (UID: \"5126f2ca-9d9e-47c5-ac5c-917192929e84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44594b37-7m92d" Nov 01 21:06:31.871440 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:31.871406 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnmvq\" (UniqueName: \"kubernetes.io/projected/f40c85e9-90d4-4d9a-a2b4-fcddafc3aa43-kube-api-access-pnmvq\") pod \"maas-api-revoke-sub-9c004f1b-mfht8\" (UID: \"f40c85e9-90d4-4d9a-a2b4-fcddafc3aa43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c004f1b-mfht8" Nov 01 21:06:31.871675 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:31.871490 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f40c85e9-90d4-4d9a-a2b4-fcddafc3aa43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c004f1b-mfht8\" (UID: \"f40c85e9-90d4-4d9a-a2b4-fcddafc3aa43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c004f1b-mfht8" Nov 01 21:06:31.871675 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:31.871631 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f40c85e9-90d4-4d9a-a2b4-fcddafc3aa43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c004f1b-mfht8\" (UID: \"f40c85e9-90d4-4d9a-a2b4-fcddafc3aa43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c004f1b-mfht8" Nov 01 21:06:31.872476 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:31.872409 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f40c85e9-90d4-4d9a-a2b4-fcddafc3aa43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c004f1b-mfht8\" (UID: \"f40c85e9-90d4-4d9a-a2b4-fcddafc3aa43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c004f1b-mfht8" Nov 01 21:06:31.874973 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:31.874360 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f40c85e9-90d4-4d9a-a2b4-fcddafc3aa43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c004f1b-mfht8\" (UID: \"f40c85e9-90d4-4d9a-a2b4-fcddafc3aa43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c004f1b-mfht8" Nov 01 21:06:31.905779 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:31.905662 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44594b37-7m92d" Nov 01 21:06:31.908975 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:31.908946 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:31.929437 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:31.929412 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnmvq\" (UniqueName: \"kubernetes.io/projected/f40c85e9-90d4-4d9a-a2b4-fcddafc3aa43-kube-api-access-pnmvq\") pod \"maas-api-revoke-sub-9c004f1b-mfht8\" (UID: \"f40c85e9-90d4-4d9a-a2b4-fcddafc3aa43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c004f1b-mfht8" Nov 01 21:06:32.037990 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:32.028448 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c004f1b-mfht8" Nov 01 21:06:32.296722 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:32.296693 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44594b37-7m92d"] Nov 01 21:06:32.296722 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:06:32.296702 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5126f2ca_9d9e_47c5_ac5c_917192929e84.slice/crio-59926e0b8fe5f8b5661c18bfd35d2b5383e779c6e4fe5bf5460560db6e2bc311 WatchSource:0}: Error finding container 59926e0b8fe5f8b5661c18bfd35d2b5383e779c6e4fe5bf5460560db6e2bc311: Status 404 returned error can't find the container with id 59926e0b8fe5f8b5661c18bfd35d2b5383e779c6e4fe5bf5460560db6e2bc311 Nov 01 21:06:32.452466 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:32.452436 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c004f1b-mfht8"] Nov 01 21:06:32.453355 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:06:32.453328 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf40c85e9_90d4_4d9a_a2b4_fcddafc3aa43.slice/crio-42a255ca53c9a283872b94a05c2b7286f99045031d1938206720553648dcd492 WatchSource:0}: Error finding container 42a255ca53c9a283872b94a05c2b7286f99045031d1938206720553648dcd492: Status 404 returned error can't find the container with id 42a255ca53c9a283872b94a05c2b7286f99045031d1938206720553648dcd492 Nov 01 21:06:32.665765 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:32.665730 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44594b37-7m92d" event={"ID":"5126f2ca-9d9e-47c5-ac5c-917192929e84","Type":"ContainerStarted","Data":"59926e0b8fe5f8b5661c18bfd35d2b5383e779c6e4fe5bf5460560db6e2bc311"} Nov 01 21:06:32.667094 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:32.667063 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c004f1b-mfht8" event={"ID":"f40c85e9-90d4-4d9a-a2b4-fcddafc3aa43","Type":"ContainerStarted","Data":"42a255ca53c9a283872b94a05c2b7286f99045031d1938206720553648dcd492"} Nov 01 21:06:33.624765 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:33.624726 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:33.788072 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:33.783219 4484 generic.go:356] "Generic (PLEG): container finished" podID="5126f2ca-9d9e-47c5-ac5c-917192929e84" containerID="8ac3ed47881de2adbcc17d46222b9e82328111e0fcc6bc407b30aead717af64a" exitCode=0 Nov 01 21:06:33.788072 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:33.783339 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44594b37-7m92d" event={"ID":"5126f2ca-9d9e-47c5-ac5c-917192929e84","Type":"ContainerDied","Data":"8ac3ed47881de2adbcc17d46222b9e82328111e0fcc6bc407b30aead717af64a"} Nov 01 21:06:33.800122 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:33.799711 4484 generic.go:356] "Generic (PLEG): container finished" podID="f40c85e9-90d4-4d9a-a2b4-fcddafc3aa43" containerID="d35f26d2950ee85f4722a7efcde33d33c9c238b38ae871e4b3af23187002913f" exitCode=0 Nov 01 21:06:33.800122 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:33.799867 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c004f1b-mfht8" event={"ID":"f40c85e9-90d4-4d9a-a2b4-fcddafc3aa43","Type":"ContainerDied","Data":"d35f26d2950ee85f4722a7efcde33d33c9c238b38ae871e4b3af23187002913f"} Nov 01 21:06:33.849670 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:33.849635 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:34.165929 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:34.165840 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:34.762174 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:34.762130 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:34.977829 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:34.977795 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44594b37-7m92d" Nov 01 21:06:35.030150 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:35.030075 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:35.034551 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:35.034528 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c004f1b-mfht8" Nov 01 21:06:35.042063 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:35.042032 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5126f2ca-9d9e-47c5-ac5c-917192929e84-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5126f2ca-9d9e-47c5-ac5c-917192929e84-maas-api-openshift-ca\") pod \"5126f2ca-9d9e-47c5-ac5c-917192929e84\" (UID: \"5126f2ca-9d9e-47c5-ac5c-917192929e84\") " Nov 01 21:06:35.042183 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:35.042161 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5126f2ca-9d9e-47c5-ac5c-917192929e84-kube-api-access-s2dpz\" (UniqueName: \"kubernetes.io/projected/5126f2ca-9d9e-47c5-ac5c-917192929e84-kube-api-access-s2dpz\") pod \"5126f2ca-9d9e-47c5-ac5c-917192929e84\" (UID: \"5126f2ca-9d9e-47c5-ac5c-917192929e84\") " Nov 01 21:06:35.042309 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:35.042290 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5126f2ca-9d9e-47c5-ac5c-917192929e84-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5126f2ca-9d9e-47c5-ac5c-917192929e84-maas-api-xks-ca\") pod \"5126f2ca-9d9e-47c5-ac5c-917192929e84\" (UID: \"5126f2ca-9d9e-47c5-ac5c-917192929e84\") " Nov 01 21:06:35.042415 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:35.042391 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5126f2ca-9d9e-47c5-ac5c-917192929e84-maas-api-openshift-ca" pod "5126f2ca-9d9e-47c5-ac5c-917192929e84" (UID: "5126f2ca-9d9e-47c5-ac5c-917192929e84"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:06:35.042982 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:35.042936 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5126f2ca-9d9e-47c5-ac5c-917192929e84-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:35.044378 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:35.044354 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5126f2ca-9d9e-47c5-ac5c-917192929e84-maas-api-xks-ca" pod "5126f2ca-9d9e-47c5-ac5c-917192929e84" (UID: "5126f2ca-9d9e-47c5-ac5c-917192929e84"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:06:35.044378 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:35.044366 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5126f2ca-9d9e-47c5-ac5c-917192929e84-kube-api-access-s2dpz" pod "5126f2ca-9d9e-47c5-ac5c-917192929e84" (UID: "5126f2ca-9d9e-47c5-ac5c-917192929e84"). InnerVolumeSpecName "kube-api-access-s2dpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:06:35.143811 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:35.143691 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f40c85e9-90d4-4d9a-a2b4-fcddafc3aa43-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f40c85e9-90d4-4d9a-a2b4-fcddafc3aa43-maas-api-xks-ca\") pod \"f40c85e9-90d4-4d9a-a2b4-fcddafc3aa43\" (UID: \"f40c85e9-90d4-4d9a-a2b4-fcddafc3aa43\") " Nov 01 21:06:35.143811 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:35.143741 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f40c85e9-90d4-4d9a-a2b4-fcddafc3aa43-kube-api-access-pnmvq\" (UniqueName: \"kubernetes.io/projected/f40c85e9-90d4-4d9a-a2b4-fcddafc3aa43-kube-api-access-pnmvq\") pod \"f40c85e9-90d4-4d9a-a2b4-fcddafc3aa43\" (UID: \"f40c85e9-90d4-4d9a-a2b4-fcddafc3aa43\") " Nov 01 21:06:35.144118 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:35.143864 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f40c85e9-90d4-4d9a-a2b4-fcddafc3aa43-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f40c85e9-90d4-4d9a-a2b4-fcddafc3aa43-maas-api-openshift-ca\") pod \"f40c85e9-90d4-4d9a-a2b4-fcddafc3aa43\" (UID: \"f40c85e9-90d4-4d9a-a2b4-fcddafc3aa43\") " Nov 01 21:06:35.144694 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:35.144642 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f40c85e9-90d4-4d9a-a2b4-fcddafc3aa43-maas-api-openshift-ca" pod "f40c85e9-90d4-4d9a-a2b4-fcddafc3aa43" (UID: "f40c85e9-90d4-4d9a-a2b4-fcddafc3aa43"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:06:35.144828 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:35.144803 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2dpz\" (UniqueName: \"kubernetes.io/projected/5126f2ca-9d9e-47c5-ac5c-917192929e84-kube-api-access-s2dpz\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:35.144909 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:35.144833 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f40c85e9-90d4-4d9a-a2b4-fcddafc3aa43-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:35.144909 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:35.144850 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5126f2ca-9d9e-47c5-ac5c-917192929e84-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:35.146226 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:35.146197 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f40c85e9-90d4-4d9a-a2b4-fcddafc3aa43-maas-api-xks-ca" pod "f40c85e9-90d4-4d9a-a2b4-fcddafc3aa43" (UID: "f40c85e9-90d4-4d9a-a2b4-fcddafc3aa43"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:06:35.146376 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:35.146356 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f40c85e9-90d4-4d9a-a2b4-fcddafc3aa43-kube-api-access-pnmvq" pod "f40c85e9-90d4-4d9a-a2b4-fcddafc3aa43" (UID: "f40c85e9-90d4-4d9a-a2b4-fcddafc3aa43"). InnerVolumeSpecName "kube-api-access-pnmvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:06:35.244738 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:35.244697 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-11a069b1-fq8q6"] Nov 01 21:06:35.248780 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:35.248746 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f40c85e9-90d4-4d9a-a2b4-fcddafc3aa43-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:35.248939 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:35.248785 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnmvq\" (UniqueName: \"kubernetes.io/projected/f40c85e9-90d4-4d9a-a2b4-fcddafc3aa43-kube-api-access-pnmvq\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:35.258947 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:35.258913 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-11a069b1-fq8q6"] Nov 01 21:06:35.373714 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:35.373636 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="638b3ec7-e860-4c27-9a43-765f613a7fe7" path="/var/lib/kubelet/pods/638b3ec7-e860-4c27-9a43-765f613a7fe7/volumes" Nov 01 21:06:35.642748 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:35.642663 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:35.816882 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:35.816848 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44594b37-7m92d" Nov 01 21:06:35.817454 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:35.816855 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44594b37-7m92d" event={"ID":"5126f2ca-9d9e-47c5-ac5c-917192929e84","Type":"ContainerDied","Data":"59926e0b8fe5f8b5661c18bfd35d2b5383e779c6e4fe5bf5460560db6e2bc311"} Nov 01 21:06:35.817621 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:35.817478 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59926e0b8fe5f8b5661c18bfd35d2b5383e779c6e4fe5bf5460560db6e2bc311" Nov 01 21:06:35.818432 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:35.818408 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c004f1b-mfht8" event={"ID":"f40c85e9-90d4-4d9a-a2b4-fcddafc3aa43","Type":"ContainerDied","Data":"42a255ca53c9a283872b94a05c2b7286f99045031d1938206720553648dcd492"} Nov 01 21:06:35.818539 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:35.818448 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42a255ca53c9a283872b94a05c2b7286f99045031d1938206720553648dcd492" Nov 01 21:06:35.818539 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:35.818447 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c004f1b-mfht8" Nov 01 21:06:37.337534 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:37.337500 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:37.513019 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:37.512746 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:37.727621 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:37.727539 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6b9305-864cc97d9f-dcsdn"] Nov 01 21:06:37.728277 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:37.728262 4484 cpu_manager.go:414] "containerMap: removing container" podUID="f40c85e9-90d4-4d9a-a2b4-fcddafc3aa43" containerName="revoke-keys" Nov 01 21:06:37.728331 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:37.728278 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f40c85e9-90d4-4d9a-a2b4-fcddafc3aa43" containerName="revoke-keys" Nov 01 21:06:37.728331 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:37.728310 4484 cpu_manager.go:414] "containerMap: removing container" podUID="5126f2ca-9d9e-47c5-ac5c-917192929e84" containerName="revoke-keys" Nov 01 21:06:37.728331 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:37.728316 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5126f2ca-9d9e-47c5-ac5c-917192929e84" containerName="revoke-keys" Nov 01 21:06:37.728429 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:37.728407 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="5126f2ca-9d9e-47c5-ac5c-917192929e84" containerName="revoke-keys" Nov 01 21:06:37.728429 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:37.728422 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="f40c85e9-90d4-4d9a-a2b4-fcddafc3aa43" containerName="revoke-keys" Nov 01 21:06:37.733265 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:37.733244 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b9305-864cc97d9f-dcsdn" Nov 01 21:06:37.789663 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:37.789626 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6b9305-864cc97d9f-dcsdn"] Nov 01 21:06:37.843015 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:37.835294 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-6b9305-dockercfg-cppcw\"" Nov 01 21:06:37.843015 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:37.835389 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-6b9305\"" Nov 01 21:06:37.843015 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:37.841015 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24468902-76d5-4e4e-b459-8833f95e525d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6b9305-864cc97d9f-dcsdn\" (UID: \"24468902-76d5-4e4e-b459-8833f95e525d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b9305-864cc97d9f-dcsdn" Nov 01 21:06:37.843015 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:37.841137 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc9nl\" (UniqueName: \"kubernetes.io/projected/24468902-76d5-4e4e-b459-8833f95e525d-kube-api-access-jc9nl\") pod \"payload-pre-processing-e2e-inf-a-6b9305-864cc97d9f-dcsdn\" (UID: \"24468902-76d5-4e4e-b459-8833f95e525d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b9305-864cc97d9f-dcsdn" Nov 01 21:06:37.982074 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:37.973493 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6b9305-6964565579-hwlhv"] Nov 01 21:06:38.014060 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:37.994671 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6b9305-6964565579-hwlhv"] Nov 01 21:06:38.014060 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:37.994795 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6b9305-6964565579-hwlhv" Nov 01 21:06:38.020125 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:38.017634 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc9nl\" (UniqueName: \"kubernetes.io/projected/24468902-76d5-4e4e-b459-8833f95e525d-kube-api-access-jc9nl\") pod \"payload-pre-processing-e2e-inf-a-6b9305-864cc97d9f-dcsdn\" (UID: \"24468902-76d5-4e4e-b459-8833f95e525d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b9305-864cc97d9f-dcsdn" Nov 01 21:06:38.020125 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:38.017989 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24468902-76d5-4e4e-b459-8833f95e525d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6b9305-864cc97d9f-dcsdn\" (UID: \"24468902-76d5-4e4e-b459-8833f95e525d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b9305-864cc97d9f-dcsdn" Nov 01 21:06:38.020125 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:38.018983 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24468902-76d5-4e4e-b459-8833f95e525d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6b9305-864cc97d9f-dcsdn\" (UID: \"24468902-76d5-4e4e-b459-8833f95e525d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b9305-864cc97d9f-dcsdn" Nov 01 21:06:38.076684 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:38.076654 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc9nl\" (UniqueName: \"kubernetes.io/projected/24468902-76d5-4e4e-b459-8833f95e525d-kube-api-access-jc9nl\") pod \"payload-pre-processing-e2e-inf-a-6b9305-864cc97d9f-dcsdn\" (UID: \"24468902-76d5-4e4e-b459-8833f95e525d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b9305-864cc97d9f-dcsdn" Nov 01 21:06:38.118708 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:38.118671 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64f18742-ed7c-43e2-bd65-1bde7fe2a2c6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6b9305-6964565579-hwlhv\" (UID: \"64f18742-ed7c-43e2-bd65-1bde7fe2a2c6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6b9305-6964565579-hwlhv" Nov 01 21:06:38.118889 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:38.118724 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8klh\" (UniqueName: \"kubernetes.io/projected/64f18742-ed7c-43e2-bd65-1bde7fe2a2c6-kube-api-access-j8klh\") pod \"payload-processing-e2e-inf-a-6b9305-6964565579-hwlhv\" (UID: \"64f18742-ed7c-43e2-bd65-1bde7fe2a2c6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6b9305-6964565579-hwlhv" Nov 01 21:06:38.219984 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:38.219937 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64f18742-ed7c-43e2-bd65-1bde7fe2a2c6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6b9305-6964565579-hwlhv\" (UID: \"64f18742-ed7c-43e2-bd65-1bde7fe2a2c6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6b9305-6964565579-hwlhv" Nov 01 21:06:38.220242 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:38.220033 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8klh\" (UniqueName: \"kubernetes.io/projected/64f18742-ed7c-43e2-bd65-1bde7fe2a2c6-kube-api-access-j8klh\") pod \"payload-processing-e2e-inf-a-6b9305-6964565579-hwlhv\" (UID: \"64f18742-ed7c-43e2-bd65-1bde7fe2a2c6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6b9305-6964565579-hwlhv" Nov 01 21:06:38.220685 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:38.220664 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64f18742-ed7c-43e2-bd65-1bde7fe2a2c6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6b9305-6964565579-hwlhv\" (UID: \"64f18742-ed7c-43e2-bd65-1bde7fe2a2c6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6b9305-6964565579-hwlhv" Nov 01 21:06:38.277596 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:38.277561 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8klh\" (UniqueName: \"kubernetes.io/projected/64f18742-ed7c-43e2-bd65-1bde7fe2a2c6-kube-api-access-j8klh\") pod \"payload-processing-e2e-inf-a-6b9305-6964565579-hwlhv\" (UID: \"64f18742-ed7c-43e2-bd65-1bde7fe2a2c6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6b9305-6964565579-hwlhv" Nov 01 21:06:38.320911 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:38.320879 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6b9305-6964565579-hwlhv" Nov 01 21:06:38.355472 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:38.355313 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b9305-864cc97d9f-dcsdn" Nov 01 21:06:38.409138 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:38.408566 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:38.748794 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:38.748762 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6b9305-6964565579-hwlhv"] Nov 01 21:06:38.755014 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:06:38.754916 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64f18742_ed7c_43e2_bd65_1bde7fe2a2c6.slice/crio-4084453bc02659c09d2b48d00ae3871eda8b20976873eb9074289bdf27ffb588 WatchSource:0}: Error finding container 4084453bc02659c09d2b48d00ae3871eda8b20976873eb9074289bdf27ffb588: Status 404 returned error can't find the container with id 4084453bc02659c09d2b48d00ae3871eda8b20976873eb9074289bdf27ffb588 Nov 01 21:06:38.797766 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:06:38.797729 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24468902_76d5_4e4e_b459_8833f95e525d.slice/crio-6ea842b4c9befd3d305fd831594223b92005b010442269927a5c46823c26ad63 WatchSource:0}: Error finding container 6ea842b4c9befd3d305fd831594223b92005b010442269927a5c46823c26ad63: Status 404 returned error can't find the container with id 6ea842b4c9befd3d305fd831594223b92005b010442269927a5c46823c26ad63 Nov 01 21:06:38.799404 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:38.799378 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6b9305-864cc97d9f-dcsdn"] Nov 01 21:06:39.037805 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:39.037701 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6b9305-6964565579-hwlhv" event={"ID":"64f18742-ed7c-43e2-bd65-1bde7fe2a2c6","Type":"ContainerStarted","Data":"902674b97e02d910395e0957f4baaa91cebf162221f610eadde4d013237772cd"} Nov 01 21:06:39.037805 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:39.037772 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6b9305-6964565579-hwlhv" event={"ID":"64f18742-ed7c-43e2-bd65-1bde7fe2a2c6","Type":"ContainerStarted","Data":"4084453bc02659c09d2b48d00ae3871eda8b20976873eb9074289bdf27ffb588"} Nov 01 21:06:39.038089 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:39.037961 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-6b9305-6964565579-hwlhv" Nov 01 21:06:39.039766 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:39.039739 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b9305-864cc97d9f-dcsdn" event={"ID":"24468902-76d5-4e4e-b459-8833f95e525d","Type":"ContainerStarted","Data":"8e7e09e3c52d3fe2dda04a21dd1997abe5fd41832f78f1d38cd1943ddc11462d"} Nov 01 21:06:39.039905 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:39.039790 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b9305-864cc97d9f-dcsdn" event={"ID":"24468902-76d5-4e4e-b459-8833f95e525d","Type":"ContainerStarted","Data":"6ea842b4c9befd3d305fd831594223b92005b010442269927a5c46823c26ad63"} Nov 01 21:06:39.039905 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:39.039840 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b9305-864cc97d9f-dcsdn" Nov 01 21:06:39.155419 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:39.155366 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-6b9305-6964565579-hwlhv" podStartSLOduration=2.155351999 podStartE2EDuration="2.155351999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:06:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:06:39.152042831 +0000 UTC m=+2149.929682978" watchObservedRunningTime="2026-11-01 21:06:39.155351999 +0000 UTC m=+2149.932992133" Nov 01 21:06:39.270134 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:39.270085 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b9305-864cc97d9f-dcsdn" podStartSLOduration=2.270071048 podStartE2EDuration="2.270071048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:06:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:06:39.26639816 +0000 UTC m=+2150.044038298" watchObservedRunningTime="2026-11-01 21:06:39.270071048 +0000 UTC m=+2150.047711182" Nov 01 21:06:40.029566 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:40.029530 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:40.292266 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:40.292185 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:40.446260 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:40.446222 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:40.496333 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:40.496303 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b27e66-9g52w"] Nov 01 21:06:40.547046 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:40.546939 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b27e66-9g52w"] Nov 01 21:06:40.648935 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:40.648908 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-deff5e-bd664cbf7-wvr5w" Nov 01 21:06:40.649125 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:40.648985 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-deff5e-77db945ff9-jtzp7" Nov 01 21:06:40.672319 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:40.672284 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:40.927242 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:40.927148 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:41.375177 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:41.375146 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9080b783-c895-4460-aa20-467c0ace8c3f" path="/var/lib/kubelet/pods/9080b783-c895-4460-aa20-467c0ace8c3f/volumes" Nov 01 21:06:45.195473 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:45.195436 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:45.354486 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:45.354450 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:45.768869 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:06:45.768825 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:06:46.514177 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:46.514144 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:46.742991 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:46.742956 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:46.898601 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:46.898472 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl"] Nov 01 21:06:46.898845 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:46.898812 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl" podUID="836acdb1-4e39-472f-ba72-a07d61f977bf" containerName="istio-proxy" containerID="cri-o://d48d997a5bf4c9ad97bff1333c606b77d823d57bc7069382150aa38f099e22d7" gracePeriod=30 Nov 01 21:06:46.955666 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:46.955623 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:47.742169 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:47.742131 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d08fd0-9z2m5"] Nov 01 21:06:47.746817 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:47.746783 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d08fd0-9z2m5" Nov 01 21:06:47.801415 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:47.801387 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d08fd0-9z2m5"] Nov 01 21:06:47.847472 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:47.847435 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:06:47.847654 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:47.847536 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:06:47.847654 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:47.847583 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:06:47.847654 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:47.847601 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:06:47.888679 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:47.888644 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71efab02-15fb-41db-87d8-251a4cf92947-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d08fd0-9z2m5\" (UID: \"71efab02-15fb-41db-87d8-251a4cf92947\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d08fd0-9z2m5" Nov 01 21:06:47.888872 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:47.888764 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx6v6\" (UniqueName: \"kubernetes.io/projected/71efab02-15fb-41db-87d8-251a4cf92947-kube-api-access-bx6v6\") pod \"maas-api-revoke-keys-e2e-api-b-d08fd0-9z2m5\" (UID: \"71efab02-15fb-41db-87d8-251a4cf92947\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d08fd0-9z2m5" Nov 01 21:06:47.888872 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:47.888797 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71efab02-15fb-41db-87d8-251a4cf92947-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d08fd0-9z2m5\" (UID: \"71efab02-15fb-41db-87d8-251a4cf92947\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d08fd0-9z2m5" Nov 01 21:06:47.990787 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:47.990750 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx6v6\" (UniqueName: \"kubernetes.io/projected/71efab02-15fb-41db-87d8-251a4cf92947-kube-api-access-bx6v6\") pod \"maas-api-revoke-keys-e2e-api-b-d08fd0-9z2m5\" (UID: \"71efab02-15fb-41db-87d8-251a4cf92947\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d08fd0-9z2m5" Nov 01 21:06:47.990787 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:47.990792 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71efab02-15fb-41db-87d8-251a4cf92947-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d08fd0-9z2m5\" (UID: \"71efab02-15fb-41db-87d8-251a4cf92947\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d08fd0-9z2m5" Nov 01 21:06:47.990983 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:47.990957 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71efab02-15fb-41db-87d8-251a4cf92947-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d08fd0-9z2m5\" (UID: \"71efab02-15fb-41db-87d8-251a4cf92947\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d08fd0-9z2m5" Nov 01 21:06:47.991645 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:47.991618 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71efab02-15fb-41db-87d8-251a4cf92947-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d08fd0-9z2m5\" (UID: \"71efab02-15fb-41db-87d8-251a4cf92947\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d08fd0-9z2m5" Nov 01 21:06:47.993296 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:47.993251 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71efab02-15fb-41db-87d8-251a4cf92947-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d08fd0-9z2m5\" (UID: \"71efab02-15fb-41db-87d8-251a4cf92947\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d08fd0-9z2m5" Nov 01 21:06:48.049303 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:48.049275 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx6v6\" (UniqueName: \"kubernetes.io/projected/71efab02-15fb-41db-87d8-251a4cf92947-kube-api-access-bx6v6\") pod \"maas-api-revoke-keys-e2e-api-b-d08fd0-9z2m5\" (UID: \"71efab02-15fb-41db-87d8-251a4cf92947\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d08fd0-9z2m5" Nov 01 21:06:48.062177 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:48.062148 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d08fd0-9z2m5" Nov 01 21:06:48.443624 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:48.443583 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d08fd0-9z2m5"] Nov 01 21:06:48.444020 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:06:48.443983 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71efab02_15fb_41db_87d8_251a4cf92947.slice/crio-67dc0f33cd4bd4bbdf0b8d822f41875ed8dea7021841c02712d072a224a5cf2a WatchSource:0}: Error finding container 67dc0f33cd4bd4bbdf0b8d822f41875ed8dea7021841c02712d072a224a5cf2a: Status 404 returned error can't find the container with id 67dc0f33cd4bd4bbdf0b8d822f41875ed8dea7021841c02712d072a224a5cf2a Nov 01 21:06:49.159632 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:49.159596 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d08fd0-9z2m5" event={"ID":"71efab02-15fb-41db-87d8-251a4cf92947","Type":"ContainerStarted","Data":"68209950f22a2dd368f2af4e2c4d23b9631553de463a12d28c6e5c1069c9b34f"} Nov 01 21:06:49.160145 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:49.159662 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d08fd0-9z2m5" event={"ID":"71efab02-15fb-41db-87d8-251a4cf92947","Type":"ContainerStarted","Data":"67dc0f33cd4bd4bbdf0b8d822f41875ed8dea7021841c02712d072a224a5cf2a"} Nov 01 21:06:49.275523 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:49.275473 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d08fd0-9z2m5" podStartSLOduration=2.275458507 podStartE2EDuration="2.275458507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:06:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:06:49.273612356 +0000 UTC m=+2160.051252514" watchObservedRunningTime="2026-11-01 21:06:49.275458507 +0000 UTC m=+2160.053098642" Nov 01 21:06:50.053015 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:50.052970 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b9305-864cc97d9f-dcsdn" Nov 01 21:06:50.053257 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:50.053072 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-6b9305-6964565579-hwlhv" Nov 01 21:06:50.168621 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:50.168586 4484 generic.go:356] "Generic (PLEG): container finished" podID="71efab02-15fb-41db-87d8-251a4cf92947" containerID="68209950f22a2dd368f2af4e2c4d23b9631553de463a12d28c6e5c1069c9b34f" exitCode=0 Nov 01 21:06:50.169084 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:50.168649 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d08fd0-9z2m5" event={"ID":"71efab02-15fb-41db-87d8-251a4cf92947","Type":"ContainerDied","Data":"68209950f22a2dd368f2af4e2c4d23b9631553de463a12d28c6e5c1069c9b34f"} Nov 01 21:06:50.303156 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:50.303079 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:51.356983 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:51.356956 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d08fd0-9z2m5" Nov 01 21:06:51.459797 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:51.459711 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71efab02-15fb-41db-87d8-251a4cf92947-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71efab02-15fb-41db-87d8-251a4cf92947-maas-api-xks-ca\") pod \"71efab02-15fb-41db-87d8-251a4cf92947\" (UID: \"71efab02-15fb-41db-87d8-251a4cf92947\") " Nov 01 21:06:51.459958 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:51.459892 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71efab02-15fb-41db-87d8-251a4cf92947-kube-api-access-bx6v6\" (UniqueName: \"kubernetes.io/projected/71efab02-15fb-41db-87d8-251a4cf92947-kube-api-access-bx6v6\") pod \"71efab02-15fb-41db-87d8-251a4cf92947\" (UID: \"71efab02-15fb-41db-87d8-251a4cf92947\") " Nov 01 21:06:51.459958 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:51.459938 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71efab02-15fb-41db-87d8-251a4cf92947-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71efab02-15fb-41db-87d8-251a4cf92947-maas-api-openshift-ca\") pod \"71efab02-15fb-41db-87d8-251a4cf92947\" (UID: \"71efab02-15fb-41db-87d8-251a4cf92947\") " Nov 01 21:06:51.460382 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:51.460345 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71efab02-15fb-41db-87d8-251a4cf92947-maas-api-openshift-ca" pod "71efab02-15fb-41db-87d8-251a4cf92947" (UID: "71efab02-15fb-41db-87d8-251a4cf92947"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:06:51.460671 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:51.460654 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71efab02-15fb-41db-87d8-251a4cf92947-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:51.462038 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:51.462009 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71efab02-15fb-41db-87d8-251a4cf92947-maas-api-xks-ca" pod "71efab02-15fb-41db-87d8-251a4cf92947" (UID: "71efab02-15fb-41db-87d8-251a4cf92947"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:06:51.462172 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:51.462152 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71efab02-15fb-41db-87d8-251a4cf92947-kube-api-access-bx6v6" pod "71efab02-15fb-41db-87d8-251a4cf92947" (UID: "71efab02-15fb-41db-87d8-251a4cf92947"). InnerVolumeSpecName "kube-api-access-bx6v6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:06:51.561652 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:51.561614 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx6v6\" (UniqueName: \"kubernetes.io/projected/71efab02-15fb-41db-87d8-251a4cf92947-kube-api-access-bx6v6\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:51.561652 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:51.561650 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71efab02-15fb-41db-87d8-251a4cf92947-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:52.160590 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:52.160508 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl" Nov 01 21:06:52.168710 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:52.168679 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/836acdb1-4e39-472f-ba72-a07d61f977bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/836acdb1-4e39-472f-ba72-a07d61f977bf-istio-podinfo\") pod \"836acdb1-4e39-472f-ba72-a07d61f977bf\" (UID: \"836acdb1-4e39-472f-ba72-a07d61f977bf\") " Nov 01 21:06:52.168871 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:52.168753 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/836acdb1-4e39-472f-ba72-a07d61f977bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/836acdb1-4e39-472f-ba72-a07d61f977bf-credential-socket\") pod \"836acdb1-4e39-472f-ba72-a07d61f977bf\" (UID: \"836acdb1-4e39-472f-ba72-a07d61f977bf\") " Nov 01 21:06:52.168871 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:52.168845 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/836acdb1-4e39-472f-ba72-a07d61f977bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/836acdb1-4e39-472f-ba72-a07d61f977bf-istio-data\") pod \"836acdb1-4e39-472f-ba72-a07d61f977bf\" (UID: \"836acdb1-4e39-472f-ba72-a07d61f977bf\") " Nov 01 21:06:52.169003 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:52.168942 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/836acdb1-4e39-472f-ba72-a07d61f977bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/836acdb1-4e39-472f-ba72-a07d61f977bf-istio-envoy\") pod \"836acdb1-4e39-472f-ba72-a07d61f977bf\" (UID: \"836acdb1-4e39-472f-ba72-a07d61f977bf\") " Nov 01 21:06:52.169138 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:52.169096 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/836acdb1-4e39-472f-ba72-a07d61f977bf-credential-socket" pod "836acdb1-4e39-472f-ba72-a07d61f977bf" (UID: "836acdb1-4e39-472f-ba72-a07d61f977bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:06:52.169217 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:52.169131 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/836acdb1-4e39-472f-ba72-a07d61f977bf-kube-api-access-bnzn6\" (UniqueName: \"kubernetes.io/projected/836acdb1-4e39-472f-ba72-a07d61f977bf-kube-api-access-bnzn6\") pod \"836acdb1-4e39-472f-ba72-a07d61f977bf\" (UID: \"836acdb1-4e39-472f-ba72-a07d61f977bf\") " Nov 01 21:06:52.169217 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:52.169171 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/836acdb1-4e39-472f-ba72-a07d61f977bf-istio-data" pod "836acdb1-4e39-472f-ba72-a07d61f977bf" (UID: "836acdb1-4e39-472f-ba72-a07d61f977bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:06:52.169324 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:52.169303 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/836acdb1-4e39-472f-ba72-a07d61f977bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/836acdb1-4e39-472f-ba72-a07d61f977bf-istiod-ca-cert\") pod \"836acdb1-4e39-472f-ba72-a07d61f977bf\" (UID: \"836acdb1-4e39-472f-ba72-a07d61f977bf\") " Nov 01 21:06:52.169394 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:52.169379 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/836acdb1-4e39-472f-ba72-a07d61f977bf-istio-token\" (UniqueName: \"kubernetes.io/projected/836acdb1-4e39-472f-ba72-a07d61f977bf-istio-token\") pod \"836acdb1-4e39-472f-ba72-a07d61f977bf\" (UID: \"836acdb1-4e39-472f-ba72-a07d61f977bf\") " Nov 01 21:06:52.169457 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:52.169434 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/836acdb1-4e39-472f-ba72-a07d61f977bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/836acdb1-4e39-472f-ba72-a07d61f977bf-workload-socket\") pod \"836acdb1-4e39-472f-ba72-a07d61f977bf\" (UID: \"836acdb1-4e39-472f-ba72-a07d61f977bf\") " Nov 01 21:06:52.169517 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:52.169491 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/836acdb1-4e39-472f-ba72-a07d61f977bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/836acdb1-4e39-472f-ba72-a07d61f977bf-workload-certs\") pod \"836acdb1-4e39-472f-ba72-a07d61f977bf\" (UID: \"836acdb1-4e39-472f-ba72-a07d61f977bf\") " Nov 01 21:06:52.169676 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:52.169656 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/836acdb1-4e39-472f-ba72-a07d61f977bf-istiod-ca-cert" pod "836acdb1-4e39-472f-ba72-a07d61f977bf" (UID: "836acdb1-4e39-472f-ba72-a07d61f977bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:06:52.170139 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:52.170086 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/836acdb1-4e39-472f-ba72-a07d61f977bf-workload-socket" pod "836acdb1-4e39-472f-ba72-a07d61f977bf" (UID: "836acdb1-4e39-472f-ba72-a07d61f977bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:06:52.170270 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:52.170171 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/836acdb1-4e39-472f-ba72-a07d61f977bf-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:52.170270 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:52.170190 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/836acdb1-4e39-472f-ba72-a07d61f977bf-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:52.170270 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:52.170203 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/836acdb1-4e39-472f-ba72-a07d61f977bf-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:52.170507 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:52.170375 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/836acdb1-4e39-472f-ba72-a07d61f977bf-workload-certs" pod "836acdb1-4e39-472f-ba72-a07d61f977bf" (UID: "836acdb1-4e39-472f-ba72-a07d61f977bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:06:52.171242 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:52.171224 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/836acdb1-4e39-472f-ba72-a07d61f977bf-istio-envoy" pod "836acdb1-4e39-472f-ba72-a07d61f977bf" (UID: "836acdb1-4e39-472f-ba72-a07d61f977bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:06:52.171329 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:52.171311 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/836acdb1-4e39-472f-ba72-a07d61f977bf-istio-podinfo" pod "836acdb1-4e39-472f-ba72-a07d61f977bf" (UID: "836acdb1-4e39-472f-ba72-a07d61f977bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:06:52.171466 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:52.171436 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/836acdb1-4e39-472f-ba72-a07d61f977bf-kube-api-access-bnzn6" pod "836acdb1-4e39-472f-ba72-a07d61f977bf" (UID: "836acdb1-4e39-472f-ba72-a07d61f977bf"). InnerVolumeSpecName "kube-api-access-bnzn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:06:52.171756 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:52.171740 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/836acdb1-4e39-472f-ba72-a07d61f977bf-istio-token" pod "836acdb1-4e39-472f-ba72-a07d61f977bf" (UID: "836acdb1-4e39-472f-ba72-a07d61f977bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:06:52.194418 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:52.194383 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d08fd0-9z2m5" Nov 01 21:06:52.194602 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:52.194391 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d08fd0-9z2m5" event={"ID":"71efab02-15fb-41db-87d8-251a4cf92947","Type":"ContainerDied","Data":"67dc0f33cd4bd4bbdf0b8d822f41875ed8dea7021841c02712d072a224a5cf2a"} Nov 01 21:06:52.194602 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:52.194513 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67dc0f33cd4bd4bbdf0b8d822f41875ed8dea7021841c02712d072a224a5cf2a" Nov 01 21:06:52.196509 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:52.196478 4484 generic.go:356] "Generic (PLEG): container finished" podID="836acdb1-4e39-472f-ba72-a07d61f977bf" containerID="d48d997a5bf4c9ad97bff1333c606b77d823d57bc7069382150aa38f099e22d7" exitCode=0 Nov 01 21:06:52.196509 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:52.196531 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl" Nov 01 21:06:52.196748 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:52.196569 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl" event={"ID":"836acdb1-4e39-472f-ba72-a07d61f977bf","Type":"ContainerDied","Data":"d48d997a5bf4c9ad97bff1333c606b77d823d57bc7069382150aa38f099e22d7"} Nov 01 21:06:52.196748 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:52.196621 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl" event={"ID":"836acdb1-4e39-472f-ba72-a07d61f977bf","Type":"ContainerDied","Data":"0224f3bba9f169c15d5cc0f2cfa3c1294a84f005fcf0539df7a6faf804dd94f1"} Nov 01 21:06:52.196748 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:52.196645 4484 scope.go:122] "RemoveContainer" containerID="d48d997a5bf4c9ad97bff1333c606b77d823d57bc7069382150aa38f099e22d7" Nov 01 21:06:52.209366 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:52.209345 4484 scope.go:122] "RemoveContainer" containerID="2bd1c28a8eb5bb55790b360ae048808b234637ab065e345fb7be2188b90906b1" Nov 01 21:06:52.219777 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:52.219754 4484 scope.go:122] "RemoveContainer" containerID="d48d997a5bf4c9ad97bff1333c606b77d823d57bc7069382150aa38f099e22d7" Nov 01 21:06:52.220194 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:06:52.220170 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d48d997a5bf4c9ad97bff1333c606b77d823d57bc7069382150aa38f099e22d7\": container with ID starting with d48d997a5bf4c9ad97bff1333c606b77d823d57bc7069382150aa38f099e22d7 not found: ID does not exist" containerID="d48d997a5bf4c9ad97bff1333c606b77d823d57bc7069382150aa38f099e22d7" Nov 01 21:06:52.220306 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:52.220204 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d48d997a5bf4c9ad97bff1333c606b77d823d57bc7069382150aa38f099e22d7"} err="failed to get container status \"d48d997a5bf4c9ad97bff1333c606b77d823d57bc7069382150aa38f099e22d7\": rpc error: code = NotFound desc = could not find container \"d48d997a5bf4c9ad97bff1333c606b77d823d57bc7069382150aa38f099e22d7\": container with ID starting with d48d997a5bf4c9ad97bff1333c606b77d823d57bc7069382150aa38f099e22d7 not found: ID does not exist" Nov 01 21:06:52.220306 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:52.220229 4484 scope.go:122] "RemoveContainer" containerID="2bd1c28a8eb5bb55790b360ae048808b234637ab065e345fb7be2188b90906b1" Nov 01 21:06:52.220646 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:06:52.220578 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bd1c28a8eb5bb55790b360ae048808b234637ab065e345fb7be2188b90906b1\": container with ID starting with 2bd1c28a8eb5bb55790b360ae048808b234637ab065e345fb7be2188b90906b1 not found: ID does not exist" containerID="2bd1c28a8eb5bb55790b360ae048808b234637ab065e345fb7be2188b90906b1" Nov 01 21:06:52.220646 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:52.220609 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bd1c28a8eb5bb55790b360ae048808b234637ab065e345fb7be2188b90906b1"} err="failed to get container status \"2bd1c28a8eb5bb55790b360ae048808b234637ab065e345fb7be2188b90906b1\": rpc error: code = NotFound desc = could not find container \"2bd1c28a8eb5bb55790b360ae048808b234637ab065e345fb7be2188b90906b1\": container with ID starting with 2bd1c28a8eb5bb55790b360ae048808b234637ab065e345fb7be2188b90906b1 not found: ID does not exist" Nov 01 21:06:52.271722 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:52.271692 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/836acdb1-4e39-472f-ba72-a07d61f977bf-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:52.271722 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:52.271721 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnzn6\" (UniqueName: \"kubernetes.io/projected/836acdb1-4e39-472f-ba72-a07d61f977bf-kube-api-access-bnzn6\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:52.271952 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:52.271736 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/836acdb1-4e39-472f-ba72-a07d61f977bf-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:52.271952 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:52.271749 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/836acdb1-4e39-472f-ba72-a07d61f977bf-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:52.271952 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:52.271760 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/836acdb1-4e39-472f-ba72-a07d61f977bf-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:52.271952 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:52.271768 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/836acdb1-4e39-472f-ba72-a07d61f977bf-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:06:52.496565 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:52.496524 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl"] Nov 01 21:06:52.500382 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:52.500352 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-b27e66-openshift-default-5cccc745f7-nkvtl"] Nov 01 21:06:53.373886 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:53.373856 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="836acdb1-4e39-472f-ba72-a07d61f977bf" path="/var/lib/kubelet/pods/836acdb1-4e39-472f-ba72-a07d61f977bf/volumes" Nov 01 21:06:55.532439 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:55.532331 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:59.875693 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:06:59.875657 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d08fd0-84d7f997dc-6zrkr"] Nov 01 21:07:00.196679 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:00.196581 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:07:00.378053 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:00.377982 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:07:00.533543 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:00.533510 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:07:00.542534 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:00.542507 4484 scope.go:122] "RemoveContainer" containerID="38608d0602c90d2bf9f6e6c0ccba139cdcef2fad47a302f59fdcd10933b1480c" Nov 01 21:07:00.700120 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:00.700086 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:07:01.915934 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:01.915896 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z4sdv"] Nov 01 21:07:01.916469 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:01.916120 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z4sdv" podUID="243b9a06-d07a-4d14-abe2-ad2239dd45fe" containerName="istio-proxy" containerID="cri-o://341a09a23ee439b432b557cc11272c45d10d5ae8721d87979f1fcfdd79c91bf9" gracePeriod=30 Nov 01 21:07:02.090246 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:02.090213 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj"] Nov 01 21:07:02.090483 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:02.090458 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj" podUID="8e7768b7-6fa1-4408-a1f5-2862a8a7d907" containerName="istio-proxy" containerID="cri-o://613085aea41e537c2798b514085f2da5f17b07a036ebc4af8f94dbdc48b023a9" gracePeriod=30 Nov 01 21:07:02.095522 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:02.095487 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk"] Nov 01 21:07:02.095745 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:02.095717 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk" podUID="17927c6e-7e23-4717-be79-d2569ebae4ca" containerName="istio-proxy" containerID="cri-o://a45831dcad5f6438e76a52c65838f3d14b088583f334964e75b98f643d744308" gracePeriod=30 Nov 01 21:07:02.400454 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:02.400414 4484 patch_prober.go:29] interesting pod/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 21:07:02.400636 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:02.400481 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk" podUID="17927c6e-7e23-4717-be79-d2569ebae4ca" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 21:07:04.876474 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:04.876420 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d08fd0-84d7f997dc-6zrkr" podUID="2f7a17d2-43cc-4001-814a-8446e8054743" containerName="payload-pre-processing" containerID="cri-o://c3ee14f5d5bc4f886ed639c213dd54da775db2a9961215e238729198c83c0dc6" gracePeriod=25 Nov 01 21:07:05.166494 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:05.166457 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d08fd0-84d7f997dc-6zrkr" Nov 01 21:07:05.263412 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:05.263382 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f7a17d2-43cc-4001-814a-8446e8054743-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f7a17d2-43cc-4001-814a-8446e8054743-plugins-config-volume\") pod \"2f7a17d2-43cc-4001-814a-8446e8054743\" (UID: \"2f7a17d2-43cc-4001-814a-8446e8054743\") " Nov 01 21:07:05.263546 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:05.263533 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f7a17d2-43cc-4001-814a-8446e8054743-kube-api-access-csts4\" (UniqueName: \"kubernetes.io/projected/2f7a17d2-43cc-4001-814a-8446e8054743-kube-api-access-csts4\") pod \"2f7a17d2-43cc-4001-814a-8446e8054743\" (UID: \"2f7a17d2-43cc-4001-814a-8446e8054743\") " Nov 01 21:07:05.263758 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:05.263736 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f7a17d2-43cc-4001-814a-8446e8054743-plugins-config-volume" pod "2f7a17d2-43cc-4001-814a-8446e8054743" (UID: "2f7a17d2-43cc-4001-814a-8446e8054743"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:07:05.264346 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:05.264324 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f7a17d2-43cc-4001-814a-8446e8054743-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:05.266041 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:05.265964 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f7a17d2-43cc-4001-814a-8446e8054743-kube-api-access-csts4" pod "2f7a17d2-43cc-4001-814a-8446e8054743" (UID: "2f7a17d2-43cc-4001-814a-8446e8054743"). InnerVolumeSpecName "kube-api-access-csts4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:07:05.326339 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:05.326301 4484 generic.go:356] "Generic (PLEG): container finished" podID="8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff" containerID="093879cd3f9e7ce1ed44b391516fca5207ccb2ff263dfbb60a9a50f436e9d87c" exitCode=0 Nov 01 21:07:05.326538 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:05.326329 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc" event={"ID":"8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff","Type":"ContainerDied","Data":"093879cd3f9e7ce1ed44b391516fca5207ccb2ff263dfbb60a9a50f436e9d87c"} Nov 01 21:07:05.326955 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:05.326930 4484 scope.go:122] "RemoveContainer" containerID="093879cd3f9e7ce1ed44b391516fca5207ccb2ff263dfbb60a9a50f436e9d87c" Nov 01 21:07:05.327986 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:05.327966 4484 generic.go:356] "Generic (PLEG): container finished" podID="2f7a17d2-43cc-4001-814a-8446e8054743" containerID="c3ee14f5d5bc4f886ed639c213dd54da775db2a9961215e238729198c83c0dc6" exitCode=0 Nov 01 21:07:05.328101 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:05.328042 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d08fd0-84d7f997dc-6zrkr" Nov 01 21:07:05.328157 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:05.328143 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d08fd0-84d7f997dc-6zrkr" event={"ID":"2f7a17d2-43cc-4001-814a-8446e8054743","Type":"ContainerDied","Data":"c3ee14f5d5bc4f886ed639c213dd54da775db2a9961215e238729198c83c0dc6"} Nov 01 21:07:05.328217 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:05.328193 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d08fd0-84d7f997dc-6zrkr" event={"ID":"2f7a17d2-43cc-4001-814a-8446e8054743","Type":"ContainerDied","Data":"73340e02325e2a06e413afa032865a6516462d0940c1610895aa10ee6e936697"} Nov 01 21:07:05.328273 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:05.328223 4484 scope.go:122] "RemoveContainer" containerID="c3ee14f5d5bc4f886ed639c213dd54da775db2a9961215e238729198c83c0dc6" Nov 01 21:07:05.329173 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:05.329131 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:07:05.329299 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:05.329223 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:07:05.329299 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:05.329275 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:07:05.345851 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:05.345832 4484 scope.go:122] "RemoveContainer" containerID="c3ee14f5d5bc4f886ed639c213dd54da775db2a9961215e238729198c83c0dc6" Nov 01 21:07:05.346212 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:07:05.346191 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3ee14f5d5bc4f886ed639c213dd54da775db2a9961215e238729198c83c0dc6\": container with ID starting with c3ee14f5d5bc4f886ed639c213dd54da775db2a9961215e238729198c83c0dc6 not found: ID does not exist" containerID="c3ee14f5d5bc4f886ed639c213dd54da775db2a9961215e238729198c83c0dc6" Nov 01 21:07:05.346317 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:05.346220 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3ee14f5d5bc4f886ed639c213dd54da775db2a9961215e238729198c83c0dc6"} err="failed to get container status \"c3ee14f5d5bc4f886ed639c213dd54da775db2a9961215e238729198c83c0dc6\": rpc error: code = NotFound desc = could not find container \"c3ee14f5d5bc4f886ed639c213dd54da775db2a9961215e238729198c83c0dc6\": container with ID starting with c3ee14f5d5bc4f886ed639c213dd54da775db2a9961215e238729198c83c0dc6 not found: ID does not exist" Nov 01 21:07:05.365990 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:05.365955 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csts4\" (UniqueName: \"kubernetes.io/projected/2f7a17d2-43cc-4001-814a-8446e8054743-kube-api-access-csts4\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:05.569990 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:05.569954 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d08fd0-84d7f997dc-6zrkr"] Nov 01 21:07:05.573951 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:05.573924 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d08fd0-84d7f997dc-6zrkr"] Nov 01 21:07:05.941531 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:05.941465 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:07:06.338056 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:06.337985 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc" event={"ID":"8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff","Type":"ContainerStarted","Data":"c040b0424a73a7fc9b8c22f4e7bfcb21e09d8f33a59865c6a6db3128c4ef431e"} Nov 01 21:07:06.397617 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:06.397581 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-z4sdv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 21:07:06.397818 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:06.397636 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z4sdv" podUID="243b9a06-d07a-4d14-abe2-ad2239dd45fe" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 21:07:06.552531 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:06.552487 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc" Nov 01 21:07:06.553755 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:06.553726 4484 patch_prober.go:29] interesting pod/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Nov 01 21:07:06.553903 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:06.553775 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc" podUID="8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Nov 01 21:07:07.355400 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.353965 4484 generic.go:356] "Generic (PLEG): container finished" podID="8e7768b7-6fa1-4408-a1f5-2862a8a7d907" containerID="613085aea41e537c2798b514085f2da5f17b07a036ebc4af8f94dbdc48b023a9" exitCode=0 Nov 01 21:07:07.355400 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.354166 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj" event={"ID":"8e7768b7-6fa1-4408-a1f5-2862a8a7d907","Type":"ContainerDied","Data":"613085aea41e537c2798b514085f2da5f17b07a036ebc4af8f94dbdc48b023a9"} Nov 01 21:07:07.355984 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.355867 4484 generic.go:356] "Generic (PLEG): container finished" podID="17927c6e-7e23-4717-be79-d2569ebae4ca" containerID="a45831dcad5f6438e76a52c65838f3d14b088583f334964e75b98f643d744308" exitCode=0 Nov 01 21:07:07.355984 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.355938 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk" event={"ID":"17927c6e-7e23-4717-be79-d2569ebae4ca","Type":"ContainerDied","Data":"a45831dcad5f6438e76a52c65838f3d14b088583f334964e75b98f643d744308"} Nov 01 21:07:07.372628 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.371967 4484 generic.go:356] "Generic (PLEG): container finished" podID="243b9a06-d07a-4d14-abe2-ad2239dd45fe" containerID="341a09a23ee439b432b557cc11272c45d10d5ae8721d87979f1fcfdd79c91bf9" exitCode=0 Nov 01 21:07:07.382720 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.382685 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f7a17d2-43cc-4001-814a-8446e8054743" path="/var/lib/kubelet/pods/2f7a17d2-43cc-4001-814a-8446e8054743/volumes" Nov 01 21:07:07.383188 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.383158 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z4sdv" event={"ID":"243b9a06-d07a-4d14-abe2-ad2239dd45fe","Type":"ContainerDied","Data":"341a09a23ee439b432b557cc11272c45d10d5ae8721d87979f1fcfdd79c91bf9"} Nov 01 21:07:07.461466 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.461204 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj" Nov 01 21:07:07.469155 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.469085 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk" Nov 01 21:07:07.497730 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.497540 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17927c6e-7e23-4717-be79-d2569ebae4ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17927c6e-7e23-4717-be79-d2569ebae4ca-istio-envoy\") pod \"17927c6e-7e23-4717-be79-d2569ebae4ca\" (UID: \"17927c6e-7e23-4717-be79-d2569ebae4ca\") " Nov 01 21:07:07.497730 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.497589 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-kube-api-access-m6cxt\" (UniqueName: \"kubernetes.io/projected/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-kube-api-access-m6cxt\") pod \"8e7768b7-6fa1-4408-a1f5-2862a8a7d907\" (UID: \"8e7768b7-6fa1-4408-a1f5-2862a8a7d907\") " Nov 01 21:07:07.497730 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.497686 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-istiod-ca-cert\") pod \"8e7768b7-6fa1-4408-a1f5-2862a8a7d907\" (UID: \"8e7768b7-6fa1-4408-a1f5-2862a8a7d907\") " Nov 01 21:07:07.497730 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.497732 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17927c6e-7e23-4717-be79-d2569ebae4ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17927c6e-7e23-4717-be79-d2569ebae4ca-istio-data\") pod \"17927c6e-7e23-4717-be79-d2569ebae4ca\" (UID: \"17927c6e-7e23-4717-be79-d2569ebae4ca\") " Nov 01 21:07:07.498034 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.497775 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-istio-envoy\") pod \"8e7768b7-6fa1-4408-a1f5-2862a8a7d907\" (UID: \"8e7768b7-6fa1-4408-a1f5-2862a8a7d907\") " Nov 01 21:07:07.498034 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.497839 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-credential-socket\") pod \"8e7768b7-6fa1-4408-a1f5-2862a8a7d907\" (UID: \"8e7768b7-6fa1-4408-a1f5-2862a8a7d907\") " Nov 01 21:07:07.498034 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.497881 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-workload-certs\") pod \"8e7768b7-6fa1-4408-a1f5-2862a8a7d907\" (UID: \"8e7768b7-6fa1-4408-a1f5-2862a8a7d907\") " Nov 01 21:07:07.498034 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.497915 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17927c6e-7e23-4717-be79-d2569ebae4ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17927c6e-7e23-4717-be79-d2569ebae4ca-istiod-ca-cert\") pod \"17927c6e-7e23-4717-be79-d2569ebae4ca\" (UID: \"17927c6e-7e23-4717-be79-d2569ebae4ca\") " Nov 01 21:07:07.498034 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.497969 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17927c6e-7e23-4717-be79-d2569ebae4ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17927c6e-7e23-4717-be79-d2569ebae4ca-workload-certs\") pod \"17927c6e-7e23-4717-be79-d2569ebae4ca\" (UID: \"17927c6e-7e23-4717-be79-d2569ebae4ca\") " Nov 01 21:07:07.498472 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.498052 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-istio-token\" (UniqueName: \"kubernetes.io/projected/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-istio-token\") pod \"8e7768b7-6fa1-4408-a1f5-2862a8a7d907\" (UID: \"8e7768b7-6fa1-4408-a1f5-2862a8a7d907\") " Nov 01 21:07:07.498472 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.498110 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-istio-podinfo\") pod \"8e7768b7-6fa1-4408-a1f5-2862a8a7d907\" (UID: \"8e7768b7-6fa1-4408-a1f5-2862a8a7d907\") " Nov 01 21:07:07.498472 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.498170 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-workload-socket\") pod \"8e7768b7-6fa1-4408-a1f5-2862a8a7d907\" (UID: \"8e7768b7-6fa1-4408-a1f5-2862a8a7d907\") " Nov 01 21:07:07.498472 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.498231 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17927c6e-7e23-4717-be79-d2569ebae4ca-istio-token\" (UniqueName: \"kubernetes.io/projected/17927c6e-7e23-4717-be79-d2569ebae4ca-istio-token\") pod \"17927c6e-7e23-4717-be79-d2569ebae4ca\" (UID: \"17927c6e-7e23-4717-be79-d2569ebae4ca\") " Nov 01 21:07:07.498472 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.498263 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17927c6e-7e23-4717-be79-d2569ebae4ca-kube-api-access-mff9h\" (UniqueName: \"kubernetes.io/projected/17927c6e-7e23-4717-be79-d2569ebae4ca-kube-api-access-mff9h\") pod \"17927c6e-7e23-4717-be79-d2569ebae4ca\" (UID: \"17927c6e-7e23-4717-be79-d2569ebae4ca\") " Nov 01 21:07:07.498472 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.498351 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-istio-data\") pod \"8e7768b7-6fa1-4408-a1f5-2862a8a7d907\" (UID: \"8e7768b7-6fa1-4408-a1f5-2862a8a7d907\") " Nov 01 21:07:07.498472 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.498391 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-credential-socket" pod "8e7768b7-6fa1-4408-a1f5-2862a8a7d907" (UID: "8e7768b7-6fa1-4408-a1f5-2862a8a7d907"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:07:07.498472 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.498400 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17927c6e-7e23-4717-be79-d2569ebae4ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17927c6e-7e23-4717-be79-d2569ebae4ca-credential-socket\") pod \"17927c6e-7e23-4717-be79-d2569ebae4ca\" (UID: \"17927c6e-7e23-4717-be79-d2569ebae4ca\") " Nov 01 21:07:07.498938 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.498482 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17927c6e-7e23-4717-be79-d2569ebae4ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17927c6e-7e23-4717-be79-d2569ebae4ca-workload-socket\") pod \"17927c6e-7e23-4717-be79-d2569ebae4ca\" (UID: \"17927c6e-7e23-4717-be79-d2569ebae4ca\") " Nov 01 21:07:07.498938 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.498537 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/17927c6e-7e23-4717-be79-d2569ebae4ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17927c6e-7e23-4717-be79-d2569ebae4ca-istio-podinfo\") pod \"17927c6e-7e23-4717-be79-d2569ebae4ca\" (UID: \"17927c6e-7e23-4717-be79-d2569ebae4ca\") " Nov 01 21:07:07.498938 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.498639 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17927c6e-7e23-4717-be79-d2569ebae4ca-credential-socket" pod "17927c6e-7e23-4717-be79-d2569ebae4ca" (UID: "17927c6e-7e23-4717-be79-d2569ebae4ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:07:07.498938 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.498834 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-istio-data" pod "8e7768b7-6fa1-4408-a1f5-2862a8a7d907" (UID: "8e7768b7-6fa1-4408-a1f5-2862a8a7d907"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:07:07.499163 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.498955 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-workload-socket" pod "8e7768b7-6fa1-4408-a1f5-2862a8a7d907" (UID: "8e7768b7-6fa1-4408-a1f5-2862a8a7d907"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:07:07.499221 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.499157 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17927c6e-7e23-4717-be79-d2569ebae4ca-istiod-ca-cert" pod "17927c6e-7e23-4717-be79-d2569ebae4ca" (UID: "17927c6e-7e23-4717-be79-d2569ebae4ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:07:07.500429 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.499394 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-workload-certs" pod "8e7768b7-6fa1-4408-a1f5-2862a8a7d907" (UID: "8e7768b7-6fa1-4408-a1f5-2862a8a7d907"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:07:07.500429 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.499541 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17927c6e-7e23-4717-be79-d2569ebae4ca-istio-data" pod "17927c6e-7e23-4717-be79-d2569ebae4ca" (UID: "17927c6e-7e23-4717-be79-d2569ebae4ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:07:07.500429 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.499663 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:07.500429 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.499685 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:07.500429 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.499701 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17927c6e-7e23-4717-be79-d2569ebae4ca-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:07.500429 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.499714 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:07.500429 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.499728 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:07.500429 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.499743 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17927c6e-7e23-4717-be79-d2569ebae4ca-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:07.500429 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.499757 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17927c6e-7e23-4717-be79-d2569ebae4ca-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:07.500429 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.500198 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17927c6e-7e23-4717-be79-d2569ebae4ca-workload-certs" pod "17927c6e-7e23-4717-be79-d2569ebae4ca" (UID: "17927c6e-7e23-4717-be79-d2569ebae4ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:07:07.501117 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.500794 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-istiod-ca-cert" pod "8e7768b7-6fa1-4408-a1f5-2862a8a7d907" (UID: "8e7768b7-6fa1-4408-a1f5-2862a8a7d907"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:07:07.501117 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.500899 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-istio-token" pod "8e7768b7-6fa1-4408-a1f5-2862a8a7d907" (UID: "8e7768b7-6fa1-4408-a1f5-2862a8a7d907"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:07:07.501117 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.500947 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17927c6e-7e23-4717-be79-d2569ebae4ca-istio-envoy" pod "17927c6e-7e23-4717-be79-d2569ebae4ca" (UID: "17927c6e-7e23-4717-be79-d2569ebae4ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:07:07.501291 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.501251 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17927c6e-7e23-4717-be79-d2569ebae4ca-workload-socket" pod "17927c6e-7e23-4717-be79-d2569ebae4ca" (UID: "17927c6e-7e23-4717-be79-d2569ebae4ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:07:07.501614 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.501589 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17927c6e-7e23-4717-be79-d2569ebae4ca-kube-api-access-mff9h" pod "17927c6e-7e23-4717-be79-d2569ebae4ca" (UID: "17927c6e-7e23-4717-be79-d2569ebae4ca"). InnerVolumeSpecName "kube-api-access-mff9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:07:07.501944 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.501918 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17927c6e-7e23-4717-be79-d2569ebae4ca-istio-token" pod "17927c6e-7e23-4717-be79-d2569ebae4ca" (UID: "17927c6e-7e23-4717-be79-d2569ebae4ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:07:07.502420 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.502396 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-kube-api-access-m6cxt" pod "8e7768b7-6fa1-4408-a1f5-2862a8a7d907" (UID: "8e7768b7-6fa1-4408-a1f5-2862a8a7d907"). InnerVolumeSpecName "kube-api-access-m6cxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:07:07.502828 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.502807 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-istio-envoy" pod "8e7768b7-6fa1-4408-a1f5-2862a8a7d907" (UID: "8e7768b7-6fa1-4408-a1f5-2862a8a7d907"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:07:07.529340 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.529173 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17927c6e-7e23-4717-be79-d2569ebae4ca-istio-podinfo" pod "17927c6e-7e23-4717-be79-d2569ebae4ca" (UID: "17927c6e-7e23-4717-be79-d2569ebae4ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:07:07.529340 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.529173 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-istio-podinfo" pod "8e7768b7-6fa1-4408-a1f5-2862a8a7d907" (UID: "8e7768b7-6fa1-4408-a1f5-2862a8a7d907"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:07:07.554540 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.554501 4484 patch_prober.go:29] interesting pod/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Nov 01 21:07:07.554710 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.554585 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc" podUID="8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Nov 01 21:07:07.601369 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.601190 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:07.601369 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.601223 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:07.601369 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.601237 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17927c6e-7e23-4717-be79-d2569ebae4ca-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:07.601369 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.601249 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mff9h\" (UniqueName: \"kubernetes.io/projected/17927c6e-7e23-4717-be79-d2569ebae4ca-kube-api-access-mff9h\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:07.601369 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.601263 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17927c6e-7e23-4717-be79-d2569ebae4ca-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:07.601369 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.601276 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17927c6e-7e23-4717-be79-d2569ebae4ca-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:07.601369 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.601288 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17927c6e-7e23-4717-be79-d2569ebae4ca-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:07.601369 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.601301 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6cxt\" (UniqueName: \"kubernetes.io/projected/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-kube-api-access-m6cxt\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:07.601369 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.601312 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:07.601369 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.601325 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e7768b7-6fa1-4408-a1f5-2862a8a7d907-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:07.601369 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.601338 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17927c6e-7e23-4717-be79-d2569ebae4ca-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:07.653379 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.653348 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z4sdv" Nov 01 21:07:07.742634 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.728660 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/243b9a06-d07a-4d14-abe2-ad2239dd45fe-kube-api-access-wtt7d\" (UniqueName: \"kubernetes.io/projected/243b9a06-d07a-4d14-abe2-ad2239dd45fe-kube-api-access-wtt7d\") pod \"243b9a06-d07a-4d14-abe2-ad2239dd45fe\" (UID: \"243b9a06-d07a-4d14-abe2-ad2239dd45fe\") " Nov 01 21:07:07.742634 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.728780 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/243b9a06-d07a-4d14-abe2-ad2239dd45fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/243b9a06-d07a-4d14-abe2-ad2239dd45fe-istio-podinfo\") pod \"243b9a06-d07a-4d14-abe2-ad2239dd45fe\" (UID: \"243b9a06-d07a-4d14-abe2-ad2239dd45fe\") " Nov 01 21:07:07.742634 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.728865 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/243b9a06-d07a-4d14-abe2-ad2239dd45fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/243b9a06-d07a-4d14-abe2-ad2239dd45fe-workload-socket\") pod \"243b9a06-d07a-4d14-abe2-ad2239dd45fe\" (UID: \"243b9a06-d07a-4d14-abe2-ad2239dd45fe\") " Nov 01 21:07:07.742634 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.728913 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/243b9a06-d07a-4d14-abe2-ad2239dd45fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/243b9a06-d07a-4d14-abe2-ad2239dd45fe-istio-data\") pod \"243b9a06-d07a-4d14-abe2-ad2239dd45fe\" (UID: \"243b9a06-d07a-4d14-abe2-ad2239dd45fe\") " Nov 01 21:07:07.742634 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.728942 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/243b9a06-d07a-4d14-abe2-ad2239dd45fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/243b9a06-d07a-4d14-abe2-ad2239dd45fe-credential-socket\") pod \"243b9a06-d07a-4d14-abe2-ad2239dd45fe\" (UID: \"243b9a06-d07a-4d14-abe2-ad2239dd45fe\") " Nov 01 21:07:07.742634 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.729250 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/243b9a06-d07a-4d14-abe2-ad2239dd45fe-workload-socket" pod "243b9a06-d07a-4d14-abe2-ad2239dd45fe" (UID: "243b9a06-d07a-4d14-abe2-ad2239dd45fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:07:07.742634 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.733272 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/243b9a06-d07a-4d14-abe2-ad2239dd45fe-credential-socket" pod "243b9a06-d07a-4d14-abe2-ad2239dd45fe" (UID: "243b9a06-d07a-4d14-abe2-ad2239dd45fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:07:07.742634 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.735035 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/243b9a06-d07a-4d14-abe2-ad2239dd45fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/243b9a06-d07a-4d14-abe2-ad2239dd45fe-istio-envoy\") pod \"243b9a06-d07a-4d14-abe2-ad2239dd45fe\" (UID: \"243b9a06-d07a-4d14-abe2-ad2239dd45fe\") " Nov 01 21:07:07.742634 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.735241 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/243b9a06-d07a-4d14-abe2-ad2239dd45fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/243b9a06-d07a-4d14-abe2-ad2239dd45fe-istiod-ca-cert\") pod \"243b9a06-d07a-4d14-abe2-ad2239dd45fe\" (UID: \"243b9a06-d07a-4d14-abe2-ad2239dd45fe\") " Nov 01 21:07:07.742634 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.735298 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/243b9a06-d07a-4d14-abe2-ad2239dd45fe-istio-token\" (UniqueName: \"kubernetes.io/projected/243b9a06-d07a-4d14-abe2-ad2239dd45fe-istio-token\") pod \"243b9a06-d07a-4d14-abe2-ad2239dd45fe\" (UID: \"243b9a06-d07a-4d14-abe2-ad2239dd45fe\") " Nov 01 21:07:07.742634 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.735339 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/243b9a06-d07a-4d14-abe2-ad2239dd45fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/243b9a06-d07a-4d14-abe2-ad2239dd45fe-workload-certs\") pod \"243b9a06-d07a-4d14-abe2-ad2239dd45fe\" (UID: \"243b9a06-d07a-4d14-abe2-ad2239dd45fe\") " Nov 01 21:07:07.742634 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.736065 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/243b9a06-d07a-4d14-abe2-ad2239dd45fe-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:07.742634 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.736092 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/243b9a06-d07a-4d14-abe2-ad2239dd45fe-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:07.742634 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.739443 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/243b9a06-d07a-4d14-abe2-ad2239dd45fe-istiod-ca-cert" pod "243b9a06-d07a-4d14-abe2-ad2239dd45fe" (UID: "243b9a06-d07a-4d14-abe2-ad2239dd45fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:07:07.742634 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.742502 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/243b9a06-d07a-4d14-abe2-ad2239dd45fe-istio-token" pod "243b9a06-d07a-4d14-abe2-ad2239dd45fe" (UID: "243b9a06-d07a-4d14-abe2-ad2239dd45fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:07:07.743532 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.743106 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/243b9a06-d07a-4d14-abe2-ad2239dd45fe-istio-data" pod "243b9a06-d07a-4d14-abe2-ad2239dd45fe" (UID: "243b9a06-d07a-4d14-abe2-ad2239dd45fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:07:07.743532 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.743119 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/243b9a06-d07a-4d14-abe2-ad2239dd45fe-workload-certs" pod "243b9a06-d07a-4d14-abe2-ad2239dd45fe" (UID: "243b9a06-d07a-4d14-abe2-ad2239dd45fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:07:07.743532 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.743523 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/243b9a06-d07a-4d14-abe2-ad2239dd45fe-istio-podinfo" pod "243b9a06-d07a-4d14-abe2-ad2239dd45fe" (UID: "243b9a06-d07a-4d14-abe2-ad2239dd45fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:07:07.744480 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.744446 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/243b9a06-d07a-4d14-abe2-ad2239dd45fe-kube-api-access-wtt7d" pod "243b9a06-d07a-4d14-abe2-ad2239dd45fe" (UID: "243b9a06-d07a-4d14-abe2-ad2239dd45fe"). InnerVolumeSpecName "kube-api-access-wtt7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:07:07.745326 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.745299 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/243b9a06-d07a-4d14-abe2-ad2239dd45fe-istio-envoy" pod "243b9a06-d07a-4d14-abe2-ad2239dd45fe" (UID: "243b9a06-d07a-4d14-abe2-ad2239dd45fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:07:07.838232 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.838197 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/243b9a06-d07a-4d14-abe2-ad2239dd45fe-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:07.838232 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.838225 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/243b9a06-d07a-4d14-abe2-ad2239dd45fe-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:07.838232 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.838235 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/243b9a06-d07a-4d14-abe2-ad2239dd45fe-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:07.838525 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.838246 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtt7d\" (UniqueName: \"kubernetes.io/projected/243b9a06-d07a-4d14-abe2-ad2239dd45fe-kube-api-access-wtt7d\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:07.838525 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.838256 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/243b9a06-d07a-4d14-abe2-ad2239dd45fe-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:07.838525 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.838267 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/243b9a06-d07a-4d14-abe2-ad2239dd45fe-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:07.838525 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.838282 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/243b9a06-d07a-4d14-abe2-ad2239dd45fe-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:07.957086 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:07.957054 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc" Nov 01 21:07:08.381943 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:08.381915 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk" Nov 01 21:07:08.381943 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:08.381933 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk" event={"ID":"17927c6e-7e23-4717-be79-d2569ebae4ca","Type":"ContainerDied","Data":"a6c519ea34576b502c913fa8b80e97cb35936a99544215e3b22757edd0e78326"} Nov 01 21:07:08.382562 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:08.382012 4484 scope.go:122] "RemoveContainer" containerID="a45831dcad5f6438e76a52c65838f3d14b088583f334964e75b98f643d744308" Nov 01 21:07:08.383708 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:08.383667 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z4sdv" Nov 01 21:07:08.383833 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:08.383710 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z4sdv" event={"ID":"243b9a06-d07a-4d14-abe2-ad2239dd45fe","Type":"ContainerDied","Data":"1ffa4eda144f6e43e9b63d458f49d3822bf0dada7c76e541f7312405018a0b9f"} Nov 01 21:07:08.385485 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:08.385465 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj" Nov 01 21:07:08.385616 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:08.385489 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj" event={"ID":"8e7768b7-6fa1-4408-a1f5-2862a8a7d907","Type":"ContainerDied","Data":"4fec9df5f5438d6fe7326ec3f16f8bc443a34b1619b7a36541b81dfdd495a113"} Nov 01 21:07:08.395892 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:08.395871 4484 scope.go:122] "RemoveContainer" containerID="341a09a23ee439b432b557cc11272c45d10d5ae8721d87979f1fcfdd79c91bf9" Nov 01 21:07:08.433620 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:08.433597 4484 scope.go:122] "RemoveContainer" containerID="613085aea41e537c2798b514085f2da5f17b07a036ebc4af8f94dbdc48b023a9" Nov 01 21:07:08.550678 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:08.550644 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z4sdv"] Nov 01 21:07:08.553018 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:08.552977 4484 patch_prober.go:29] interesting pod/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Nov 01 21:07:08.553103 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:08.553042 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc" podUID="8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Nov 01 21:07:08.601009 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:08.600962 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-z4sdv"] Nov 01 21:07:08.667125 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:08.667042 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj"] Nov 01 21:07:08.670824 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:08.670791 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-db0157-openshift-default-68856dc5c7-psptj"] Nov 01 21:07:08.785228 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:08.785193 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk"] Nov 01 21:07:08.788920 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:08.788892 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd038b-openshift-default-7d9df7cfd7-jwfhk"] Nov 01 21:07:09.373510 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:09.373474 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17927c6e-7e23-4717-be79-d2569ebae4ca" path="/var/lib/kubelet/pods/17927c6e-7e23-4717-be79-d2569ebae4ca/volumes" Nov 01 21:07:09.373984 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:09.373970 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="243b9a06-d07a-4d14-abe2-ad2239dd45fe" path="/var/lib/kubelet/pods/243b9a06-d07a-4d14-abe2-ad2239dd45fe/volumes" Nov 01 21:07:09.374425 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:09.374413 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e7768b7-6fa1-4408-a1f5-2862a8a7d907" path="/var/lib/kubelet/pods/8e7768b7-6fa1-4408-a1f5-2862a8a7d907/volumes" Nov 01 21:07:09.552778 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:09.552740 4484 patch_prober.go:29] interesting pod/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Nov 01 21:07:09.553211 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:09.552799 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc" podUID="8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Nov 01 21:07:10.553006 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:10.552961 4484 patch_prober.go:29] interesting pod/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Nov 01 21:07:10.553403 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:10.553042 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc" podUID="8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Nov 01 21:07:11.072384 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:11.072349 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:07:11.552810 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:11.552771 4484 patch_prober.go:29] interesting pod/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Nov 01 21:07:11.553019 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:11.552852 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc" podUID="8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Nov 01 21:07:12.557207 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:12.557177 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc" Nov 01 21:07:12.558332 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:12.558308 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc" Nov 01 21:07:15.768175 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:07:15.768138 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:07:16.123725 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:16.123639 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:07:16.279939 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:16.279886 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:07:16.483788 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:16.483701 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:07:17.747962 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:17.747927 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:07:18.011068 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:18.010967 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:07:21.518717 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:21.518679 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:07:23.509935 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:23.509904 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:07:23.675780 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:23.675750 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:07:23.782119 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:23.782025 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98"] Nov 01 21:07:23.782316 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:23.782264 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98" podUID="7403ccd9-e6f4-4ceb-9d97-22e08152f92e" containerName="istio-proxy" containerID="cri-o://e6d50989d5066e8d0f1d39a5579b51762c212c9e0d638f2d4f1316a9a8105208" gracePeriod=30 Nov 01 21:07:23.791461 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:23.791433 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8"] Nov 01 21:07:23.791652 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:23.791632 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8" podUID="b18c47d2-1694-4961-9638-be5cb8868cc3" containerName="istio-proxy" containerID="cri-o://771d793b32480a9618c7aa5de6bb995d3bb681998a7fdd1002773ec584fe5bcb" gracePeriod=30 Nov 01 21:07:23.852721 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:23.852685 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:07:26.531842 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:26.531804 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:07:27.780231 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:27.780191 4484 patch_prober.go:29] interesting pod/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 21:07:27.780680 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:27.780265 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98" podUID="7403ccd9-e6f4-4ceb-9d97-22e08152f92e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 21:07:27.961327 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:27.961282 4484 patch_prober.go:29] interesting pod/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 21:07:27.961495 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:27.961353 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8" podUID="b18c47d2-1694-4961-9638-be5cb8868cc3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 21:07:29.081072 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.081038 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98" Nov 01 21:07:29.109527 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.109505 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8" Nov 01 21:07:29.135310 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.135284 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-istiod-ca-cert\") pod \"7403ccd9-e6f4-4ceb-9d97-22e08152f92e\" (UID: \"7403ccd9-e6f4-4ceb-9d97-22e08152f92e\") " Nov 01 21:07:29.135458 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.135337 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-istio-data\") pod \"7403ccd9-e6f4-4ceb-9d97-22e08152f92e\" (UID: \"7403ccd9-e6f4-4ceb-9d97-22e08152f92e\") " Nov 01 21:07:29.135458 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.135400 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-istio-envoy\") pod \"7403ccd9-e6f4-4ceb-9d97-22e08152f92e\" (UID: \"7403ccd9-e6f4-4ceb-9d97-22e08152f92e\") " Nov 01 21:07:29.135458 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.135446 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-kube-api-access-c8c7t\" (UniqueName: \"kubernetes.io/projected/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-kube-api-access-c8c7t\") pod \"7403ccd9-e6f4-4ceb-9d97-22e08152f92e\" (UID: \"7403ccd9-e6f4-4ceb-9d97-22e08152f92e\") " Nov 01 21:07:29.135595 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.135524 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-istio-podinfo\") pod \"7403ccd9-e6f4-4ceb-9d97-22e08152f92e\" (UID: \"7403ccd9-e6f4-4ceb-9d97-22e08152f92e\") " Nov 01 21:07:29.135595 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.135548 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-istio-token\" (UniqueName: \"kubernetes.io/projected/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-istio-token\") pod \"7403ccd9-e6f4-4ceb-9d97-22e08152f92e\" (UID: \"7403ccd9-e6f4-4ceb-9d97-22e08152f92e\") " Nov 01 21:07:29.135595 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.135572 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-workload-socket\") pod \"7403ccd9-e6f4-4ceb-9d97-22e08152f92e\" (UID: \"7403ccd9-e6f4-4ceb-9d97-22e08152f92e\") " Nov 01 21:07:29.135851 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.135650 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-istio-data" pod "7403ccd9-e6f4-4ceb-9d97-22e08152f92e" (UID: "7403ccd9-e6f4-4ceb-9d97-22e08152f92e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:07:29.135851 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.135683 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-istiod-ca-cert" pod "7403ccd9-e6f4-4ceb-9d97-22e08152f92e" (UID: "7403ccd9-e6f4-4ceb-9d97-22e08152f92e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:07:29.135851 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.135753 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-credential-socket\") pod \"7403ccd9-e6f4-4ceb-9d97-22e08152f92e\" (UID: \"7403ccd9-e6f4-4ceb-9d97-22e08152f92e\") " Nov 01 21:07:29.135851 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.135783 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-workload-certs\") pod \"7403ccd9-e6f4-4ceb-9d97-22e08152f92e\" (UID: \"7403ccd9-e6f4-4ceb-9d97-22e08152f92e\") " Nov 01 21:07:29.136146 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.135974 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-workload-socket" pod "7403ccd9-e6f4-4ceb-9d97-22e08152f92e" (UID: "7403ccd9-e6f4-4ceb-9d97-22e08152f92e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:07:29.136146 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.136082 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-credential-socket" pod "7403ccd9-e6f4-4ceb-9d97-22e08152f92e" (UID: "7403ccd9-e6f4-4ceb-9d97-22e08152f92e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:07:29.136283 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.136211 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-workload-certs" pod "7403ccd9-e6f4-4ceb-9d97-22e08152f92e" (UID: "7403ccd9-e6f4-4ceb-9d97-22e08152f92e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:07:29.136509 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.136486 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:29.136590 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.136513 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:29.136590 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.136530 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:29.136590 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.136545 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:29.136590 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.136558 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:29.137743 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.137715 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-istio-envoy" pod "7403ccd9-e6f4-4ceb-9d97-22e08152f92e" (UID: "7403ccd9-e6f4-4ceb-9d97-22e08152f92e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:07:29.137743 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.137715 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-istio-podinfo" pod "7403ccd9-e6f4-4ceb-9d97-22e08152f92e" (UID: "7403ccd9-e6f4-4ceb-9d97-22e08152f92e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:07:29.137886 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.137810 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-istio-token" pod "7403ccd9-e6f4-4ceb-9d97-22e08152f92e" (UID: "7403ccd9-e6f4-4ceb-9d97-22e08152f92e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:07:29.137886 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.137835 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-kube-api-access-c8c7t" pod "7403ccd9-e6f4-4ceb-9d97-22e08152f92e" (UID: "7403ccd9-e6f4-4ceb-9d97-22e08152f92e"). InnerVolumeSpecName "kube-api-access-c8c7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:07:29.237243 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.237205 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b18c47d2-1694-4961-9638-be5cb8868cc3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b18c47d2-1694-4961-9638-be5cb8868cc3-istio-podinfo\") pod \"b18c47d2-1694-4961-9638-be5cb8868cc3\" (UID: \"b18c47d2-1694-4961-9638-be5cb8868cc3\") " Nov 01 21:07:29.237448 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.237267 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b18c47d2-1694-4961-9638-be5cb8868cc3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b18c47d2-1694-4961-9638-be5cb8868cc3-workload-certs\") pod \"b18c47d2-1694-4961-9638-be5cb8868cc3\" (UID: \"b18c47d2-1694-4961-9638-be5cb8868cc3\") " Nov 01 21:07:29.237448 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.237295 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b18c47d2-1694-4961-9638-be5cb8868cc3-istio-token\" (UniqueName: \"kubernetes.io/projected/b18c47d2-1694-4961-9638-be5cb8868cc3-istio-token\") pod \"b18c47d2-1694-4961-9638-be5cb8868cc3\" (UID: \"b18c47d2-1694-4961-9638-be5cb8868cc3\") " Nov 01 21:07:29.237448 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.237343 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b18c47d2-1694-4961-9638-be5cb8868cc3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b18c47d2-1694-4961-9638-be5cb8868cc3-istio-data\") pod \"b18c47d2-1694-4961-9638-be5cb8868cc3\" (UID: \"b18c47d2-1694-4961-9638-be5cb8868cc3\") " Nov 01 21:07:29.237448 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.237426 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b18c47d2-1694-4961-9638-be5cb8868cc3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b18c47d2-1694-4961-9638-be5cb8868cc3-istiod-ca-cert\") pod \"b18c47d2-1694-4961-9638-be5cb8868cc3\" (UID: \"b18c47d2-1694-4961-9638-be5cb8868cc3\") " Nov 01 21:07:29.237687 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.237516 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b18c47d2-1694-4961-9638-be5cb8868cc3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b18c47d2-1694-4961-9638-be5cb8868cc3-credential-socket\") pod \"b18c47d2-1694-4961-9638-be5cb8868cc3\" (UID: \"b18c47d2-1694-4961-9638-be5cb8868cc3\") " Nov 01 21:07:29.237687 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.237596 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b18c47d2-1694-4961-9638-be5cb8868cc3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b18c47d2-1694-4961-9638-be5cb8868cc3-workload-socket\") pod \"b18c47d2-1694-4961-9638-be5cb8868cc3\" (UID: \"b18c47d2-1694-4961-9638-be5cb8868cc3\") " Nov 01 21:07:29.237687 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.237644 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b18c47d2-1694-4961-9638-be5cb8868cc3-workload-certs" pod "b18c47d2-1694-4961-9638-be5cb8868cc3" (UID: "b18c47d2-1694-4961-9638-be5cb8868cc3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:07:29.237687 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.237657 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b18c47d2-1694-4961-9638-be5cb8868cc3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b18c47d2-1694-4961-9638-be5cb8868cc3-istio-envoy\") pod \"b18c47d2-1694-4961-9638-be5cb8868cc3\" (UID: \"b18c47d2-1694-4961-9638-be5cb8868cc3\") " Nov 01 21:07:29.237890 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.237685 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b18c47d2-1694-4961-9638-be5cb8868cc3-istio-data" pod "b18c47d2-1694-4961-9638-be5cb8868cc3" (UID: "b18c47d2-1694-4961-9638-be5cb8868cc3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:07:29.237890 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.237820 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b18c47d2-1694-4961-9638-be5cb8868cc3-kube-api-access-7g8gg\" (UniqueName: \"kubernetes.io/projected/b18c47d2-1694-4961-9638-be5cb8868cc3-kube-api-access-7g8gg\") pod \"b18c47d2-1694-4961-9638-be5cb8868cc3\" (UID: \"b18c47d2-1694-4961-9638-be5cb8868cc3\") " Nov 01 21:07:29.238044 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.237912 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b18c47d2-1694-4961-9638-be5cb8868cc3-istiod-ca-cert" pod "b18c47d2-1694-4961-9638-be5cb8868cc3" (UID: "b18c47d2-1694-4961-9638-be5cb8868cc3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:07:29.238044 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.237930 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b18c47d2-1694-4961-9638-be5cb8868cc3-workload-socket" pod "b18c47d2-1694-4961-9638-be5cb8868cc3" (UID: "b18c47d2-1694-4961-9638-be5cb8868cc3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:07:29.238044 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.237916 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b18c47d2-1694-4961-9638-be5cb8868cc3-credential-socket" pod "b18c47d2-1694-4961-9638-be5cb8868cc3" (UID: "b18c47d2-1694-4961-9638-be5cb8868cc3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:07:29.238671 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.238514 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b18c47d2-1694-4961-9638-be5cb8868cc3-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:29.238671 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.238544 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8c7t\" (UniqueName: \"kubernetes.io/projected/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-kube-api-access-c8c7t\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:29.238671 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.238561 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:29.238671 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.238574 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:29.238671 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.238587 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b18c47d2-1694-4961-9638-be5cb8868cc3-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:29.238671 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.238600 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b18c47d2-1694-4961-9638-be5cb8868cc3-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:29.238671 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.238619 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b18c47d2-1694-4961-9638-be5cb8868cc3-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:29.238671 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.238634 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b18c47d2-1694-4961-9638-be5cb8868cc3-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:29.238671 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.238647 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7403ccd9-e6f4-4ceb-9d97-22e08152f92e-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:29.240302 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.240274 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b18c47d2-1694-4961-9638-be5cb8868cc3-istio-podinfo" pod "b18c47d2-1694-4961-9638-be5cb8868cc3" (UID: "b18c47d2-1694-4961-9638-be5cb8868cc3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:07:29.240451 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.240325 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b18c47d2-1694-4961-9638-be5cb8868cc3-istio-token" pod "b18c47d2-1694-4961-9638-be5cb8868cc3" (UID: "b18c47d2-1694-4961-9638-be5cb8868cc3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:07:29.240451 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.240347 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b18c47d2-1694-4961-9638-be5cb8868cc3-kube-api-access-7g8gg" pod "b18c47d2-1694-4961-9638-be5cb8868cc3" (UID: "b18c47d2-1694-4961-9638-be5cb8868cc3"). InnerVolumeSpecName "kube-api-access-7g8gg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:07:29.240602 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.240579 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b18c47d2-1694-4961-9638-be5cb8868cc3-istio-envoy" pod "b18c47d2-1694-4961-9638-be5cb8868cc3" (UID: "b18c47d2-1694-4961-9638-be5cb8868cc3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:07:29.339927 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.339840 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b18c47d2-1694-4961-9638-be5cb8868cc3-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:29.339927 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.339872 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g8gg\" (UniqueName: \"kubernetes.io/projected/b18c47d2-1694-4961-9638-be5cb8868cc3-kube-api-access-7g8gg\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:29.339927 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.339882 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b18c47d2-1694-4961-9638-be5cb8868cc3-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:29.339927 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.339892 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b18c47d2-1694-4961-9638-be5cb8868cc3-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:07:29.607455 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.607361 4484 generic.go:356] "Generic (PLEG): container finished" podID="7403ccd9-e6f4-4ceb-9d97-22e08152f92e" containerID="e6d50989d5066e8d0f1d39a5579b51762c212c9e0d638f2d4f1316a9a8105208" exitCode=0 Nov 01 21:07:29.607455 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.607409 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98" Nov 01 21:07:29.607672 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.607455 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98" event={"ID":"7403ccd9-e6f4-4ceb-9d97-22e08152f92e","Type":"ContainerDied","Data":"e6d50989d5066e8d0f1d39a5579b51762c212c9e0d638f2d4f1316a9a8105208"} Nov 01 21:07:29.607672 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.607511 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98" event={"ID":"7403ccd9-e6f4-4ceb-9d97-22e08152f92e","Type":"ContainerDied","Data":"d3c205ed8af7603c1f1264e9ad7b95defc80e4ff3a282c1449c226d1ecf807ff"} Nov 01 21:07:29.607672 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.607541 4484 scope.go:122] "RemoveContainer" containerID="e6d50989d5066e8d0f1d39a5579b51762c212c9e0d638f2d4f1316a9a8105208" Nov 01 21:07:29.609038 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.608980 4484 generic.go:356] "Generic (PLEG): container finished" podID="b18c47d2-1694-4961-9638-be5cb8868cc3" containerID="771d793b32480a9618c7aa5de6bb995d3bb681998a7fdd1002773ec584fe5bcb" exitCode=0 Nov 01 21:07:29.609183 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.609033 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8" event={"ID":"b18c47d2-1694-4961-9638-be5cb8868cc3","Type":"ContainerDied","Data":"771d793b32480a9618c7aa5de6bb995d3bb681998a7fdd1002773ec584fe5bcb"} Nov 01 21:07:29.609183 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.609058 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8" Nov 01 21:07:29.609183 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.609086 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8" event={"ID":"b18c47d2-1694-4961-9638-be5cb8868cc3","Type":"ContainerDied","Data":"0e39dc9779d16f4a34d8ee3961473fd9ba83cc2bd90eb1aa7e7c39f2d20d6075"} Nov 01 21:07:29.618915 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.618889 4484 scope.go:122] "RemoveContainer" containerID="e6d50989d5066e8d0f1d39a5579b51762c212c9e0d638f2d4f1316a9a8105208" Nov 01 21:07:29.619349 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:07:29.619327 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6d50989d5066e8d0f1d39a5579b51762c212c9e0d638f2d4f1316a9a8105208\": container with ID starting with e6d50989d5066e8d0f1d39a5579b51762c212c9e0d638f2d4f1316a9a8105208 not found: ID does not exist" containerID="e6d50989d5066e8d0f1d39a5579b51762c212c9e0d638f2d4f1316a9a8105208" Nov 01 21:07:29.619433 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.619354 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6d50989d5066e8d0f1d39a5579b51762c212c9e0d638f2d4f1316a9a8105208"} err="failed to get container status \"e6d50989d5066e8d0f1d39a5579b51762c212c9e0d638f2d4f1316a9a8105208\": rpc error: code = NotFound desc = could not find container \"e6d50989d5066e8d0f1d39a5579b51762c212c9e0d638f2d4f1316a9a8105208\": container with ID starting with e6d50989d5066e8d0f1d39a5579b51762c212c9e0d638f2d4f1316a9a8105208 not found: ID does not exist" Nov 01 21:07:29.619433 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.619375 4484 scope.go:122] "RemoveContainer" containerID="771d793b32480a9618c7aa5de6bb995d3bb681998a7fdd1002773ec584fe5bcb" Nov 01 21:07:29.632100 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.632077 4484 scope.go:122] "RemoveContainer" containerID="771d793b32480a9618c7aa5de6bb995d3bb681998a7fdd1002773ec584fe5bcb" Nov 01 21:07:29.632495 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:07:29.632467 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"771d793b32480a9618c7aa5de6bb995d3bb681998a7fdd1002773ec584fe5bcb\": container with ID starting with 771d793b32480a9618c7aa5de6bb995d3bb681998a7fdd1002773ec584fe5bcb not found: ID does not exist" containerID="771d793b32480a9618c7aa5de6bb995d3bb681998a7fdd1002773ec584fe5bcb" Nov 01 21:07:29.632549 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.632507 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"771d793b32480a9618c7aa5de6bb995d3bb681998a7fdd1002773ec584fe5bcb"} err="failed to get container status \"771d793b32480a9618c7aa5de6bb995d3bb681998a7fdd1002773ec584fe5bcb\": rpc error: code = NotFound desc = could not find container \"771d793b32480a9618c7aa5de6bb995d3bb681998a7fdd1002773ec584fe5bcb\": container with ID starting with 771d793b32480a9618c7aa5de6bb995d3bb681998a7fdd1002773ec584fe5bcb not found: ID does not exist" Nov 01 21:07:29.777273 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.777242 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98"] Nov 01 21:07:29.783276 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.783226 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-gkl98"] Nov 01 21:07:29.898481 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.898396 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8"] Nov 01 21:07:29.901644 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:29.901622 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-11a069b1-openshift-default-7d5dd8f549-xcsv8"] Nov 01 21:07:31.271030 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:31.270865 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:07:31.375263 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:31.375231 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7403ccd9-e6f4-4ceb-9d97-22e08152f92e" path="/var/lib/kubelet/pods/7403ccd9-e6f4-4ceb-9d97-22e08152f92e/volumes" Nov 01 21:07:31.375700 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:31.375686 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b18c47d2-1694-4961-9638-be5cb8868cc3" path="/var/lib/kubelet/pods/b18c47d2-1694-4961-9638-be5cb8868cc3/volumes" Nov 01 21:07:36.040486 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:36.040418 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:07:37.619136 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:37.619099 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:07:37.779807 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:37.779771 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:07:40.656245 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:40.656210 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:07:40.814080 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:40.814043 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:07:40.988869 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:40.988840 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:07:45.528948 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:45.528912 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:07:45.767214 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:45.767184 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:07:45.768096 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:07:45.768066 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:07:50.541371 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:50.541327 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:07:50.700015 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:50.699950 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:07:50.864093 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:50.863979 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:07:54.960741 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:54.960703 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:07:56.507083 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:56.507048 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:07:56.730774 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:56.730705 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:07:56.892584 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:56.892493 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:07:56.993741 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:56.993706 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9ae21-494d5"] Nov 01 21:07:57.044025 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:57.043967 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9ae21-494d5"] Nov 01 21:07:57.373045 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:57.372984 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b20f2ca-5b06-4e4e-a5ef-f68b8414f2ac" path="/var/lib/kubelet/pods/8b20f2ca-5b06-4e4e-a5ef-f68b8414f2ac/volumes" Nov 01 21:07:57.398217 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:07:57.398176 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:00.607784 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:00.607744 4484 scope.go:122] "RemoveContainer" containerID="696d1ee186f7008f36a3641a228eebb9ebbe5076949519e17d2bd78ca87dcc77" Nov 01 21:08:02.035186 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:02.035148 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:03.170102 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:03.170059 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:03.862943 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:03.862908 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:04.222722 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:04.222628 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:04.388832 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:04.388798 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:04.563449 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:04.563413 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:05.408065 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.408022 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2"] Nov 01 21:08:05.408711 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.408696 4484 cpu_manager.go:414] "containerMap: removing container" podUID="836acdb1-4e39-472f-ba72-a07d61f977bf" containerName="istio-proxy" Nov 01 21:08:05.408757 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.408711 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="836acdb1-4e39-472f-ba72-a07d61f977bf" containerName="istio-proxy" Nov 01 21:08:05.408757 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.408725 4484 cpu_manager.go:414] "containerMap: removing container" podUID="7403ccd9-e6f4-4ceb-9d97-22e08152f92e" containerName="istio-proxy" Nov 01 21:08:05.408757 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.408730 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7403ccd9-e6f4-4ceb-9d97-22e08152f92e" containerName="istio-proxy" Nov 01 21:08:05.408757 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.408742 4484 cpu_manager.go:414] "containerMap: removing container" podUID="836acdb1-4e39-472f-ba72-a07d61f977bf" containerName="istio-proxy" Nov 01 21:08:05.408757 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.408747 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="836acdb1-4e39-472f-ba72-a07d61f977bf" containerName="istio-proxy" Nov 01 21:08:05.408757 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.408754 4484 cpu_manager.go:414] "containerMap: removing container" podUID="8e7768b7-6fa1-4408-a1f5-2862a8a7d907" containerName="istio-proxy" Nov 01 21:08:05.408757 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.408758 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e7768b7-6fa1-4408-a1f5-2862a8a7d907" containerName="istio-proxy" Nov 01 21:08:05.408985 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.408768 4484 cpu_manager.go:414] "containerMap: removing container" podUID="17927c6e-7e23-4717-be79-d2569ebae4ca" containerName="istio-proxy" Nov 01 21:08:05.408985 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.408773 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17927c6e-7e23-4717-be79-d2569ebae4ca" containerName="istio-proxy" Nov 01 21:08:05.408985 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.408783 4484 cpu_manager.go:414] "containerMap: removing container" podUID="243b9a06-d07a-4d14-abe2-ad2239dd45fe" containerName="istio-proxy" Nov 01 21:08:05.408985 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.408787 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="243b9a06-d07a-4d14-abe2-ad2239dd45fe" containerName="istio-proxy" Nov 01 21:08:05.408985 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.408799 4484 cpu_manager.go:414] "containerMap: removing container" podUID="2f7a17d2-43cc-4001-814a-8446e8054743" containerName="payload-pre-processing" Nov 01 21:08:05.408985 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.408804 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f7a17d2-43cc-4001-814a-8446e8054743" containerName="payload-pre-processing" Nov 01 21:08:05.408985 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.408814 4484 cpu_manager.go:414] "containerMap: removing container" podUID="b18c47d2-1694-4961-9638-be5cb8868cc3" containerName="istio-proxy" Nov 01 21:08:05.408985 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.408819 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b18c47d2-1694-4961-9638-be5cb8868cc3" containerName="istio-proxy" Nov 01 21:08:05.408985 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.408827 4484 cpu_manager.go:414] "containerMap: removing container" podUID="71efab02-15fb-41db-87d8-251a4cf92947" containerName="revoke-keys" Nov 01 21:08:05.408985 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.408832 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71efab02-15fb-41db-87d8-251a4cf92947" containerName="revoke-keys" Nov 01 21:08:05.408985 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.408925 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="71efab02-15fb-41db-87d8-251a4cf92947" containerName="revoke-keys" Nov 01 21:08:05.408985 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.408940 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="836acdb1-4e39-472f-ba72-a07d61f977bf" containerName="istio-proxy" Nov 01 21:08:05.408985 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.408946 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="7403ccd9-e6f4-4ceb-9d97-22e08152f92e" containerName="istio-proxy" Nov 01 21:08:05.408985 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.408951 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="836acdb1-4e39-472f-ba72-a07d61f977bf" containerName="istio-proxy" Nov 01 21:08:05.408985 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.408957 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e7768b7-6fa1-4408-a1f5-2862a8a7d907" containerName="istio-proxy" Nov 01 21:08:05.408985 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.408962 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="17927c6e-7e23-4717-be79-d2569ebae4ca" containerName="istio-proxy" Nov 01 21:08:05.408985 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.408968 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="243b9a06-d07a-4d14-abe2-ad2239dd45fe" containerName="istio-proxy" Nov 01 21:08:05.408985 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.408977 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f7a17d2-43cc-4001-814a-8446e8054743" containerName="payload-pre-processing" Nov 01 21:08:05.408985 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.408982 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="b18c47d2-1694-4961-9638-be5cb8868cc3" containerName="istio-proxy" Nov 01 21:08:05.414582 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.414556 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2" Nov 01 21:08:05.471022 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.470068 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-a41242-openshift-default-dockercfg-4rz98\"" Nov 01 21:08:05.504018 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.502921 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/297fe96b-76ad-499e-b9b7-f536d1851712-workload-socket\") pod \"e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2\" (UID: \"297fe96b-76ad-499e-b9b7-f536d1851712\") " pod="openshift-ingress/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2" Nov 01 21:08:05.504018 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.502971 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/297fe96b-76ad-499e-b9b7-f536d1851712-workload-certs\") pod \"e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2\" (UID: \"297fe96b-76ad-499e-b9b7-f536d1851712\") " pod="openshift-ingress/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2" Nov 01 21:08:05.504018 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.503037 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mh25\" (UniqueName: \"kubernetes.io/projected/297fe96b-76ad-499e-b9b7-f536d1851712-kube-api-access-9mh25\") pod \"e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2\" (UID: \"297fe96b-76ad-499e-b9b7-f536d1851712\") " pod="openshift-ingress/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2" Nov 01 21:08:05.504018 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.503163 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/297fe96b-76ad-499e-b9b7-f536d1851712-credential-socket\") pod \"e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2\" (UID: \"297fe96b-76ad-499e-b9b7-f536d1851712\") " pod="openshift-ingress/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2" Nov 01 21:08:05.504018 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.503244 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/297fe96b-76ad-499e-b9b7-f536d1851712-istio-podinfo\") pod \"e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2\" (UID: \"297fe96b-76ad-499e-b9b7-f536d1851712\") " pod="openshift-ingress/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2" Nov 01 21:08:05.504018 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.503291 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/297fe96b-76ad-499e-b9b7-f536d1851712-istio-envoy\") pod \"e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2\" (UID: \"297fe96b-76ad-499e-b9b7-f536d1851712\") " pod="openshift-ingress/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2" Nov 01 21:08:05.504018 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.503327 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/297fe96b-76ad-499e-b9b7-f536d1851712-istio-data\") pod \"e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2\" (UID: \"297fe96b-76ad-499e-b9b7-f536d1851712\") " pod="openshift-ingress/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2" Nov 01 21:08:05.504018 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.503362 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/297fe96b-76ad-499e-b9b7-f536d1851712-istio-token\") pod \"e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2\" (UID: \"297fe96b-76ad-499e-b9b7-f536d1851712\") " pod="openshift-ingress/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2" Nov 01 21:08:05.504018 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.503418 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/297fe96b-76ad-499e-b9b7-f536d1851712-istiod-ca-cert\") pod \"e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2\" (UID: \"297fe96b-76ad-499e-b9b7-f536d1851712\") " pod="openshift-ingress/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2" Nov 01 21:08:05.524753 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.524719 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2"] Nov 01 21:08:05.528613 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.528580 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:05.604668 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.604633 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/297fe96b-76ad-499e-b9b7-f536d1851712-istiod-ca-cert\") pod \"e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2\" (UID: \"297fe96b-76ad-499e-b9b7-f536d1851712\") " pod="openshift-ingress/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2" Nov 01 21:08:05.604891 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.604687 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/297fe96b-76ad-499e-b9b7-f536d1851712-workload-socket\") pod \"e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2\" (UID: \"297fe96b-76ad-499e-b9b7-f536d1851712\") " pod="openshift-ingress/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2" Nov 01 21:08:05.604891 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.604706 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/297fe96b-76ad-499e-b9b7-f536d1851712-workload-certs\") pod \"e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2\" (UID: \"297fe96b-76ad-499e-b9b7-f536d1851712\") " pod="openshift-ingress/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2" Nov 01 21:08:05.604891 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.604745 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mh25\" (UniqueName: \"kubernetes.io/projected/297fe96b-76ad-499e-b9b7-f536d1851712-kube-api-access-9mh25\") pod \"e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2\" (UID: \"297fe96b-76ad-499e-b9b7-f536d1851712\") " pod="openshift-ingress/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2" Nov 01 21:08:05.604891 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.604831 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/297fe96b-76ad-499e-b9b7-f536d1851712-credential-socket\") pod \"e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2\" (UID: \"297fe96b-76ad-499e-b9b7-f536d1851712\") " pod="openshift-ingress/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2" Nov 01 21:08:05.604891 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.604882 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/297fe96b-76ad-499e-b9b7-f536d1851712-istio-podinfo\") pod \"e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2\" (UID: \"297fe96b-76ad-499e-b9b7-f536d1851712\") " pod="openshift-ingress/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2" Nov 01 21:08:05.605192 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.604916 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/297fe96b-76ad-499e-b9b7-f536d1851712-istio-envoy\") pod \"e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2\" (UID: \"297fe96b-76ad-499e-b9b7-f536d1851712\") " pod="openshift-ingress/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2" Nov 01 21:08:05.605192 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.604937 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/297fe96b-76ad-499e-b9b7-f536d1851712-istio-data\") pod \"e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2\" (UID: \"297fe96b-76ad-499e-b9b7-f536d1851712\") " pod="openshift-ingress/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2" Nov 01 21:08:05.605192 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.604957 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/297fe96b-76ad-499e-b9b7-f536d1851712-istio-token\") pod \"e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2\" (UID: \"297fe96b-76ad-499e-b9b7-f536d1851712\") " pod="openshift-ingress/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2" Nov 01 21:08:05.605192 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.605165 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/297fe96b-76ad-499e-b9b7-f536d1851712-workload-socket\") pod \"e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2\" (UID: \"297fe96b-76ad-499e-b9b7-f536d1851712\") " pod="openshift-ingress/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2" Nov 01 21:08:05.605946 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.605532 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/297fe96b-76ad-499e-b9b7-f536d1851712-istiod-ca-cert\") pod \"e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2\" (UID: \"297fe96b-76ad-499e-b9b7-f536d1851712\") " pod="openshift-ingress/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2" Nov 01 21:08:05.605946 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.605530 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/297fe96b-76ad-499e-b9b7-f536d1851712-credential-socket\") pod \"e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2\" (UID: \"297fe96b-76ad-499e-b9b7-f536d1851712\") " pod="openshift-ingress/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2" Nov 01 21:08:05.605946 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.605567 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/297fe96b-76ad-499e-b9b7-f536d1851712-istio-data\") pod \"e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2\" (UID: \"297fe96b-76ad-499e-b9b7-f536d1851712\") " pod="openshift-ingress/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2" Nov 01 21:08:05.605946 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.605629 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/297fe96b-76ad-499e-b9b7-f536d1851712-workload-certs\") pod \"e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2\" (UID: \"297fe96b-76ad-499e-b9b7-f536d1851712\") " pod="openshift-ingress/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2" Nov 01 21:08:05.607410 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.607389 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/297fe96b-76ad-499e-b9b7-f536d1851712-istio-envoy\") pod \"e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2\" (UID: \"297fe96b-76ad-499e-b9b7-f536d1851712\") " pod="openshift-ingress/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2" Nov 01 21:08:05.607835 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.607805 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/297fe96b-76ad-499e-b9b7-f536d1851712-istio-podinfo\") pod \"e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2\" (UID: \"297fe96b-76ad-499e-b9b7-f536d1851712\") " pod="openshift-ingress/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2" Nov 01 21:08:05.668739 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.668661 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mh25\" (UniqueName: \"kubernetes.io/projected/297fe96b-76ad-499e-b9b7-f536d1851712-kube-api-access-9mh25\") pod \"e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2\" (UID: \"297fe96b-76ad-499e-b9b7-f536d1851712\") " pod="openshift-ingress/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2" Nov 01 21:08:05.669091 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.669068 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/297fe96b-76ad-499e-b9b7-f536d1851712-istio-token\") pod \"e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2\" (UID: \"297fe96b-76ad-499e-b9b7-f536d1851712\") " pod="openshift-ingress/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2" Nov 01 21:08:05.730101 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.730064 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2" Nov 01 21:08:05.950707 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:05.950601 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:06.171344 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:06.171310 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2"] Nov 01 21:08:06.171952 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:08:06.171919 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod297fe96b_76ad_499e_b9b7_f536d1851712.slice/crio-2819b88d9211839434141b0623eeceffa5ba014cda4f1bb3c8d391c4cf5626a2 WatchSource:0}: Error finding container 2819b88d9211839434141b0623eeceffa5ba014cda4f1bb3c8d391c4cf5626a2: Status 404 returned error can't find the container with id 2819b88d9211839434141b0623eeceffa5ba014cda4f1bb3c8d391c4cf5626a2 Nov 01 21:08:06.174935 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:06.174900 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:08:06.175052 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:06.175014 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:08:06.175052 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:06.175049 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:08:06.996057 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:06.996016 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2" event={"ID":"297fe96b-76ad-499e-b9b7-f536d1851712","Type":"ContainerStarted","Data":"0bb6eab4f96a0a4c0b346f8fa050f8575cf18099997779b566985b9b7a370e1b"} Nov 01 21:08:06.996538 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:06.996075 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2" event={"ID":"297fe96b-76ad-499e-b9b7-f536d1851712","Type":"ContainerStarted","Data":"2819b88d9211839434141b0623eeceffa5ba014cda4f1bb3c8d391c4cf5626a2"} Nov 01 21:08:07.121138 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:07.121062 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2" podStartSLOduration=2.121041775 podStartE2EDuration="2.121041775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:08:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:08:07.118288025 +0000 UTC m=+2237.895928155" watchObservedRunningTime="2026-11-01 21:08:07.121041775 +0000 UTC m=+2237.898681911" Nov 01 21:08:07.285639 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:07.285548 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:07.731168 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:07.731081 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2" Nov 01 21:08:07.737344 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:07.737314 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2" Nov 01 21:08:08.003947 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:08.003919 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2" Nov 01 21:08:08.005109 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:08.005088 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2" Nov 01 21:08:09.975496 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:09.975461 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:10.148486 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:10.148437 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:10.317778 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:10.317742 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:11.770806 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:11.770770 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:13.001296 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:13.001233 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:13.173522 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:13.173486 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:13.356342 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:13.356220 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:13.544419 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:13.544351 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:13.640012 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:13.639921 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a41242-5956bf7979-dsqpz"] Nov 01 21:08:13.645358 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:13.645335 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a41242-5956bf7979-dsqpz" Nov 01 21:08:13.655318 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:13.655288 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcqbj\" (UniqueName: \"kubernetes.io/projected/b2d3a183-2efc-4840-be4d-6d583e48f3e2-kube-api-access-dcqbj\") pod \"payload-pre-processing-e2e-api-b-a41242-5956bf7979-dsqpz\" (UID: \"b2d3a183-2efc-4840-be4d-6d583e48f3e2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a41242-5956bf7979-dsqpz" Nov 01 21:08:13.655471 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:13.655451 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2d3a183-2efc-4840-be4d-6d583e48f3e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a41242-5956bf7979-dsqpz\" (UID: \"b2d3a183-2efc-4840-be4d-6d583e48f3e2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a41242-5956bf7979-dsqpz" Nov 01 21:08:13.728918 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:13.728871 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a41242-5956bf7979-dsqpz"] Nov 01 21:08:13.861028 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:13.860452 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-a41242-dockercfg-k9qgt\"" Nov 01 21:08:13.861028 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:13.860545 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-a41242\"" Nov 01 21:08:13.870454 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:13.866442 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcqbj\" (UniqueName: \"kubernetes.io/projected/b2d3a183-2efc-4840-be4d-6d583e48f3e2-kube-api-access-dcqbj\") pod \"payload-pre-processing-e2e-api-b-a41242-5956bf7979-dsqpz\" (UID: \"b2d3a183-2efc-4840-be4d-6d583e48f3e2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a41242-5956bf7979-dsqpz" Nov 01 21:08:13.870454 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:13.866567 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2d3a183-2efc-4840-be4d-6d583e48f3e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a41242-5956bf7979-dsqpz\" (UID: \"b2d3a183-2efc-4840-be4d-6d583e48f3e2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a41242-5956bf7979-dsqpz" Nov 01 21:08:13.870454 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:13.867551 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:13.870454 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:13.868957 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2d3a183-2efc-4840-be4d-6d583e48f3e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a41242-5956bf7979-dsqpz\" (UID: \"b2d3a183-2efc-4840-be4d-6d583e48f3e2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a41242-5956bf7979-dsqpz" Nov 01 21:08:13.876538 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:13.876505 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a41242-6bfc8bf88f-cs272"] Nov 01 21:08:13.882658 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:13.882631 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a41242-6bfc8bf88f-cs272" Nov 01 21:08:13.884871 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:13.884845 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a41242-6bfc8bf88f-cs272"] Nov 01 21:08:13.929345 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:13.929311 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcqbj\" (UniqueName: \"kubernetes.io/projected/b2d3a183-2efc-4840-be4d-6d583e48f3e2-kube-api-access-dcqbj\") pod \"payload-pre-processing-e2e-api-b-a41242-5956bf7979-dsqpz\" (UID: \"b2d3a183-2efc-4840-be4d-6d583e48f3e2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a41242-5956bf7979-dsqpz" Nov 01 21:08:13.961299 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:13.961260 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a41242-5956bf7979-dsqpz" Nov 01 21:08:13.967435 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:13.967405 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d919681-4cac-4aba-836b-88051de45b3f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a41242-6bfc8bf88f-cs272\" (UID: \"2d919681-4cac-4aba-836b-88051de45b3f\") " pod="openshift-ingress/payload-processing-e2e-api-b-a41242-6bfc8bf88f-cs272" Nov 01 21:08:13.967599 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:13.967574 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqmsz\" (UniqueName: \"kubernetes.io/projected/2d919681-4cac-4aba-836b-88051de45b3f-kube-api-access-jqmsz\") pod \"payload-processing-e2e-api-b-a41242-6bfc8bf88f-cs272\" (UID: \"2d919681-4cac-4aba-836b-88051de45b3f\") " pod="openshift-ingress/payload-processing-e2e-api-b-a41242-6bfc8bf88f-cs272" Nov 01 21:08:14.049406 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:14.049370 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:14.068756 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:14.068707 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d919681-4cac-4aba-836b-88051de45b3f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a41242-6bfc8bf88f-cs272\" (UID: \"2d919681-4cac-4aba-836b-88051de45b3f\") " pod="openshift-ingress/payload-processing-e2e-api-b-a41242-6bfc8bf88f-cs272" Nov 01 21:08:14.068918 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:14.068856 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqmsz\" (UniqueName: \"kubernetes.io/projected/2d919681-4cac-4aba-836b-88051de45b3f-kube-api-access-jqmsz\") pod \"payload-processing-e2e-api-b-a41242-6bfc8bf88f-cs272\" (UID: \"2d919681-4cac-4aba-836b-88051de45b3f\") " pod="openshift-ingress/payload-processing-e2e-api-b-a41242-6bfc8bf88f-cs272" Nov 01 21:08:14.069798 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:14.069772 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d919681-4cac-4aba-836b-88051de45b3f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a41242-6bfc8bf88f-cs272\" (UID: \"2d919681-4cac-4aba-836b-88051de45b3f\") " pod="openshift-ingress/payload-processing-e2e-api-b-a41242-6bfc8bf88f-cs272" Nov 01 21:08:14.130107 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:14.130068 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqmsz\" (UniqueName: \"kubernetes.io/projected/2d919681-4cac-4aba-836b-88051de45b3f-kube-api-access-jqmsz\") pod \"payload-processing-e2e-api-b-a41242-6bfc8bf88f-cs272\" (UID: \"2d919681-4cac-4aba-836b-88051de45b3f\") " pod="openshift-ingress/payload-processing-e2e-api-b-a41242-6bfc8bf88f-cs272" Nov 01 21:08:14.198344 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:14.198237 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a41242-6bfc8bf88f-cs272" Nov 01 21:08:14.209849 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:14.209820 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:14.275511 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:14.275474 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d08fd0-9z2m5"] Nov 01 21:08:14.280421 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:14.280390 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d08fd0-9z2m5"] Nov 01 21:08:14.387478 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:14.387452 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a41242-5956bf7979-dsqpz"] Nov 01 21:08:14.387744 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:08:14.387722 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2d3a183_2efc_4840_be4d_6d583e48f3e2.slice/crio-285944cd3e4c18903398dfa40e6871dc7b3a340cb9d845d33d0d30b6fbf1b382 WatchSource:0}: Error finding container 285944cd3e4c18903398dfa40e6871dc7b3a340cb9d845d33d0d30b6fbf1b382: Status 404 returned error can't find the container with id 285944cd3e4c18903398dfa40e6871dc7b3a340cb9d845d33d0d30b6fbf1b382 Nov 01 21:08:14.579798 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:14.579764 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a41242-6bfc8bf88f-cs272"] Nov 01 21:08:14.580799 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:08:14.580769 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d919681_4cac_4aba_836b_88051de45b3f.slice/crio-66e1e466c983332a3b86067054d944f3a03a3409db3f92940a6f8dc314c3dcc1 WatchSource:0}: Error finding container 66e1e466c983332a3b86067054d944f3a03a3409db3f92940a6f8dc314c3dcc1: Status 404 returned error can't find the container with id 66e1e466c983332a3b86067054d944f3a03a3409db3f92940a6f8dc314c3dcc1 Nov 01 21:08:14.684360 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:14.684329 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:15.166640 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:15.166594 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a41242-6bfc8bf88f-cs272" event={"ID":"2d919681-4cac-4aba-836b-88051de45b3f","Type":"ContainerStarted","Data":"b68f986cf7861b744670b0766727381103890969bca916cdd51cea94e4687cc2"} Nov 01 21:08:15.167126 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:15.166671 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-a41242-6bfc8bf88f-cs272" Nov 01 21:08:15.167126 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:15.166694 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a41242-6bfc8bf88f-cs272" event={"ID":"2d919681-4cac-4aba-836b-88051de45b3f","Type":"ContainerStarted","Data":"66e1e466c983332a3b86067054d944f3a03a3409db3f92940a6f8dc314c3dcc1"} Nov 01 21:08:15.168483 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:15.168457 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a41242-5956bf7979-dsqpz" event={"ID":"b2d3a183-2efc-4840-be4d-6d583e48f3e2","Type":"ContainerStarted","Data":"c9fcc666b4805c194351590c916ac68dd661fd6120913d6792834661f4160bfe"} Nov 01 21:08:15.168601 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:15.168499 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a41242-5956bf7979-dsqpz" Nov 01 21:08:15.168601 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:15.168517 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a41242-5956bf7979-dsqpz" event={"ID":"b2d3a183-2efc-4840-be4d-6d583e48f3e2","Type":"ContainerStarted","Data":"285944cd3e4c18903398dfa40e6871dc7b3a340cb9d845d33d0d30b6fbf1b382"} Nov 01 21:08:15.282909 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:15.282860 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-a41242-6bfc8bf88f-cs272" podStartSLOduration=2.2828461190000002 podStartE2EDuration="2.282846119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:08:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:08:15.282422149 +0000 UTC m=+2246.060062286" watchObservedRunningTime="2026-11-01 21:08:15.282846119 +0000 UTC m=+2246.060486254" Nov 01 21:08:15.373851 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:15.373820 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71efab02-15fb-41db-87d8-251a4cf92947" path="/var/lib/kubelet/pods/71efab02-15fb-41db-87d8-251a4cf92947/volumes" Nov 01 21:08:15.430067 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:15.429714 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a41242-5956bf7979-dsqpz" podStartSLOduration=2.429690811 podStartE2EDuration="2.429690811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:08:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:08:15.412142851 +0000 UTC m=+2246.189782988" watchObservedRunningTime="2026-11-01 21:08:15.429690811 +0000 UTC m=+2246.207330948" Nov 01 21:08:15.767755 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:08:15.767721 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:08:17.195244 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:17.195208 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-dggk5"] Nov 01 21:08:17.199810 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:17.199789 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-dggk5" Nov 01 21:08:17.313874 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:17.313832 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da24464-e245-4061-a89b-9e9e9f61036d-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-dggk5\" (UID: \"8da24464-e245-4061-a89b-9e9e9f61036d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-dggk5" Nov 01 21:08:17.314082 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:17.313990 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8da24464-e245-4061-a89b-9e9e9f61036d-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-dggk5\" (UID: \"8da24464-e245-4061-a89b-9e9e9f61036d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-dggk5" Nov 01 21:08:17.314182 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:17.314150 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8da24464-e245-4061-a89b-9e9e9f61036d-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-dggk5\" (UID: \"8da24464-e245-4061-a89b-9e9e9f61036d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-dggk5" Nov 01 21:08:17.314323 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:17.314303 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8da24464-e245-4061-a89b-9e9e9f61036d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-dggk5\" (UID: \"8da24464-e245-4061-a89b-9e9e9f61036d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-dggk5" Nov 01 21:08:17.314458 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:17.314436 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da24464-e245-4061-a89b-9e9e9f61036d-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-dggk5\" (UID: \"8da24464-e245-4061-a89b-9e9e9f61036d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-dggk5" Nov 01 21:08:17.314557 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:17.314536 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8da24464-e245-4061-a89b-9e9e9f61036d-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-dggk5\" (UID: \"8da24464-e245-4061-a89b-9e9e9f61036d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-dggk5" Nov 01 21:08:17.314640 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:17.314622 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8da24464-e245-4061-a89b-9e9e9f61036d-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-dggk5\" (UID: \"8da24464-e245-4061-a89b-9e9e9f61036d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-dggk5" Nov 01 21:08:17.314683 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:17.314666 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh782\" (UniqueName: \"kubernetes.io/projected/8da24464-e245-4061-a89b-9e9e9f61036d-kube-api-access-vh782\") pod \"maas-default-gateway-openshift-default-58c6585447-dggk5\" (UID: \"8da24464-e245-4061-a89b-9e9e9f61036d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-dggk5" Nov 01 21:08:17.314747 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:17.314726 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8da24464-e245-4061-a89b-9e9e9f61036d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-dggk5\" (UID: \"8da24464-e245-4061-a89b-9e9e9f61036d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-dggk5" Nov 01 21:08:17.400561 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:17.400527 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-dggk5"] Nov 01 21:08:17.416477 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:17.416441 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8da24464-e245-4061-a89b-9e9e9f61036d-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-dggk5\" (UID: \"8da24464-e245-4061-a89b-9e9e9f61036d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-dggk5" Nov 01 21:08:17.416965 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:17.416931 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8da24464-e245-4061-a89b-9e9e9f61036d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-dggk5\" (UID: \"8da24464-e245-4061-a89b-9e9e9f61036d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-dggk5" Nov 01 21:08:17.417243 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:17.417224 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da24464-e245-4061-a89b-9e9e9f61036d-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-dggk5\" (UID: \"8da24464-e245-4061-a89b-9e9e9f61036d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-dggk5" Nov 01 21:08:17.417478 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:17.417459 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8da24464-e245-4061-a89b-9e9e9f61036d-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-dggk5\" (UID: \"8da24464-e245-4061-a89b-9e9e9f61036d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-dggk5" Nov 01 21:08:17.417625 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:17.417607 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8da24464-e245-4061-a89b-9e9e9f61036d-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-dggk5\" (UID: \"8da24464-e245-4061-a89b-9e9e9f61036d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-dggk5" Nov 01 21:08:17.417762 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:17.417743 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh782\" (UniqueName: \"kubernetes.io/projected/8da24464-e245-4061-a89b-9e9e9f61036d-kube-api-access-vh782\") pod \"maas-default-gateway-openshift-default-58c6585447-dggk5\" (UID: \"8da24464-e245-4061-a89b-9e9e9f61036d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-dggk5" Nov 01 21:08:17.417884 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:17.417870 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8da24464-e245-4061-a89b-9e9e9f61036d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-dggk5\" (UID: \"8da24464-e245-4061-a89b-9e9e9f61036d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-dggk5" Nov 01 21:08:17.417981 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:17.417954 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da24464-e245-4061-a89b-9e9e9f61036d-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-dggk5\" (UID: \"8da24464-e245-4061-a89b-9e9e9f61036d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-dggk5" Nov 01 21:08:17.418146 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:17.418127 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da24464-e245-4061-a89b-9e9e9f61036d-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-dggk5\" (UID: \"8da24464-e245-4061-a89b-9e9e9f61036d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-dggk5" Nov 01 21:08:17.418337 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:17.418319 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8da24464-e245-4061-a89b-9e9e9f61036d-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-dggk5\" (UID: \"8da24464-e245-4061-a89b-9e9e9f61036d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-dggk5" Nov 01 21:08:17.418862 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:17.418839 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8da24464-e245-4061-a89b-9e9e9f61036d-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-dggk5\" (UID: \"8da24464-e245-4061-a89b-9e9e9f61036d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-dggk5" Nov 01 21:08:17.419281 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:17.419253 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8da24464-e245-4061-a89b-9e9e9f61036d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-dggk5\" (UID: \"8da24464-e245-4061-a89b-9e9e9f61036d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-dggk5" Nov 01 21:08:17.419951 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:17.419926 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8da24464-e245-4061-a89b-9e9e9f61036d-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-dggk5\" (UID: \"8da24464-e245-4061-a89b-9e9e9f61036d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-dggk5" Nov 01 21:08:17.420262 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:17.420243 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da24464-e245-4061-a89b-9e9e9f61036d-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-dggk5\" (UID: \"8da24464-e245-4061-a89b-9e9e9f61036d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-dggk5" Nov 01 21:08:17.422301 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:17.422271 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8da24464-e245-4061-a89b-9e9e9f61036d-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-dggk5\" (UID: \"8da24464-e245-4061-a89b-9e9e9f61036d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-dggk5" Nov 01 21:08:17.422857 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:17.422831 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8da24464-e245-4061-a89b-9e9e9f61036d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-dggk5\" (UID: \"8da24464-e245-4061-a89b-9e9e9f61036d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-dggk5" Nov 01 21:08:17.476890 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:17.476818 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh782\" (UniqueName: \"kubernetes.io/projected/8da24464-e245-4061-a89b-9e9e9f61036d-kube-api-access-vh782\") pod \"maas-default-gateway-openshift-default-58c6585447-dggk5\" (UID: \"8da24464-e245-4061-a89b-9e9e9f61036d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-dggk5" Nov 01 21:08:17.476890 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:17.476819 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8da24464-e245-4061-a89b-9e9e9f61036d-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-dggk5\" (UID: \"8da24464-e245-4061-a89b-9e9e9f61036d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-dggk5" Nov 01 21:08:17.516714 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:17.516675 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-dggk5" Nov 01 21:08:17.957329 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:17.957299 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-dggk5"] Nov 01 21:08:17.957428 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:08:17.957378 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8da24464_e245_4061_a89b_9e9e9f61036d.slice/crio-230d184dc80cbdfa017a89ce76352edc7cf13ab354e82386bddf7e0b75e6f96a WatchSource:0}: Error finding container 230d184dc80cbdfa017a89ce76352edc7cf13ab354e82386bddf7e0b75e6f96a: Status 404 returned error can't find the container with id 230d184dc80cbdfa017a89ce76352edc7cf13ab354e82386bddf7e0b75e6f96a Nov 01 21:08:17.960277 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:17.960245 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:08:17.960346 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:17.960312 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:08:17.960384 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:17.960344 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:08:18.204708 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:18.204611 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-dggk5" event={"ID":"8da24464-e245-4061-a89b-9e9e9f61036d","Type":"ContainerStarted","Data":"eaa64059018ae201593fde21c20af3245ded97dabd4da9ff05a906d47dde4fb9"} Nov 01 21:08:18.204708 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:18.204672 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-dggk5" event={"ID":"8da24464-e245-4061-a89b-9e9e9f61036d","Type":"ContainerStarted","Data":"230d184dc80cbdfa017a89ce76352edc7cf13ab354e82386bddf7e0b75e6f96a"} Nov 01 21:08:18.328169 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:18.328106 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-dggk5" podStartSLOduration=1.3280913810000001 podStartE2EDuration="1.328091381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:08:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:08:18.325870669 +0000 UTC m=+2249.103510804" watchObservedRunningTime="2026-11-01 21:08:18.328091381 +0000 UTC m=+2249.105731642" Nov 01 21:08:18.517722 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:18.517688 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-dggk5" Nov 01 21:08:19.029569 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:19.029526 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:19.522518 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:19.522478 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-dggk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" start-of-body= Nov 01 21:08:19.523166 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:19.523130 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-dggk5" podUID="8da24464-e245-4061-a89b-9e9e9f61036d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" Nov 01 21:08:20.531055 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:20.529661 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-dggk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" start-of-body= Nov 01 21:08:20.531055 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:20.529743 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-dggk5" podUID="8da24464-e245-4061-a89b-9e9e9f61036d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" Nov 01 21:08:20.871464 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:20.871381 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:21.059456 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:21.059404 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:21.167390 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:21.167297 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z"] Nov 01 21:08:21.167597 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:21.167568 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z" podUID="21a7e141-0aee-418a-99ca-1b5c3993c303" containerName="istio-proxy" containerID="cri-o://766aebdb3136f65bcf6239952a9446716f79d8ccef470d5e2f44baffc7f4e155" gracePeriod=30 Nov 01 21:08:21.284041 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:21.283947 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:21.522223 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:21.522184 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-dggk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" start-of-body= Nov 01 21:08:21.522532 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:21.522501 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-dggk5" podUID="8da24464-e245-4061-a89b-9e9e9f61036d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" Nov 01 21:08:21.943080 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:21.942960 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:22.434665 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:22.434631 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk"] Nov 01 21:08:22.440360 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:22.440336 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk" Nov 01 21:08:22.494397 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:22.494362 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-3ab520-openshift-default-dockercfg-sd8xq\"" Nov 01 21:08:22.496255 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:22.496224 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-workload-socket\") pod \"e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk\" (UID: \"2d4f02fb-d74b-40e5-b470-f8d1e65b120c\") " pod="openshift-ingress/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk" Nov 01 21:08:22.496532 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:22.496510 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-istiod-ca-cert\") pod \"e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk\" (UID: \"2d4f02fb-d74b-40e5-b470-f8d1e65b120c\") " pod="openshift-ingress/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk" Nov 01 21:08:22.496723 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:22.496706 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-credential-socket\") pod \"e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk\" (UID: \"2d4f02fb-d74b-40e5-b470-f8d1e65b120c\") " pod="openshift-ingress/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk" Nov 01 21:08:22.496899 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:22.496884 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-istio-podinfo\") pod \"e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk\" (UID: \"2d4f02fb-d74b-40e5-b470-f8d1e65b120c\") " pod="openshift-ingress/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk" Nov 01 21:08:22.518448 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:22.518411 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-dggk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" start-of-body= Nov 01 21:08:22.518726 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:22.518699 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-dggk5" podUID="8da24464-e245-4061-a89b-9e9e9f61036d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" Nov 01 21:08:22.520502 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:22.520477 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-workload-certs\") pod \"e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk\" (UID: \"2d4f02fb-d74b-40e5-b470-f8d1e65b120c\") " pod="openshift-ingress/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk" Nov 01 21:08:22.534211 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:22.534175 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-istio-envoy\") pod \"e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk\" (UID: \"2d4f02fb-d74b-40e5-b470-f8d1e65b120c\") " pod="openshift-ingress/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk" Nov 01 21:08:22.534649 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:22.534625 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klhsm\" (UniqueName: \"kubernetes.io/projected/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-kube-api-access-klhsm\") pod \"e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk\" (UID: \"2d4f02fb-d74b-40e5-b470-f8d1e65b120c\") " pod="openshift-ingress/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk" Nov 01 21:08:22.534937 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:22.534919 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-istio-token\") pod \"e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk\" (UID: \"2d4f02fb-d74b-40e5-b470-f8d1e65b120c\") " pod="openshift-ingress/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk" Nov 01 21:08:22.535126 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:22.535110 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-istio-data\") pod \"e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk\" (UID: \"2d4f02fb-d74b-40e5-b470-f8d1e65b120c\") " pod="openshift-ingress/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk" Nov 01 21:08:22.596495 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:22.596455 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk"] Nov 01 21:08:22.646031 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:22.629273 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:22.668396 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:22.662523 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-istio-envoy\") pod \"e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk\" (UID: \"2d4f02fb-d74b-40e5-b470-f8d1e65b120c\") " pod="openshift-ingress/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk" Nov 01 21:08:22.668812 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:22.668773 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klhsm\" (UniqueName: \"kubernetes.io/projected/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-kube-api-access-klhsm\") pod \"e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk\" (UID: \"2d4f02fb-d74b-40e5-b470-f8d1e65b120c\") " pod="openshift-ingress/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk" Nov 01 21:08:22.669049 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:22.669019 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-istio-token\") pod \"e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk\" (UID: \"2d4f02fb-d74b-40e5-b470-f8d1e65b120c\") " pod="openshift-ingress/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk" Nov 01 21:08:22.669237 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:22.669206 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-istio-data\") pod \"e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk\" (UID: \"2d4f02fb-d74b-40e5-b470-f8d1e65b120c\") " pod="openshift-ingress/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk" Nov 01 21:08:22.669475 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:22.669442 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-workload-socket\") pod \"e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk\" (UID: \"2d4f02fb-d74b-40e5-b470-f8d1e65b120c\") " pod="openshift-ingress/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk" Nov 01 21:08:22.669696 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:22.669664 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-istiod-ca-cert\") pod \"e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk\" (UID: \"2d4f02fb-d74b-40e5-b470-f8d1e65b120c\") " pod="openshift-ingress/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk" Nov 01 21:08:22.669892 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:22.669861 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-credential-socket\") pod \"e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk\" (UID: \"2d4f02fb-d74b-40e5-b470-f8d1e65b120c\") " pod="openshift-ingress/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk" Nov 01 21:08:22.670104 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:22.670074 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-istio-podinfo\") pod \"e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk\" (UID: \"2d4f02fb-d74b-40e5-b470-f8d1e65b120c\") " pod="openshift-ingress/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk" Nov 01 21:08:22.670360 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:22.670330 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-workload-certs\") pod \"e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk\" (UID: \"2d4f02fb-d74b-40e5-b470-f8d1e65b120c\") " pod="openshift-ingress/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk" Nov 01 21:08:22.671011 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:22.670967 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-workload-certs\") pod \"e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk\" (UID: \"2d4f02fb-d74b-40e5-b470-f8d1e65b120c\") " pod="openshift-ingress/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk" Nov 01 21:08:22.671943 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:22.671913 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-istio-data\") pod \"e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk\" (UID: \"2d4f02fb-d74b-40e5-b470-f8d1e65b120c\") " pod="openshift-ingress/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk" Nov 01 21:08:22.672887 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:22.672859 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-workload-socket\") pod \"e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk\" (UID: \"2d4f02fb-d74b-40e5-b470-f8d1e65b120c\") " pod="openshift-ingress/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk" Nov 01 21:08:22.673760 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:22.673724 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-istiod-ca-cert\") pod \"e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk\" (UID: \"2d4f02fb-d74b-40e5-b470-f8d1e65b120c\") " pod="openshift-ingress/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk" Nov 01 21:08:22.674215 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:22.674180 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-credential-socket\") pod \"e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk\" (UID: \"2d4f02fb-d74b-40e5-b470-f8d1e65b120c\") " pod="openshift-ingress/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk" Nov 01 21:08:22.679317 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:22.679289 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-istio-envoy\") pod \"e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk\" (UID: \"2d4f02fb-d74b-40e5-b470-f8d1e65b120c\") " pod="openshift-ingress/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk" Nov 01 21:08:22.687180 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:22.687112 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-istio-podinfo\") pod \"e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk\" (UID: \"2d4f02fb-d74b-40e5-b470-f8d1e65b120c\") " pod="openshift-ingress/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk" Nov 01 21:08:22.744446 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:22.744410 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-istio-token\") pod \"e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk\" (UID: \"2d4f02fb-d74b-40e5-b470-f8d1e65b120c\") " pod="openshift-ingress/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk" Nov 01 21:08:22.745323 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:22.745298 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klhsm\" (UniqueName: \"kubernetes.io/projected/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-kube-api-access-klhsm\") pod \"e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk\" (UID: \"2d4f02fb-d74b-40e5-b470-f8d1e65b120c\") " pod="openshift-ingress/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk" Nov 01 21:08:22.780082 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:22.780044 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk" Nov 01 21:08:22.966930 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:22.966772 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:23.241508 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:23.241478 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk"] Nov 01 21:08:23.243142 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:08:23.243116 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d4f02fb_d74b_40e5_b470_f8d1e65b120c.slice/crio-da06263bdf5430453760dbe738495cea3896b8e9abe99d832ef89c9e1cf118c4 WatchSource:0}: Error finding container da06263bdf5430453760dbe738495cea3896b8e9abe99d832ef89c9e1cf118c4: Status 404 returned error can't find the container with id da06263bdf5430453760dbe738495cea3896b8e9abe99d832ef89c9e1cf118c4 Nov 01 21:08:23.246036 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:23.245987 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:08:23.246140 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:23.246082 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:08:23.246140 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:23.246122 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:08:23.350809 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:23.350774 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk" event={"ID":"2d4f02fb-d74b-40e5-b470-f8d1e65b120c","Type":"ContainerStarted","Data":"ca2e831ab9700ed80a68b6bf74bd5f3b11c7f2b7bc066043094ff2be8b087e49"} Nov 01 21:08:23.351019 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:23.350839 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk" event={"ID":"2d4f02fb-d74b-40e5-b470-f8d1e65b120c","Type":"ContainerStarted","Data":"da06263bdf5430453760dbe738495cea3896b8e9abe99d832ef89c9e1cf118c4"} Nov 01 21:08:23.494108 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:23.493986 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk" podStartSLOduration=1.49397241 podStartE2EDuration="1.49397241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:08:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:08:23.493177938 +0000 UTC m=+2254.270818086" watchObservedRunningTime="2026-11-01 21:08:23.49397241 +0000 UTC m=+2254.271612544" Nov 01 21:08:23.521030 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:23.520419 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-dggk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" start-of-body= Nov 01 21:08:23.521030 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:23.520477 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-dggk5" podUID="8da24464-e245-4061-a89b-9e9e9f61036d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" Nov 01 21:08:23.780605 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:23.780527 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk" Nov 01 21:08:23.847017 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:23.841278 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:24.517974 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:24.517937 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-dggk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" start-of-body= Nov 01 21:08:24.518389 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:24.518007 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-dggk5" podUID="8da24464-e245-4061-a89b-9e9e9f61036d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" Nov 01 21:08:24.787454 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:24.787373 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk" Nov 01 21:08:25.373158 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:25.373126 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk" Nov 01 21:08:25.373397 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:25.373195 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk" Nov 01 21:08:25.517141 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:25.517108 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-dggk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" start-of-body= Nov 01 21:08:25.517322 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:25.517163 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-dggk5" podUID="8da24464-e245-4061-a89b-9e9e9f61036d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" Nov 01 21:08:26.179441 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:26.179269 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a41242-5956bf7979-dsqpz" Nov 01 21:08:26.179781 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:26.179474 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-a41242-6bfc8bf88f-cs272" Nov 01 21:08:26.395601 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:26.395562 4484 generic.go:356] "Generic (PLEG): container finished" podID="21a7e141-0aee-418a-99ca-1b5c3993c303" containerID="766aebdb3136f65bcf6239952a9446716f79d8ccef470d5e2f44baffc7f4e155" exitCode=0 Nov 01 21:08:26.396816 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:26.396786 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z" event={"ID":"21a7e141-0aee-418a-99ca-1b5c3993c303","Type":"ContainerDied","Data":"766aebdb3136f65bcf6239952a9446716f79d8ccef470d5e2f44baffc7f4e155"} Nov 01 21:08:26.397070 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:26.397048 4484 scope.go:122] "RemoveContainer" containerID="506e6634ce972c722bec301c2d798fe34ee03b584c2dae5dbfb440fa2ef5a6b1" Nov 01 21:08:26.523846 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:26.523768 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z" Nov 01 21:08:26.628832 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:26.628787 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21a7e141-0aee-418a-99ca-1b5c3993c303-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21a7e141-0aee-418a-99ca-1b5c3993c303-istio-podinfo\") pod \"21a7e141-0aee-418a-99ca-1b5c3993c303\" (UID: \"21a7e141-0aee-418a-99ca-1b5c3993c303\") " Nov 01 21:08:26.629050 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:26.628928 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21a7e141-0aee-418a-99ca-1b5c3993c303-kube-api-access-gs5qf\" (UniqueName: \"kubernetes.io/projected/21a7e141-0aee-418a-99ca-1b5c3993c303-kube-api-access-gs5qf\") pod \"21a7e141-0aee-418a-99ca-1b5c3993c303\" (UID: \"21a7e141-0aee-418a-99ca-1b5c3993c303\") " Nov 01 21:08:26.629050 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:26.628977 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21a7e141-0aee-418a-99ca-1b5c3993c303-istio-token\" (UniqueName: \"kubernetes.io/projected/21a7e141-0aee-418a-99ca-1b5c3993c303-istio-token\") pod \"21a7e141-0aee-418a-99ca-1b5c3993c303\" (UID: \"21a7e141-0aee-418a-99ca-1b5c3993c303\") " Nov 01 21:08:26.629207 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:26.629076 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21a7e141-0aee-418a-99ca-1b5c3993c303-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21a7e141-0aee-418a-99ca-1b5c3993c303-workload-certs\") pod \"21a7e141-0aee-418a-99ca-1b5c3993c303\" (UID: \"21a7e141-0aee-418a-99ca-1b5c3993c303\") " Nov 01 21:08:26.629207 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:26.629112 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21a7e141-0aee-418a-99ca-1b5c3993c303-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21a7e141-0aee-418a-99ca-1b5c3993c303-istio-envoy\") pod \"21a7e141-0aee-418a-99ca-1b5c3993c303\" (UID: \"21a7e141-0aee-418a-99ca-1b5c3993c303\") " Nov 01 21:08:26.629320 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:26.629255 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21a7e141-0aee-418a-99ca-1b5c3993c303-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21a7e141-0aee-418a-99ca-1b5c3993c303-credential-socket\") pod \"21a7e141-0aee-418a-99ca-1b5c3993c303\" (UID: \"21a7e141-0aee-418a-99ca-1b5c3993c303\") " Nov 01 21:08:26.629376 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:26.629335 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21a7e141-0aee-418a-99ca-1b5c3993c303-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21a7e141-0aee-418a-99ca-1b5c3993c303-istio-data\") pod \"21a7e141-0aee-418a-99ca-1b5c3993c303\" (UID: \"21a7e141-0aee-418a-99ca-1b5c3993c303\") " Nov 01 21:08:26.629376 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:26.629366 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21a7e141-0aee-418a-99ca-1b5c3993c303-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21a7e141-0aee-418a-99ca-1b5c3993c303-workload-socket\") pod \"21a7e141-0aee-418a-99ca-1b5c3993c303\" (UID: \"21a7e141-0aee-418a-99ca-1b5c3993c303\") " Nov 01 21:08:26.629488 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:26.629448 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21a7e141-0aee-418a-99ca-1b5c3993c303-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21a7e141-0aee-418a-99ca-1b5c3993c303-istiod-ca-cert\") pod \"21a7e141-0aee-418a-99ca-1b5c3993c303\" (UID: \"21a7e141-0aee-418a-99ca-1b5c3993c303\") " Nov 01 21:08:26.629547 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:26.629502 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21a7e141-0aee-418a-99ca-1b5c3993c303-workload-certs" pod "21a7e141-0aee-418a-99ca-1b5c3993c303" (UID: "21a7e141-0aee-418a-99ca-1b5c3993c303"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:08:26.629748 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:26.629720 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21a7e141-0aee-418a-99ca-1b5c3993c303-credential-socket" pod "21a7e141-0aee-418a-99ca-1b5c3993c303" (UID: "21a7e141-0aee-418a-99ca-1b5c3993c303"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:08:26.630260 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:26.630226 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21a7e141-0aee-418a-99ca-1b5c3993c303-workload-socket" pod "21a7e141-0aee-418a-99ca-1b5c3993c303" (UID: "21a7e141-0aee-418a-99ca-1b5c3993c303"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:08:26.630518 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:26.630492 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21a7e141-0aee-418a-99ca-1b5c3993c303-istiod-ca-cert" pod "21a7e141-0aee-418a-99ca-1b5c3993c303" (UID: "21a7e141-0aee-418a-99ca-1b5c3993c303"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:08:26.630598 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:26.630572 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21a7e141-0aee-418a-99ca-1b5c3993c303-istio-data" pod "21a7e141-0aee-418a-99ca-1b5c3993c303" (UID: "21a7e141-0aee-418a-99ca-1b5c3993c303"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:08:26.630759 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:26.630738 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21a7e141-0aee-418a-99ca-1b5c3993c303-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:26.630847 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:26.630768 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21a7e141-0aee-418a-99ca-1b5c3993c303-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:26.631784 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:26.631757 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21a7e141-0aee-418a-99ca-1b5c3993c303-istio-token" pod "21a7e141-0aee-418a-99ca-1b5c3993c303" (UID: "21a7e141-0aee-418a-99ca-1b5c3993c303"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:08:26.631900 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:26.631870 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21a7e141-0aee-418a-99ca-1b5c3993c303-kube-api-access-gs5qf" pod "21a7e141-0aee-418a-99ca-1b5c3993c303" (UID: "21a7e141-0aee-418a-99ca-1b5c3993c303"). InnerVolumeSpecName "kube-api-access-gs5qf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:08:26.632483 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:26.632451 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21a7e141-0aee-418a-99ca-1b5c3993c303-istio-envoy" pod "21a7e141-0aee-418a-99ca-1b5c3993c303" (UID: "21a7e141-0aee-418a-99ca-1b5c3993c303"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:08:26.632951 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:26.632928 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21a7e141-0aee-418a-99ca-1b5c3993c303-istio-podinfo" pod "21a7e141-0aee-418a-99ca-1b5c3993c303" (UID: "21a7e141-0aee-418a-99ca-1b5c3993c303"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:08:26.712200 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:26.712100 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:26.732474 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:26.732445 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21a7e141-0aee-418a-99ca-1b5c3993c303-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:26.732474 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:26.732473 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21a7e141-0aee-418a-99ca-1b5c3993c303-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:26.732735 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:26.732482 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21a7e141-0aee-418a-99ca-1b5c3993c303-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:26.732735 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:26.732492 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21a7e141-0aee-418a-99ca-1b5c3993c303-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:26.732735 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:26.732502 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gs5qf\" (UniqueName: \"kubernetes.io/projected/21a7e141-0aee-418a-99ca-1b5c3993c303-kube-api-access-gs5qf\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:26.732735 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:26.732510 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21a7e141-0aee-418a-99ca-1b5c3993c303-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:26.732735 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:26.732518 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21a7e141-0aee-418a-99ca-1b5c3993c303-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:26.747572 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:26.744187 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-dggk5" Nov 01 21:08:26.747572 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:26.746650 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-dggk5" Nov 01 21:08:26.747572 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:26.747482 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-dggk5" Nov 01 21:08:26.931200 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:26.931166 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:27.173123 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:27.173091 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:27.404144 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:27.404117 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z" Nov 01 21:08:27.404630 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:27.404156 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z" event={"ID":"21a7e141-0aee-418a-99ca-1b5c3993c303","Type":"ContainerDied","Data":"99fd9e282990a1cefa59ff224c3c7a2f7b0314f00833eabfe50afc6916bccabf"} Nov 01 21:08:27.404630 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:27.404214 4484 scope.go:122] "RemoveContainer" containerID="766aebdb3136f65bcf6239952a9446716f79d8ccef470d5e2f44baffc7f4e155" Nov 01 21:08:27.580283 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:27.580253 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z"] Nov 01 21:08:27.631338 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:27.631306 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-d08fd0-openshift-default-5fd6b47cd7-28k4z"] Nov 01 21:08:28.966899 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:28.966864 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:29.381161 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:29.381134 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21a7e141-0aee-418a-99ca-1b5c3993c303" path="/var/lib/kubelet/pods/21a7e141-0aee-418a-99ca-1b5c3993c303/volumes" Nov 01 21:08:29.996264 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:29.996231 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:30.165584 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:30.165552 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:30.378365 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:30.378284 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3ab520-848dc5bd6c-k9zhf"] Nov 01 21:08:30.379068 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:30.379051 4484 cpu_manager.go:414] "containerMap: removing container" podUID="21a7e141-0aee-418a-99ca-1b5c3993c303" containerName="istio-proxy" Nov 01 21:08:30.379133 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:30.379068 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21a7e141-0aee-418a-99ca-1b5c3993c303" containerName="istio-proxy" Nov 01 21:08:30.379218 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:30.379209 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="21a7e141-0aee-418a-99ca-1b5c3993c303" containerName="istio-proxy" Nov 01 21:08:30.384295 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:30.384275 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3ab520-848dc5bd6c-k9zhf" Nov 01 21:08:30.440094 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:30.440059 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3ab520-848dc5bd6c-k9zhf"] Nov 01 21:08:30.503026 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:30.502870 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-3ab520-dockercfg-zj6mv\"" Nov 01 21:08:30.503026 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:30.502989 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-3ab520\"" Nov 01 21:08:30.517029 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:30.516740 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgt58\" (UniqueName: \"kubernetes.io/projected/23aa6e10-cab2-459f-8bfe-f1f4772e343a-kube-api-access-bgt58\") pod \"payload-pre-processing-e2e-inf-a-3ab520-848dc5bd6c-k9zhf\" (UID: \"23aa6e10-cab2-459f-8bfe-f1f4772e343a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3ab520-848dc5bd6c-k9zhf" Nov 01 21:08:30.517419 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:30.517124 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23aa6e10-cab2-459f-8bfe-f1f4772e343a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3ab520-848dc5bd6c-k9zhf\" (UID: \"23aa6e10-cab2-459f-8bfe-f1f4772e343a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3ab520-848dc5bd6c-k9zhf" Nov 01 21:08:30.599762 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:30.596480 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3ab520-7b7f4ff56c-vszhw"] Nov 01 21:08:30.599762 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:30.597685 4484 cpu_manager.go:414] "containerMap: removing container" podUID="21a7e141-0aee-418a-99ca-1b5c3993c303" containerName="istio-proxy" Nov 01 21:08:30.599762 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:30.597707 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21a7e141-0aee-418a-99ca-1b5c3993c303" containerName="istio-proxy" Nov 01 21:08:30.599762 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:30.597837 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="21a7e141-0aee-418a-99ca-1b5c3993c303" containerName="istio-proxy" Nov 01 21:08:30.605406 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:30.605332 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3ab520-7b7f4ff56c-vszhw"] Nov 01 21:08:30.605592 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:30.605426 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3ab520-7b7f4ff56c-vszhw" Nov 01 21:08:30.619047 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:30.618975 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23aa6e10-cab2-459f-8bfe-f1f4772e343a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3ab520-848dc5bd6c-k9zhf\" (UID: \"23aa6e10-cab2-459f-8bfe-f1f4772e343a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3ab520-848dc5bd6c-k9zhf" Nov 01 21:08:30.619208 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:30.619157 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgt58\" (UniqueName: \"kubernetes.io/projected/23aa6e10-cab2-459f-8bfe-f1f4772e343a-kube-api-access-bgt58\") pod \"payload-pre-processing-e2e-inf-a-3ab520-848dc5bd6c-k9zhf\" (UID: \"23aa6e10-cab2-459f-8bfe-f1f4772e343a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3ab520-848dc5bd6c-k9zhf" Nov 01 21:08:30.621768 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:30.621742 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23aa6e10-cab2-459f-8bfe-f1f4772e343a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3ab520-848dc5bd6c-k9zhf\" (UID: \"23aa6e10-cab2-459f-8bfe-f1f4772e343a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3ab520-848dc5bd6c-k9zhf" Nov 01 21:08:30.678581 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:30.678501 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgt58\" (UniqueName: \"kubernetes.io/projected/23aa6e10-cab2-459f-8bfe-f1f4772e343a-kube-api-access-bgt58\") pod \"payload-pre-processing-e2e-inf-a-3ab520-848dc5bd6c-k9zhf\" (UID: \"23aa6e10-cab2-459f-8bfe-f1f4772e343a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3ab520-848dc5bd6c-k9zhf" Nov 01 21:08:30.699081 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:30.699044 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3ab520-848dc5bd6c-k9zhf" Nov 01 21:08:30.720905 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:30.720845 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45202ae4-063e-42eb-a800-a7118450a9e5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3ab520-7b7f4ff56c-vszhw\" (UID: \"45202ae4-063e-42eb-a800-a7118450a9e5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3ab520-7b7f4ff56c-vszhw" Nov 01 21:08:30.721082 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:30.721012 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbjtk\" (UniqueName: \"kubernetes.io/projected/45202ae4-063e-42eb-a800-a7118450a9e5-kube-api-access-qbjtk\") pod \"payload-processing-e2e-inf-a-3ab520-7b7f4ff56c-vszhw\" (UID: \"45202ae4-063e-42eb-a800-a7118450a9e5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3ab520-7b7f4ff56c-vszhw" Nov 01 21:08:30.822776 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:30.822735 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45202ae4-063e-42eb-a800-a7118450a9e5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3ab520-7b7f4ff56c-vszhw\" (UID: \"45202ae4-063e-42eb-a800-a7118450a9e5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3ab520-7b7f4ff56c-vszhw" Nov 01 21:08:30.822984 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:30.822789 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbjtk\" (UniqueName: \"kubernetes.io/projected/45202ae4-063e-42eb-a800-a7118450a9e5-kube-api-access-qbjtk\") pod \"payload-processing-e2e-inf-a-3ab520-7b7f4ff56c-vszhw\" (UID: \"45202ae4-063e-42eb-a800-a7118450a9e5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3ab520-7b7f4ff56c-vszhw" Nov 01 21:08:30.823541 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:30.823519 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45202ae4-063e-42eb-a800-a7118450a9e5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3ab520-7b7f4ff56c-vszhw\" (UID: \"45202ae4-063e-42eb-a800-a7118450a9e5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3ab520-7b7f4ff56c-vszhw" Nov 01 21:08:30.880575 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:30.880543 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbjtk\" (UniqueName: \"kubernetes.io/projected/45202ae4-063e-42eb-a800-a7118450a9e5-kube-api-access-qbjtk\") pod \"payload-processing-e2e-inf-a-3ab520-7b7f4ff56c-vszhw\" (UID: \"45202ae4-063e-42eb-a800-a7118450a9e5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3ab520-7b7f4ff56c-vszhw" Nov 01 21:08:30.924096 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:30.924057 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3ab520-7b7f4ff56c-vszhw" Nov 01 21:08:31.080871 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:31.080844 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3ab520-848dc5bd6c-k9zhf"] Nov 01 21:08:31.081267 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:08:31.081211 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23aa6e10_cab2_459f_8bfe_f1f4772e343a.slice/crio-7b261bd949d427bd9325ddac45111c207e012d1ead8898128c95368feacea34b WatchSource:0}: Error finding container 7b261bd949d427bd9325ddac45111c207e012d1ead8898128c95368feacea34b: Status 404 returned error can't find the container with id 7b261bd949d427bd9325ddac45111c207e012d1ead8898128c95368feacea34b Nov 01 21:08:31.302215 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:08:31.302186 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45202ae4_063e_42eb_a800_a7118450a9e5.slice/crio-54072412428de32458a273e4488ad793d57b42be42c890c31ced7506622248ee WatchSource:0}: Error finding container 54072412428de32458a273e4488ad793d57b42be42c890c31ced7506622248ee: Status 404 returned error can't find the container with id 54072412428de32458a273e4488ad793d57b42be42c890c31ced7506622248ee Nov 01 21:08:31.303424 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:31.303397 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3ab520-7b7f4ff56c-vszhw"] Nov 01 21:08:31.435893 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:31.435854 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3ab520-848dc5bd6c-k9zhf" event={"ID":"23aa6e10-cab2-459f-8bfe-f1f4772e343a","Type":"ContainerStarted","Data":"06ba08968a62529995fd28cd07703495d8ea44fb8309af02d4adc98023b2239e"} Nov 01 21:08:31.436243 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:31.435920 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3ab520-848dc5bd6c-k9zhf" event={"ID":"23aa6e10-cab2-459f-8bfe-f1f4772e343a","Type":"ContainerStarted","Data":"7b261bd949d427bd9325ddac45111c207e012d1ead8898128c95368feacea34b"} Nov 01 21:08:31.436243 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:31.435951 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3ab520-848dc5bd6c-k9zhf" Nov 01 21:08:31.437643 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:31.437617 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3ab520-7b7f4ff56c-vszhw" event={"ID":"45202ae4-063e-42eb-a800-a7118450a9e5","Type":"ContainerStarted","Data":"b97410525adb1cafdd39591d28b4635178788a1df7feadf4271d7c396146e005"} Nov 01 21:08:31.437734 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:31.437669 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-3ab520-7b7f4ff56c-vszhw" Nov 01 21:08:31.437734 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:31.437689 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3ab520-7b7f4ff56c-vszhw" event={"ID":"45202ae4-063e-42eb-a800-a7118450a9e5","Type":"ContainerStarted","Data":"54072412428de32458a273e4488ad793d57b42be42c890c31ced7506622248ee"} Nov 01 21:08:31.552235 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:31.552136 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3ab520-848dc5bd6c-k9zhf" podStartSLOduration=1.5521224139999998 podStartE2EDuration="1.552122414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:08:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:08:31.549750261 +0000 UTC m=+2262.327390452" watchObservedRunningTime="2026-11-01 21:08:31.552122414 +0000 UTC m=+2262.329762548" Nov 01 21:08:31.702022 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:31.698133 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-3ab520-7b7f4ff56c-vszhw" podStartSLOduration=1.698110753 podStartE2EDuration="1.698110753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:08:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:08:31.677586476 +0000 UTC m=+2262.455226611" watchObservedRunningTime="2026-11-01 21:08:31.698110753 +0000 UTC m=+2262.475750889" Nov 01 21:08:32.179616 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:32.179578 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qswqs"] Nov 01 21:08:32.185523 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:32.185487 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qswqs" Nov 01 21:08:32.242926 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:32.242879 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a48915d-f25c-4aae-bd0d-2714992416d1-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-qswqs\" (UID: \"6a48915d-f25c-4aae-bd0d-2714992416d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qswqs" Nov 01 21:08:32.243149 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:32.242931 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a48915d-f25c-4aae-bd0d-2714992416d1-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-qswqs\" (UID: \"6a48915d-f25c-4aae-bd0d-2714992416d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qswqs" Nov 01 21:08:32.243149 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:32.243037 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4m97\" (UniqueName: \"kubernetes.io/projected/6a48915d-f25c-4aae-bd0d-2714992416d1-kube-api-access-h4m97\") pod \"maas-default-gateway-openshift-default-58c6585447-qswqs\" (UID: \"6a48915d-f25c-4aae-bd0d-2714992416d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qswqs" Nov 01 21:08:32.243149 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:32.243120 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a48915d-f25c-4aae-bd0d-2714992416d1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-qswqs\" (UID: \"6a48915d-f25c-4aae-bd0d-2714992416d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qswqs" Nov 01 21:08:32.243349 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:32.243156 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a48915d-f25c-4aae-bd0d-2714992416d1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-qswqs\" (UID: \"6a48915d-f25c-4aae-bd0d-2714992416d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qswqs" Nov 01 21:08:32.243349 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:32.243213 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a48915d-f25c-4aae-bd0d-2714992416d1-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-qswqs\" (UID: \"6a48915d-f25c-4aae-bd0d-2714992416d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qswqs" Nov 01 21:08:32.243349 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:32.243265 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a48915d-f25c-4aae-bd0d-2714992416d1-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-qswqs\" (UID: \"6a48915d-f25c-4aae-bd0d-2714992416d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qswqs" Nov 01 21:08:32.243349 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:32.243305 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a48915d-f25c-4aae-bd0d-2714992416d1-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-qswqs\" (UID: \"6a48915d-f25c-4aae-bd0d-2714992416d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qswqs" Nov 01 21:08:32.243349 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:32.243334 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a48915d-f25c-4aae-bd0d-2714992416d1-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-qswqs\" (UID: \"6a48915d-f25c-4aae-bd0d-2714992416d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qswqs" Nov 01 21:08:32.277266 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:32.277227 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qswqs"] Nov 01 21:08:32.344488 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:32.344450 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a48915d-f25c-4aae-bd0d-2714992416d1-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-qswqs\" (UID: \"6a48915d-f25c-4aae-bd0d-2714992416d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qswqs" Nov 01 21:08:32.344660 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:32.344517 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a48915d-f25c-4aae-bd0d-2714992416d1-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-qswqs\" (UID: \"6a48915d-f25c-4aae-bd0d-2714992416d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qswqs" Nov 01 21:08:32.344660 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:32.344552 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4m97\" (UniqueName: \"kubernetes.io/projected/6a48915d-f25c-4aae-bd0d-2714992416d1-kube-api-access-h4m97\") pod \"maas-default-gateway-openshift-default-58c6585447-qswqs\" (UID: \"6a48915d-f25c-4aae-bd0d-2714992416d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qswqs" Nov 01 21:08:32.344660 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:32.344615 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a48915d-f25c-4aae-bd0d-2714992416d1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-qswqs\" (UID: \"6a48915d-f25c-4aae-bd0d-2714992416d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qswqs" Nov 01 21:08:32.344861 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:32.344658 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a48915d-f25c-4aae-bd0d-2714992416d1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-qswqs\" (UID: \"6a48915d-f25c-4aae-bd0d-2714992416d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qswqs" Nov 01 21:08:32.344861 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:32.344701 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a48915d-f25c-4aae-bd0d-2714992416d1-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-qswqs\" (UID: \"6a48915d-f25c-4aae-bd0d-2714992416d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qswqs" Nov 01 21:08:32.344861 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:32.344744 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a48915d-f25c-4aae-bd0d-2714992416d1-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-qswqs\" (UID: \"6a48915d-f25c-4aae-bd0d-2714992416d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qswqs" Nov 01 21:08:32.344861 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:32.344807 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a48915d-f25c-4aae-bd0d-2714992416d1-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-qswqs\" (UID: \"6a48915d-f25c-4aae-bd0d-2714992416d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qswqs" Nov 01 21:08:32.344861 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:32.344839 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a48915d-f25c-4aae-bd0d-2714992416d1-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-qswqs\" (UID: \"6a48915d-f25c-4aae-bd0d-2714992416d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qswqs" Nov 01 21:08:32.345192 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:32.345024 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a48915d-f25c-4aae-bd0d-2714992416d1-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-qswqs\" (UID: \"6a48915d-f25c-4aae-bd0d-2714992416d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qswqs" Nov 01 21:08:32.345268 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:32.345247 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a48915d-f25c-4aae-bd0d-2714992416d1-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-qswqs\" (UID: \"6a48915d-f25c-4aae-bd0d-2714992416d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qswqs" Nov 01 21:08:32.345332 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:32.345314 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a48915d-f25c-4aae-bd0d-2714992416d1-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-qswqs\" (UID: \"6a48915d-f25c-4aae-bd0d-2714992416d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qswqs" Nov 01 21:08:32.345389 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:32.345366 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a48915d-f25c-4aae-bd0d-2714992416d1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-qswqs\" (UID: \"6a48915d-f25c-4aae-bd0d-2714992416d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qswqs" Nov 01 21:08:32.345428 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:32.345390 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a48915d-f25c-4aae-bd0d-2714992416d1-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-qswqs\" (UID: \"6a48915d-f25c-4aae-bd0d-2714992416d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qswqs" Nov 01 21:08:32.347577 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:32.347554 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a48915d-f25c-4aae-bd0d-2714992416d1-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-qswqs\" (UID: \"6a48915d-f25c-4aae-bd0d-2714992416d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qswqs" Nov 01 21:08:32.347775 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:32.347756 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a48915d-f25c-4aae-bd0d-2714992416d1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-qswqs\" (UID: \"6a48915d-f25c-4aae-bd0d-2714992416d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qswqs" Nov 01 21:08:32.402970 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:32.402935 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4m97\" (UniqueName: \"kubernetes.io/projected/6a48915d-f25c-4aae-bd0d-2714992416d1-kube-api-access-h4m97\") pod \"maas-default-gateway-openshift-default-58c6585447-qswqs\" (UID: \"6a48915d-f25c-4aae-bd0d-2714992416d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qswqs" Nov 01 21:08:32.403403 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:32.403374 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a48915d-f25c-4aae-bd0d-2714992416d1-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-qswqs\" (UID: \"6a48915d-f25c-4aae-bd0d-2714992416d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qswqs" Nov 01 21:08:32.503576 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:32.503547 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qswqs" Nov 01 21:08:32.930802 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:08:32.930772 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a48915d_f25c_4aae_bd0d_2714992416d1.slice/crio-53117ec2c9a2a668e16e46b7392bff298d85e20f205345361bc27b7a4348e0b6 WatchSource:0}: Error finding container 53117ec2c9a2a668e16e46b7392bff298d85e20f205345361bc27b7a4348e0b6: Status 404 returned error can't find the container with id 53117ec2c9a2a668e16e46b7392bff298d85e20f205345361bc27b7a4348e0b6 Nov 01 21:08:32.930943 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:32.930859 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qswqs"] Nov 01 21:08:32.933535 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:32.933503 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:08:32.933622 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:32.933567 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:08:32.933622 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:32.933599 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:08:33.456491 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:33.456455 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qswqs" event={"ID":"6a48915d-f25c-4aae-bd0d-2714992416d1","Type":"ContainerStarted","Data":"139285c996226cba5cc0a951c9b400fa866c0978fb3a53da6718469884696333"} Nov 01 21:08:33.456909 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:33.456513 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qswqs" event={"ID":"6a48915d-f25c-4aae-bd0d-2714992416d1","Type":"ContainerStarted","Data":"53117ec2c9a2a668e16e46b7392bff298d85e20f205345361bc27b7a4348e0b6"} Nov 01 21:08:33.505141 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:33.505100 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qswqs" Nov 01 21:08:33.581622 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:33.581566 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qswqs" podStartSLOduration=1.5815519569999998 podStartE2EDuration="1.581551957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:08:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:08:33.578807533 +0000 UTC m=+2264.356447667" watchObservedRunningTime="2026-11-01 21:08:33.581551957 +0000 UTC m=+2264.359192155" Nov 01 21:08:33.977580 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:33.977539 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:34.528203 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:34.528161 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qswqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" start-of-body= Nov 01 21:08:34.528646 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:34.528233 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qswqs" podUID="6a48915d-f25c-4aae-bd0d-2714992416d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" Nov 01 21:08:35.528023 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:35.524383 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qswqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" start-of-body= Nov 01 21:08:35.528023 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:35.524446 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qswqs" podUID="6a48915d-f25c-4aae-bd0d-2714992416d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" Nov 01 21:08:36.528058 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:36.509440 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qswqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" start-of-body= Nov 01 21:08:36.528058 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:36.509510 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qswqs" podUID="6a48915d-f25c-4aae-bd0d-2714992416d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" Nov 01 21:08:37.526024 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:37.525483 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qswqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" start-of-body= Nov 01 21:08:37.526024 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:37.525556 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qswqs" podUID="6a48915d-f25c-4aae-bd0d-2714992416d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" Nov 01 21:08:38.522020 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:38.518393 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qswqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" start-of-body= Nov 01 21:08:38.522020 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:38.518470 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qswqs" podUID="6a48915d-f25c-4aae-bd0d-2714992416d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" Nov 01 21:08:38.867703 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:38.867602 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:39.028347 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:39.028305 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:39.504406 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:39.504362 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qswqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" start-of-body= Nov 01 21:08:39.504599 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:39.504427 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qswqs" podUID="6a48915d-f25c-4aae-bd0d-2714992416d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" Nov 01 21:08:40.504441 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:40.504405 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qswqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" start-of-body= Nov 01 21:08:40.504851 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:40.504461 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qswqs" podUID="6a48915d-f25c-4aae-bd0d-2714992416d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" Nov 01 21:08:41.510412 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:41.510382 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qswqs" Nov 01 21:08:41.510915 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:41.510773 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qswqs" Nov 01 21:08:41.511677 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:41.511655 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qswqs" Nov 01 21:08:41.592144 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:41.592113 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:42.017284 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:42.017251 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:42.448392 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:42.448313 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-3ab520-7b7f4ff56c-vszhw" Nov 01 21:08:42.448392 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:42.448376 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3ab520-848dc5bd6c-k9zhf" Nov 01 21:08:43.136236 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:43.136203 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:43.342950 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:43.342911 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:44.056929 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:44.056892 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:45.769787 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:08:45.769745 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:08:45.871983 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:45.871944 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:47.964867 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:47.964817 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:48.166963 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:48.166914 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:48.392220 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:48.392182 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:49.137610 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:49.137572 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:52.367209 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:52.367181 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:52.545529 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:52.545481 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:52.756246 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:52.756206 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7091fd-6d79589dd7-mqf74"] Nov 01 21:08:52.761376 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:52.761348 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7091fd-6d79589dd7-mqf74" Nov 01 21:08:52.807158 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:52.807122 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e127ff65-ebee-4f72-952b-f4464d7592f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-7091fd-6d79589dd7-mqf74\" (UID: \"e127ff65-ebee-4f72-952b-f4464d7592f9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7091fd-6d79589dd7-mqf74" Nov 01 21:08:52.807340 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:52.807187 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbmpq\" (UniqueName: \"kubernetes.io/projected/e127ff65-ebee-4f72-952b-f4464d7592f9-kube-api-access-lbmpq\") pod \"payload-pre-processing-e2e-inf-b-7091fd-6d79589dd7-mqf74\" (UID: \"e127ff65-ebee-4f72-952b-f4464d7592f9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7091fd-6d79589dd7-mqf74" Nov 01 21:08:52.815384 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:52.815348 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7091fd-6d79589dd7-mqf74"] Nov 01 21:08:52.986514 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:52.984870 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-7091fd\"" Nov 01 21:08:52.986514 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:52.985087 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-7091fd-dockercfg-7lxt7\"" Nov 01 21:08:52.992398 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:52.992277 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e127ff65-ebee-4f72-952b-f4464d7592f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-7091fd-6d79589dd7-mqf74\" (UID: \"e127ff65-ebee-4f72-952b-f4464d7592f9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7091fd-6d79589dd7-mqf74" Nov 01 21:08:52.992892 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:52.992863 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbmpq\" (UniqueName: \"kubernetes.io/projected/e127ff65-ebee-4f72-952b-f4464d7592f9-kube-api-access-lbmpq\") pod \"payload-pre-processing-e2e-inf-b-7091fd-6d79589dd7-mqf74\" (UID: \"e127ff65-ebee-4f72-952b-f4464d7592f9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7091fd-6d79589dd7-mqf74" Nov 01 21:08:52.993795 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:52.993744 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e127ff65-ebee-4f72-952b-f4464d7592f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-7091fd-6d79589dd7-mqf74\" (UID: \"e127ff65-ebee-4f72-952b-f4464d7592f9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7091fd-6d79589dd7-mqf74" Nov 01 21:08:52.998078 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:52.997190 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7091fd-98896dbf7-6r5f6"] Nov 01 21:08:53.003163 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:53.003135 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7091fd-98896dbf7-6r5f6"] Nov 01 21:08:53.003278 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:53.003250 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7091fd-98896dbf7-6r5f6" Nov 01 21:08:53.050522 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:53.050492 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbmpq\" (UniqueName: \"kubernetes.io/projected/e127ff65-ebee-4f72-952b-f4464d7592f9-kube-api-access-lbmpq\") pod \"payload-pre-processing-e2e-inf-b-7091fd-6d79589dd7-mqf74\" (UID: \"e127ff65-ebee-4f72-952b-f4464d7592f9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7091fd-6d79589dd7-mqf74" Nov 01 21:08:53.079539 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:53.079512 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7091fd-6d79589dd7-mqf74" Nov 01 21:08:53.094979 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:53.094948 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dphzd\" (UniqueName: \"kubernetes.io/projected/ab517a39-c7e7-4567-b294-484c21d3d799-kube-api-access-dphzd\") pod \"payload-processing-e2e-inf-b-7091fd-98896dbf7-6r5f6\" (UID: \"ab517a39-c7e7-4567-b294-484c21d3d799\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7091fd-98896dbf7-6r5f6" Nov 01 21:08:53.095199 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:53.095179 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab517a39-c7e7-4567-b294-484c21d3d799-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-7091fd-98896dbf7-6r5f6\" (UID: \"ab517a39-c7e7-4567-b294-484c21d3d799\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7091fd-98896dbf7-6r5f6" Nov 01 21:08:53.196861 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:53.196827 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab517a39-c7e7-4567-b294-484c21d3d799-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-7091fd-98896dbf7-6r5f6\" (UID: \"ab517a39-c7e7-4567-b294-484c21d3d799\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7091fd-98896dbf7-6r5f6" Nov 01 21:08:53.197021 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:53.196934 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dphzd\" (UniqueName: \"kubernetes.io/projected/ab517a39-c7e7-4567-b294-484c21d3d799-kube-api-access-dphzd\") pod \"payload-processing-e2e-inf-b-7091fd-98896dbf7-6r5f6\" (UID: \"ab517a39-c7e7-4567-b294-484c21d3d799\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7091fd-98896dbf7-6r5f6" Nov 01 21:08:53.197638 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:53.197616 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab517a39-c7e7-4567-b294-484c21d3d799-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-7091fd-98896dbf7-6r5f6\" (UID: \"ab517a39-c7e7-4567-b294-484c21d3d799\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7091fd-98896dbf7-6r5f6" Nov 01 21:08:53.256449 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:53.256415 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dphzd\" (UniqueName: \"kubernetes.io/projected/ab517a39-c7e7-4567-b294-484c21d3d799-kube-api-access-dphzd\") pod \"payload-processing-e2e-inf-b-7091fd-98896dbf7-6r5f6\" (UID: \"ab517a39-c7e7-4567-b294-484c21d3d799\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7091fd-98896dbf7-6r5f6" Nov 01 21:08:53.328074 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:53.327377 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7091fd-98896dbf7-6r5f6" Nov 01 21:08:53.478703 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:53.478670 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7091fd-6d79589dd7-mqf74"] Nov 01 21:08:53.735090 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:08:53.734967 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab517a39_c7e7_4567_b294_484c21d3d799.slice/crio-ce2dcfc03f1766e06d88ec09bff80eb100c18ffe8bd57a095bbae0fb2eb42703 WatchSource:0}: Error finding container ce2dcfc03f1766e06d88ec09bff80eb100c18ffe8bd57a095bbae0fb2eb42703: Status 404 returned error can't find the container with id ce2dcfc03f1766e06d88ec09bff80eb100c18ffe8bd57a095bbae0fb2eb42703 Nov 01 21:08:53.735373 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:53.735347 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7091fd-98896dbf7-6r5f6"] Nov 01 21:08:53.742008 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:53.741969 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7091fd-6d79589dd7-mqf74" event={"ID":"e127ff65-ebee-4f72-952b-f4464d7592f9","Type":"ContainerStarted","Data":"2c65b50f57384b1b87b8389593422b84d76a17447bb9470b2bdafb57539d733f"} Nov 01 21:08:53.742117 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:53.742055 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7091fd-6d79589dd7-mqf74" event={"ID":"e127ff65-ebee-4f72-952b-f4464d7592f9","Type":"ContainerStarted","Data":"248a8cd703cc8081e2370291eb94e4a90439a2764778e1714dcc9e8c066b5e26"} Nov 01 21:08:53.742117 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:53.742087 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7091fd-6d79589dd7-mqf74" Nov 01 21:08:53.858465 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:53.858411 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7091fd-6d79589dd7-mqf74" podStartSLOduration=1.858396232 podStartE2EDuration="1.858396232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:08:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:08:53.856042073 +0000 UTC m=+2284.633682209" watchObservedRunningTime="2026-11-01 21:08:53.858396232 +0000 UTC m=+2284.636036366" Nov 01 21:08:54.424694 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:54.424520 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:54.752523 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:54.752481 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7091fd-98896dbf7-6r5f6" event={"ID":"ab517a39-c7e7-4567-b294-484c21d3d799","Type":"ContainerStarted","Data":"deaa910fc4ac76c40cf15cd323d28e09e38e14f8df02fb6640b538bf332b1750"} Nov 01 21:08:54.752952 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:54.752547 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7091fd-98896dbf7-6r5f6" event={"ID":"ab517a39-c7e7-4567-b294-484c21d3d799","Type":"ContainerStarted","Data":"ce2dcfc03f1766e06d88ec09bff80eb100c18ffe8bd57a095bbae0fb2eb42703"} Nov 01 21:08:54.752952 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:54.752592 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-7091fd-98896dbf7-6r5f6" Nov 01 21:08:54.886960 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:54.886907 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-7091fd-98896dbf7-6r5f6" podStartSLOduration=2.8868921260000002 podStartE2EDuration="2.886892126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:08:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:08:54.88423526 +0000 UTC m=+2285.661875396" watchObservedRunningTime="2026-11-01 21:08:54.886892126 +0000 UTC m=+2285.664532319" Nov 01 21:08:59.382561 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:08:59.382521 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:09:00.680185 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:00.680142 4484 patch_prober.go:29] interesting pod/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 21:09:00.680616 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:00.680208 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26" podUID="69124be6-394b-45bd-9d52-6de87dfc5648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 21:09:02.817474 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:02.817435 4484 generic.go:356] "Generic (PLEG): container finished" podID="69124be6-394b-45bd-9d52-6de87dfc5648" containerID="7d182619c2a5d6c4a794f450da9d548e759cabb34a4972bf9152cc237cc01aab" exitCode=0 Nov 01 21:09:02.817919 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:02.817487 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26" event={"ID":"69124be6-394b-45bd-9d52-6de87dfc5648","Type":"ContainerDied","Data":"7d182619c2a5d6c4a794f450da9d548e759cabb34a4972bf9152cc237cc01aab"} Nov 01 21:09:02.817964 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:02.817916 4484 scope.go:122] "RemoveContainer" containerID="7d182619c2a5d6c4a794f450da9d548e759cabb34a4972bf9152cc237cc01aab" Nov 01 21:09:02.819656 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:02.819623 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:09:02.819730 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:02.819696 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:09:02.819767 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:02.819728 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:09:03.201331 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:03.201250 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd0eb5ca-7hmxn"] Nov 01 21:09:03.204678 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:03.204660 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd0eb5ca-7hmxn" Nov 01 21:09:03.262882 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:03.262843 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:09:03.263355 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:03.262975 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:09:03.263355 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:03.263054 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:09:03.263789 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:03.263401 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:09:03.267167 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:03.267139 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd0eb5ca-7hmxn"] Nov 01 21:09:03.394004 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:03.393959 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8347318b-2d4c-4db1-af2d-881a04c8a5c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd0eb5ca-7hmxn\" (UID: \"8347318b-2d4c-4db1-af2d-881a04c8a5c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd0eb5ca-7hmxn" Nov 01 21:09:03.394240 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:03.394173 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8347318b-2d4c-4db1-af2d-881a04c8a5c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd0eb5ca-7hmxn\" (UID: \"8347318b-2d4c-4db1-af2d-881a04c8a5c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd0eb5ca-7hmxn" Nov 01 21:09:03.394240 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:03.394216 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75rw5\" (UniqueName: \"kubernetes.io/projected/8347318b-2d4c-4db1-af2d-881a04c8a5c0-kube-api-access-75rw5\") pod \"maas-api-revoke-sub-dd0eb5ca-7hmxn\" (UID: \"8347318b-2d4c-4db1-af2d-881a04c8a5c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd0eb5ca-7hmxn" Nov 01 21:09:03.497836 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:03.497795 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8347318b-2d4c-4db1-af2d-881a04c8a5c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd0eb5ca-7hmxn\" (UID: \"8347318b-2d4c-4db1-af2d-881a04c8a5c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd0eb5ca-7hmxn" Nov 01 21:09:03.498174 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:03.498151 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75rw5\" (UniqueName: \"kubernetes.io/projected/8347318b-2d4c-4db1-af2d-881a04c8a5c0-kube-api-access-75rw5\") pod \"maas-api-revoke-sub-dd0eb5ca-7hmxn\" (UID: \"8347318b-2d4c-4db1-af2d-881a04c8a5c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd0eb5ca-7hmxn" Nov 01 21:09:03.499098 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:03.499072 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8347318b-2d4c-4db1-af2d-881a04c8a5c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd0eb5ca-7hmxn\" (UID: \"8347318b-2d4c-4db1-af2d-881a04c8a5c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd0eb5ca-7hmxn" Nov 01 21:09:03.499754 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:03.499732 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8347318b-2d4c-4db1-af2d-881a04c8a5c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd0eb5ca-7hmxn\" (UID: \"8347318b-2d4c-4db1-af2d-881a04c8a5c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd0eb5ca-7hmxn" Nov 01 21:09:03.521511 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:03.521476 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8347318b-2d4c-4db1-af2d-881a04c8a5c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd0eb5ca-7hmxn\" (UID: \"8347318b-2d4c-4db1-af2d-881a04c8a5c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd0eb5ca-7hmxn" Nov 01 21:09:03.556769 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:03.556731 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75rw5\" (UniqueName: \"kubernetes.io/projected/8347318b-2d4c-4db1-af2d-881a04c8a5c0-kube-api-access-75rw5\") pod \"maas-api-revoke-sub-dd0eb5ca-7hmxn\" (UID: \"8347318b-2d4c-4db1-af2d-881a04c8a5c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd0eb5ca-7hmxn" Nov 01 21:09:03.828371 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:03.828265 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd0eb5ca-7hmxn" Nov 01 21:09:03.837553 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:03.837520 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26" event={"ID":"69124be6-394b-45bd-9d52-6de87dfc5648","Type":"ContainerStarted","Data":"5e2c52f3040aa9adf29fbbc3cb9ea39368be840f802d18240cccf290f3d38c22"} Nov 01 21:09:04.094685 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:04.094597 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:09:04.228067 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:04.226038 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd0eb5ca-7hmxn"] Nov 01 21:09:04.234037 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:09:04.231327 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8347318b_2d4c_4db1_af2d_881a04c8a5c0.slice/crio-01e22bae7e78a450d867092d3201bf7ec06d9ce89c0ffaa78b37a47aedff63a7 WatchSource:0}: Error finding container 01e22bae7e78a450d867092d3201bf7ec06d9ce89c0ffaa78b37a47aedff63a7: Status 404 returned error can't find the container with id 01e22bae7e78a450d867092d3201bf7ec06d9ce89c0ffaa78b37a47aedff63a7 Nov 01 21:09:04.755910 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:04.755884 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7091fd-6d79589dd7-mqf74" Nov 01 21:09:04.767588 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:04.767559 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26" Nov 01 21:09:04.767858 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:04.767837 4484 patch_prober.go:29] interesting pod/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Nov 01 21:09:04.767903 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:04.767887 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26" podUID="69124be6-394b-45bd-9d52-6de87dfc5648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Nov 01 21:09:04.847541 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:04.847501 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd0eb5ca-7hmxn" event={"ID":"8347318b-2d4c-4db1-af2d-881a04c8a5c0","Type":"ContainerStarted","Data":"da84c929870134471bed08ccb9e480182d572a25708a2e2493781d7cf52e844e"} Nov 01 21:09:04.848068 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:04.847562 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd0eb5ca-7hmxn" event={"ID":"8347318b-2d4c-4db1-af2d-881a04c8a5c0","Type":"ContainerStarted","Data":"01e22bae7e78a450d867092d3201bf7ec06d9ce89c0ffaa78b37a47aedff63a7"} Nov 01 21:09:04.999406 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:04.999349 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd0eb5ca-7hmxn" podStartSLOduration=1.9993324989999999 podStartE2EDuration="1.999332499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:09:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:09:04.996432265 +0000 UTC m=+2295.774072398" watchObservedRunningTime="2026-11-01 21:09:04.999332499 +0000 UTC m=+2295.776972626" Nov 01 21:09:05.763375 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:05.763346 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-7091fd-98896dbf7-6r5f6" Nov 01 21:09:05.768189 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:05.768162 4484 patch_prober.go:29] interesting pod/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Nov 01 21:09:05.768342 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:05.768211 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26" podUID="69124be6-394b-45bd-9d52-6de87dfc5648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Nov 01 21:09:05.867599 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:05.867567 4484 generic.go:356] "Generic (PLEG): container finished" podID="8347318b-2d4c-4db1-af2d-881a04c8a5c0" containerID="da84c929870134471bed08ccb9e480182d572a25708a2e2493781d7cf52e844e" exitCode=0 Nov 01 21:09:05.867981 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:05.867653 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd0eb5ca-7hmxn" event={"ID":"8347318b-2d4c-4db1-af2d-881a04c8a5c0","Type":"ContainerDied","Data":"da84c929870134471bed08ccb9e480182d572a25708a2e2493781d7cf52e844e"} Nov 01 21:09:06.768139 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:06.768106 4484 patch_prober.go:29] interesting pod/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Nov 01 21:09:06.768349 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:06.768160 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26" podUID="69124be6-394b-45bd-9d52-6de87dfc5648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Nov 01 21:09:07.087290 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:07.087255 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd0eb5ca-7hmxn" Nov 01 21:09:07.184174 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:07.184142 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8347318b-2d4c-4db1-af2d-881a04c8a5c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8347318b-2d4c-4db1-af2d-881a04c8a5c0-maas-api-openshift-ca\") pod \"8347318b-2d4c-4db1-af2d-881a04c8a5c0\" (UID: \"8347318b-2d4c-4db1-af2d-881a04c8a5c0\") " Nov 01 21:09:07.184365 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:07.184222 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8347318b-2d4c-4db1-af2d-881a04c8a5c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8347318b-2d4c-4db1-af2d-881a04c8a5c0-maas-api-xks-ca\") pod \"8347318b-2d4c-4db1-af2d-881a04c8a5c0\" (UID: \"8347318b-2d4c-4db1-af2d-881a04c8a5c0\") " Nov 01 21:09:07.184448 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:07.184435 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8347318b-2d4c-4db1-af2d-881a04c8a5c0-kube-api-access-75rw5\" (UniqueName: \"kubernetes.io/projected/8347318b-2d4c-4db1-af2d-881a04c8a5c0-kube-api-access-75rw5\") pod \"8347318b-2d4c-4db1-af2d-881a04c8a5c0\" (UID: \"8347318b-2d4c-4db1-af2d-881a04c8a5c0\") " Nov 01 21:09:07.184619 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:07.184589 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8347318b-2d4c-4db1-af2d-881a04c8a5c0-maas-api-openshift-ca" pod "8347318b-2d4c-4db1-af2d-881a04c8a5c0" (UID: "8347318b-2d4c-4db1-af2d-881a04c8a5c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:09:07.184828 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:07.184808 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8347318b-2d4c-4db1-af2d-881a04c8a5c0-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:09:07.186454 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:07.186426 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8347318b-2d4c-4db1-af2d-881a04c8a5c0-maas-api-xks-ca" pod "8347318b-2d4c-4db1-af2d-881a04c8a5c0" (UID: "8347318b-2d4c-4db1-af2d-881a04c8a5c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:09:07.186556 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:07.186462 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8347318b-2d4c-4db1-af2d-881a04c8a5c0-kube-api-access-75rw5" pod "8347318b-2d4c-4db1-af2d-881a04c8a5c0" (UID: "8347318b-2d4c-4db1-af2d-881a04c8a5c0"). InnerVolumeSpecName "kube-api-access-75rw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:09:07.286470 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:07.286433 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75rw5\" (UniqueName: \"kubernetes.io/projected/8347318b-2d4c-4db1-af2d-881a04c8a5c0-kube-api-access-75rw5\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:09:07.286470 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:07.286468 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8347318b-2d4c-4db1-af2d-881a04c8a5c0-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:09:07.767368 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:07.767328 4484 patch_prober.go:29] interesting pod/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Nov 01 21:09:07.767531 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:07.767387 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26" podUID="69124be6-394b-45bd-9d52-6de87dfc5648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Nov 01 21:09:07.881417 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:07.881379 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:09:07.885545 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:07.885520 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd0eb5ca-7hmxn" Nov 01 21:09:07.885701 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:07.885555 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd0eb5ca-7hmxn" event={"ID":"8347318b-2d4c-4db1-af2d-881a04c8a5c0","Type":"ContainerDied","Data":"01e22bae7e78a450d867092d3201bf7ec06d9ce89c0ffaa78b37a47aedff63a7"} Nov 01 21:09:07.885701 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:07.885603 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01e22bae7e78a450d867092d3201bf7ec06d9ce89c0ffaa78b37a47aedff63a7" Nov 01 21:09:08.549162 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:08.549126 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:09:08.768168 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:08.768131 4484 patch_prober.go:29] interesting pod/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Nov 01 21:09:08.768375 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:08.768190 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26" podUID="69124be6-394b-45bd-9d52-6de87dfc5648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Nov 01 21:09:08.995264 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:08.995229 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk"] Nov 01 21:09:08.996269 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:08.996241 4484 cpu_manager.go:414] "containerMap: removing container" podUID="8347318b-2d4c-4db1-af2d-881a04c8a5c0" containerName="revoke-keys" Nov 01 21:09:08.996269 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:08.996260 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8347318b-2d4c-4db1-af2d-881a04c8a5c0" containerName="revoke-keys" Nov 01 21:09:08.996461 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:08.996419 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="8347318b-2d4c-4db1-af2d-881a04c8a5c0" containerName="revoke-keys" Nov 01 21:09:08.998901 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:08.998885 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk" Nov 01 21:09:09.055048 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:09.052045 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-670df9-openshift-default-dockercfg-kwsh8\"" Nov 01 21:09:09.060505 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:09.060475 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:09:09.101222 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:09.101187 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk"] Nov 01 21:09:09.254538 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:09.223069 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-istio-envoy\") pod \"e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk\" (UID: \"8a5b2e73-9f47-47aa-9ed7-c75d17b1f658\") " pod="openshift-ingress/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk" Nov 01 21:09:09.254877 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:09.254838 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-istiod-ca-cert\") pod \"e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk\" (UID: \"8a5b2e73-9f47-47aa-9ed7-c75d17b1f658\") " pod="openshift-ingress/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk" Nov 01 21:09:09.262170 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:09.262132 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-workload-certs\") pod \"e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk\" (UID: \"8a5b2e73-9f47-47aa-9ed7-c75d17b1f658\") " pod="openshift-ingress/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk" Nov 01 21:09:09.262431 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:09.262410 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-istio-data\") pod \"e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk\" (UID: \"8a5b2e73-9f47-47aa-9ed7-c75d17b1f658\") " pod="openshift-ingress/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk" Nov 01 21:09:09.271252 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:09.271217 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-istio-token\") pod \"e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk\" (UID: \"8a5b2e73-9f47-47aa-9ed7-c75d17b1f658\") " pod="openshift-ingress/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk" Nov 01 21:09:09.279190 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:09.279146 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-workload-socket\") pod \"e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk\" (UID: \"8a5b2e73-9f47-47aa-9ed7-c75d17b1f658\") " pod="openshift-ingress/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk" Nov 01 21:09:09.279456 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:09.279434 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-credential-socket\") pod \"e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk\" (UID: \"8a5b2e73-9f47-47aa-9ed7-c75d17b1f658\") " pod="openshift-ingress/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk" Nov 01 21:09:09.279582 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:09.279565 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhcp5\" (UniqueName: \"kubernetes.io/projected/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-kube-api-access-rhcp5\") pod \"e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk\" (UID: \"8a5b2e73-9f47-47aa-9ed7-c75d17b1f658\") " pod="openshift-ingress/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk" Nov 01 21:09:09.294153 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:09.244858 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:09:09.294409 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:09.294382 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-istio-podinfo\") pod \"e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk\" (UID: \"8a5b2e73-9f47-47aa-9ed7-c75d17b1f658\") " pod="openshift-ingress/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk" Nov 01 21:09:09.396966 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:09.396936 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-istio-token\") pod \"e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk\" (UID: \"8a5b2e73-9f47-47aa-9ed7-c75d17b1f658\") " pod="openshift-ingress/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk" Nov 01 21:09:09.397183 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:09.397163 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-workload-socket\") pod \"e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk\" (UID: \"8a5b2e73-9f47-47aa-9ed7-c75d17b1f658\") " pod="openshift-ingress/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk" Nov 01 21:09:09.397261 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:09.397219 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-credential-socket\") pod \"e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk\" (UID: \"8a5b2e73-9f47-47aa-9ed7-c75d17b1f658\") " pod="openshift-ingress/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk" Nov 01 21:09:09.397319 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:09.397257 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhcp5\" (UniqueName: \"kubernetes.io/projected/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-kube-api-access-rhcp5\") pod \"e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk\" (UID: \"8a5b2e73-9f47-47aa-9ed7-c75d17b1f658\") " pod="openshift-ingress/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk" Nov 01 21:09:09.397531 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:09.397506 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-istio-podinfo\") pod \"e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk\" (UID: \"8a5b2e73-9f47-47aa-9ed7-c75d17b1f658\") " pod="openshift-ingress/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk" Nov 01 21:09:09.397625 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:09.397594 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-istio-envoy\") pod \"e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk\" (UID: \"8a5b2e73-9f47-47aa-9ed7-c75d17b1f658\") " pod="openshift-ingress/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk" Nov 01 21:09:09.397687 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:09.397664 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-istiod-ca-cert\") pod \"e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk\" (UID: \"8a5b2e73-9f47-47aa-9ed7-c75d17b1f658\") " pod="openshift-ingress/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk" Nov 01 21:09:09.397746 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:09.397725 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-workload-certs\") pod \"e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk\" (UID: \"8a5b2e73-9f47-47aa-9ed7-c75d17b1f658\") " pod="openshift-ingress/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk" Nov 01 21:09:09.397815 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:09.397798 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-istio-data\") pod \"e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk\" (UID: \"8a5b2e73-9f47-47aa-9ed7-c75d17b1f658\") " pod="openshift-ingress/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk" Nov 01 21:09:09.399380 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:09.399136 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-istio-data\") pod \"e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk\" (UID: \"8a5b2e73-9f47-47aa-9ed7-c75d17b1f658\") " pod="openshift-ingress/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk" Nov 01 21:09:09.399380 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:09.399256 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-credential-socket\") pod \"e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk\" (UID: \"8a5b2e73-9f47-47aa-9ed7-c75d17b1f658\") " pod="openshift-ingress/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk" Nov 01 21:09:09.399684 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:09.399659 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-workload-certs\") pod \"e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk\" (UID: \"8a5b2e73-9f47-47aa-9ed7-c75d17b1f658\") " pod="openshift-ingress/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk" Nov 01 21:09:09.400056 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:09.400030 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-istiod-ca-cert\") pod \"e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk\" (UID: \"8a5b2e73-9f47-47aa-9ed7-c75d17b1f658\") " pod="openshift-ingress/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk" Nov 01 21:09:09.404218 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:09.404193 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-workload-socket\") pod \"e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk\" (UID: \"8a5b2e73-9f47-47aa-9ed7-c75d17b1f658\") " pod="openshift-ingress/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk" Nov 01 21:09:09.415668 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:09.415008 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-istio-podinfo\") pod \"e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk\" (UID: \"8a5b2e73-9f47-47aa-9ed7-c75d17b1f658\") " pod="openshift-ingress/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk" Nov 01 21:09:09.417968 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:09.417941 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-istio-envoy\") pod \"e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk\" (UID: \"8a5b2e73-9f47-47aa-9ed7-c75d17b1f658\") " pod="openshift-ingress/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk" Nov 01 21:09:09.474506 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:09.474452 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:09:09.476856 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:09.476803 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-istio-token\") pod \"e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk\" (UID: \"8a5b2e73-9f47-47aa-9ed7-c75d17b1f658\") " pod="openshift-ingress/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk" Nov 01 21:09:09.486709 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:09.486674 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhcp5\" (UniqueName: \"kubernetes.io/projected/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-kube-api-access-rhcp5\") pod \"e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk\" (UID: \"8a5b2e73-9f47-47aa-9ed7-c75d17b1f658\") " pod="openshift-ingress/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk" Nov 01 21:09:09.640490 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:09.640393 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk" Nov 01 21:09:09.773488 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:09.773452 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26" Nov 01 21:09:09.773958 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:09.773933 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26" Nov 01 21:09:09.775130 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:09.775107 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26" Nov 01 21:09:10.073033 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:10.072985 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk"] Nov 01 21:09:10.074125 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:09:10.074090 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a5b2e73_9f47_47aa_9ed7_c75d17b1f658.slice/crio-eaec53cd0ebea9ebd6e54848169ed6803a78420901f55e631b0514fe70a9416a WatchSource:0}: Error finding container eaec53cd0ebea9ebd6e54848169ed6803a78420901f55e631b0514fe70a9416a: Status 404 returned error can't find the container with id eaec53cd0ebea9ebd6e54848169ed6803a78420901f55e631b0514fe70a9416a Nov 01 21:09:10.077350 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:10.077318 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:09:10.077432 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:10.077386 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:09:10.077432 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:10.077422 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:09:10.944758 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:10.944717 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk" event={"ID":"8a5b2e73-9f47-47aa-9ed7-c75d17b1f658","Type":"ContainerStarted","Data":"4696272076f0d71ea68d0e91f545e2f24273c5a522684fc28be907d79aab81f7"} Nov 01 21:09:10.945267 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:10.944783 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk" event={"ID":"8a5b2e73-9f47-47aa-9ed7-c75d17b1f658","Type":"ContainerStarted","Data":"eaec53cd0ebea9ebd6e54848169ed6803a78420901f55e631b0514fe70a9416a"} Nov 01 21:09:11.067705 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:11.067648 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk" podStartSLOduration=3.067632875 podStartE2EDuration="3.067632875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:09:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:09:11.063836449 +0000 UTC m=+2301.841476584" watchObservedRunningTime="2026-11-01 21:09:11.067632875 +0000 UTC m=+2301.845273003" Nov 01 21:09:11.641346 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:11.641306 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk" Nov 01 21:09:11.647779 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:11.647748 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk" Nov 01 21:09:11.954520 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:11.954445 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk" Nov 01 21:09:11.955535 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:11.955516 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk" Nov 01 21:09:14.018665 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:14.018606 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:09:14.173531 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:14.173498 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:09:14.495698 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:14.495663 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:09:14.674261 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:14.674225 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:09:14.854508 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:14.854401 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:09:15.768225 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:09:15.768189 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:09:15.879758 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:15.879717 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-dd038b-px89s"] Nov 01 21:09:15.884855 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:15.884832 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-dd038b-px89s" Nov 01 21:09:15.938233 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:15.938202 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:09:15.939716 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:15.939688 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-dd038b-px89s"] Nov 01 21:09:15.985775 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:15.985729 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:09:15.985775 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:15.985753 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:09:15.986049 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:15.985848 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:09:16.006827 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:16.006787 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch6gb\" (UniqueName: \"kubernetes.io/projected/11dcc6c3-7e00-40d3-8df5-f9b93d2dbc60-kube-api-access-ch6gb\") pod \"maas-api-revoke-keys-e2e-worker-w6-dd038b-px89s\" (UID: \"11dcc6c3-7e00-40d3-8df5-f9b93d2dbc60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-dd038b-px89s" Nov 01 21:09:16.007042 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:16.006982 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11dcc6c3-7e00-40d3-8df5-f9b93d2dbc60-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-dd038b-px89s\" (UID: \"11dcc6c3-7e00-40d3-8df5-f9b93d2dbc60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-dd038b-px89s" Nov 01 21:09:16.007169 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:16.007144 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11dcc6c3-7e00-40d3-8df5-f9b93d2dbc60-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-dd038b-px89s\" (UID: \"11dcc6c3-7e00-40d3-8df5-f9b93d2dbc60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-dd038b-px89s" Nov 01 21:09:16.108861 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:16.108781 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11dcc6c3-7e00-40d3-8df5-f9b93d2dbc60-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-dd038b-px89s\" (UID: \"11dcc6c3-7e00-40d3-8df5-f9b93d2dbc60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-dd038b-px89s" Nov 01 21:09:16.108861 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:16.108840 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11dcc6c3-7e00-40d3-8df5-f9b93d2dbc60-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-dd038b-px89s\" (UID: \"11dcc6c3-7e00-40d3-8df5-f9b93d2dbc60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-dd038b-px89s" Nov 01 21:09:16.109079 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:16.108921 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch6gb\" (UniqueName: \"kubernetes.io/projected/11dcc6c3-7e00-40d3-8df5-f9b93d2dbc60-kube-api-access-ch6gb\") pod \"maas-api-revoke-keys-e2e-worker-w6-dd038b-px89s\" (UID: \"11dcc6c3-7e00-40d3-8df5-f9b93d2dbc60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-dd038b-px89s" Nov 01 21:09:16.109629 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:16.109598 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11dcc6c3-7e00-40d3-8df5-f9b93d2dbc60-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-dd038b-px89s\" (UID: \"11dcc6c3-7e00-40d3-8df5-f9b93d2dbc60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-dd038b-px89s" Nov 01 21:09:16.111143 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:16.111121 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11dcc6c3-7e00-40d3-8df5-f9b93d2dbc60-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-dd038b-px89s\" (UID: \"11dcc6c3-7e00-40d3-8df5-f9b93d2dbc60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-dd038b-px89s" Nov 01 21:09:16.167511 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:16.167475 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch6gb\" (UniqueName: \"kubernetes.io/projected/11dcc6c3-7e00-40d3-8df5-f9b93d2dbc60-kube-api-access-ch6gb\") pod \"maas-api-revoke-keys-e2e-worker-w6-dd038b-px89s\" (UID: \"11dcc6c3-7e00-40d3-8df5-f9b93d2dbc60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-dd038b-px89s" Nov 01 21:09:16.199713 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:16.199665 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-dd038b-px89s" Nov 01 21:09:16.580376 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:09:16.580335 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11dcc6c3_7e00_40d3_8df5_f9b93d2dbc60.slice/crio-1c6327145f686bca762311fcf208d2a63405950d59e3e7d17061468dfe062710 WatchSource:0}: Error finding container 1c6327145f686bca762311fcf208d2a63405950d59e3e7d17061468dfe062710: Status 404 returned error can't find the container with id 1c6327145f686bca762311fcf208d2a63405950d59e3e7d17061468dfe062710 Nov 01 21:09:16.581310 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:16.581280 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-dd038b-px89s"] Nov 01 21:09:16.997340 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:16.997298 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-dd038b-px89s" event={"ID":"11dcc6c3-7e00-40d3-8df5-f9b93d2dbc60","Type":"ContainerStarted","Data":"91c1faf51b1598e1b9d909446f90217ec650012b77aee07fc176cf543e3b5e4d"} Nov 01 21:09:16.997759 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:16.997357 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-dd038b-px89s" event={"ID":"11dcc6c3-7e00-40d3-8df5-f9b93d2dbc60","Type":"ContainerStarted","Data":"1c6327145f686bca762311fcf208d2a63405950d59e3e7d17061468dfe062710"} Nov 01 21:09:17.139813 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:17.139760 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-dd038b-px89s" podStartSLOduration=2.139745385 podStartE2EDuration="2.139745385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:09:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:09:17.138300874 +0000 UTC m=+2307.915941009" watchObservedRunningTime="2026-11-01 21:09:17.139745385 +0000 UTC m=+2307.917385520" Nov 01 21:09:18.006212 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:18.006169 4484 generic.go:356] "Generic (PLEG): container finished" podID="11dcc6c3-7e00-40d3-8df5-f9b93d2dbc60" containerID="91c1faf51b1598e1b9d909446f90217ec650012b77aee07fc176cf543e3b5e4d" exitCode=0 Nov 01 21:09:18.006606 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:18.006248 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-dd038b-px89s" event={"ID":"11dcc6c3-7e00-40d3-8df5-f9b93d2dbc60","Type":"ContainerDied","Data":"91c1faf51b1598e1b9d909446f90217ec650012b77aee07fc176cf543e3b5e4d"} Nov 01 21:09:18.805810 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:18.805770 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:09:19.053491 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:19.053443 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq"] Nov 01 21:09:19.145532 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:19.145504 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:09:19.177372 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:19.177340 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-dd038b-px89s" Nov 01 21:09:19.265549 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:19.265513 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11dcc6c3-7e00-40d3-8df5-f9b93d2dbc60-kube-api-access-ch6gb\" (UniqueName: \"kubernetes.io/projected/11dcc6c3-7e00-40d3-8df5-f9b93d2dbc60-kube-api-access-ch6gb\") pod \"11dcc6c3-7e00-40d3-8df5-f9b93d2dbc60\" (UID: \"11dcc6c3-7e00-40d3-8df5-f9b93d2dbc60\") " Nov 01 21:09:19.265810 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:19.265789 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11dcc6c3-7e00-40d3-8df5-f9b93d2dbc60-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11dcc6c3-7e00-40d3-8df5-f9b93d2dbc60-maas-api-openshift-ca\") pod \"11dcc6c3-7e00-40d3-8df5-f9b93d2dbc60\" (UID: \"11dcc6c3-7e00-40d3-8df5-f9b93d2dbc60\") " Nov 01 21:09:19.265988 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:19.265968 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11dcc6c3-7e00-40d3-8df5-f9b93d2dbc60-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11dcc6c3-7e00-40d3-8df5-f9b93d2dbc60-maas-api-xks-ca\") pod \"11dcc6c3-7e00-40d3-8df5-f9b93d2dbc60\" (UID: \"11dcc6c3-7e00-40d3-8df5-f9b93d2dbc60\") " Nov 01 21:09:19.266135 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:19.266105 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11dcc6c3-7e00-40d3-8df5-f9b93d2dbc60-maas-api-openshift-ca" pod "11dcc6c3-7e00-40d3-8df5-f9b93d2dbc60" (UID: "11dcc6c3-7e00-40d3-8df5-f9b93d2dbc60"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:09:19.266654 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:19.266630 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11dcc6c3-7e00-40d3-8df5-f9b93d2dbc60-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:09:19.267618 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:19.267594 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11dcc6c3-7e00-40d3-8df5-f9b93d2dbc60-kube-api-access-ch6gb" pod "11dcc6c3-7e00-40d3-8df5-f9b93d2dbc60" (UID: "11dcc6c3-7e00-40d3-8df5-f9b93d2dbc60"). InnerVolumeSpecName "kube-api-access-ch6gb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:09:19.267942 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:19.267916 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11dcc6c3-7e00-40d3-8df5-f9b93d2dbc60-maas-api-xks-ca" pod "11dcc6c3-7e00-40d3-8df5-f9b93d2dbc60" (UID: "11dcc6c3-7e00-40d3-8df5-f9b93d2dbc60"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:09:19.312355 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:19.312309 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:09:19.368249 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:19.368156 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11dcc6c3-7e00-40d3-8df5-f9b93d2dbc60-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:09:19.368249 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:19.368184 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch6gb\" (UniqueName: \"kubernetes.io/projected/11dcc6c3-7e00-40d3-8df5-f9b93d2dbc60-kube-api-access-ch6gb\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:09:19.506026 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:19.504552 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:09:19.670080 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:19.669931 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:09:19.722674 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:19.722598 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-670df9-6b57976fb7-rgk2d"] Nov 01 21:09:19.723655 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:19.723630 4484 cpu_manager.go:414] "containerMap: removing container" podUID="11dcc6c3-7e00-40d3-8df5-f9b93d2dbc60" containerName="revoke-keys" Nov 01 21:09:19.723655 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:19.723652 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11dcc6c3-7e00-40d3-8df5-f9b93d2dbc60" containerName="revoke-keys" Nov 01 21:09:19.723827 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:19.723785 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="11dcc6c3-7e00-40d3-8df5-f9b93d2dbc60" containerName="revoke-keys" Nov 01 21:09:19.727773 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:19.727754 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-670df9-6b57976fb7-rgk2d" Nov 01 21:09:19.783150 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:19.783118 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-670df9-6b57976fb7-rgk2d"] Nov 01 21:09:19.785354 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:19.785329 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-670df9-dockercfg-k5kx9\"" Nov 01 21:09:19.785514 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:19.785410 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-670df9\"" Nov 01 21:09:19.883630 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:19.883061 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f34be5c3-8cbe-441e-b86d-7fcbb910db3d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-670df9-6b57976fb7-rgk2d\" (UID: \"f34be5c3-8cbe-441e-b86d-7fcbb910db3d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-670df9-6b57976fb7-rgk2d" Nov 01 21:09:19.883630 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:19.883219 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnjsz\" (UniqueName: \"kubernetes.io/projected/f34be5c3-8cbe-441e-b86d-7fcbb910db3d-kube-api-access-bnjsz\") pod \"payload-pre-processing-e2e-api-b-670df9-6b57976fb7-rgk2d\" (UID: \"f34be5c3-8cbe-441e-b86d-7fcbb910db3d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-670df9-6b57976fb7-rgk2d" Nov 01 21:09:19.986976 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:19.985517 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f34be5c3-8cbe-441e-b86d-7fcbb910db3d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-670df9-6b57976fb7-rgk2d\" (UID: \"f34be5c3-8cbe-441e-b86d-7fcbb910db3d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-670df9-6b57976fb7-rgk2d" Nov 01 21:09:19.986976 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:19.985571 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnjsz\" (UniqueName: \"kubernetes.io/projected/f34be5c3-8cbe-441e-b86d-7fcbb910db3d-kube-api-access-bnjsz\") pod \"payload-pre-processing-e2e-api-b-670df9-6b57976fb7-rgk2d\" (UID: \"f34be5c3-8cbe-441e-b86d-7fcbb910db3d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-670df9-6b57976fb7-rgk2d" Nov 01 21:09:19.986976 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:19.986878 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f34be5c3-8cbe-441e-b86d-7fcbb910db3d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-670df9-6b57976fb7-rgk2d\" (UID: \"f34be5c3-8cbe-441e-b86d-7fcbb910db3d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-670df9-6b57976fb7-rgk2d" Nov 01 21:09:20.023638 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:20.023604 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-dd038b-px89s" Nov 01 21:09:20.023638 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:20.023623 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-dd038b-px89s" event={"ID":"11dcc6c3-7e00-40d3-8df5-f9b93d2dbc60","Type":"ContainerDied","Data":"1c6327145f686bca762311fcf208d2a63405950d59e3e7d17061468dfe062710"} Nov 01 21:09:20.023887 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:20.023666 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c6327145f686bca762311fcf208d2a63405950d59e3e7d17061468dfe062710" Nov 01 21:09:20.053954 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:20.053926 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnjsz\" (UniqueName: \"kubernetes.io/projected/f34be5c3-8cbe-441e-b86d-7fcbb910db3d-kube-api-access-bnjsz\") pod \"payload-pre-processing-e2e-api-b-670df9-6b57976fb7-rgk2d\" (UID: \"f34be5c3-8cbe-441e-b86d-7fcbb910db3d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-670df9-6b57976fb7-rgk2d" Nov 01 21:09:20.343019 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:20.342835 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-670df9-6b57976fb7-rgk2d" Nov 01 21:09:20.405276 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:20.405185 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:09:20.597314 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:20.595759 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:09:20.769299 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:20.769267 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-670df9-6b57976fb7-rgk2d"] Nov 01 21:09:20.769453 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:09:20.769316 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf34be5c3_8cbe_441e_b86d_7fcbb910db3d.slice/crio-afe9077f1bbd576b5ca024747c0c0a5696a62a1b629e53dc4728f312bc07142e WatchSource:0}: Error finding container afe9077f1bbd576b5ca024747c0c0a5696a62a1b629e53dc4728f312bc07142e: Status 404 returned error can't find the container with id afe9077f1bbd576b5ca024747c0c0a5696a62a1b629e53dc4728f312bc07142e Nov 01 21:09:21.032676 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:21.032640 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-670df9-6b57976fb7-rgk2d" event={"ID":"f34be5c3-8cbe-441e-b86d-7fcbb910db3d","Type":"ContainerStarted","Data":"09094612853c5ef1a71feb5503566aeba88f113bddd212a123f7a987e67f6c4e"} Nov 01 21:09:21.032878 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:21.032701 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-670df9-6b57976fb7-rgk2d" Nov 01 21:09:21.032878 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:21.032723 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-670df9-6b57976fb7-rgk2d" event={"ID":"f34be5c3-8cbe-441e-b86d-7fcbb910db3d","Type":"ContainerStarted","Data":"afe9077f1bbd576b5ca024747c0c0a5696a62a1b629e53dc4728f312bc07142e"} Nov 01 21:09:21.151804 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:21.151702 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-670df9-6b57976fb7-rgk2d" podStartSLOduration=2.15168776 podStartE2EDuration="2.15168776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:09:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:09:21.146989631 +0000 UTC m=+2311.924629767" watchObservedRunningTime="2026-11-01 21:09:21.15168776 +0000 UTC m=+2311.929327895" Nov 01 21:09:24.263460 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:24.263417 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:09:24.430869 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:24.430830 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:09:24.607074 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:24.604043 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:09:27.494486 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:27.494453 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:09:27.770571 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:27.770478 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:09:27.887824 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:27.887765 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-044440-79875748f5-mwv7c"] Nov 01 21:09:27.892703 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:27.892683 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-044440-79875748f5-mwv7c" Nov 01 21:09:27.945557 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:27.945526 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-044440-79875748f5-mwv7c"] Nov 01 21:09:27.958385 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:27.958332 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9027b7a9-7a43-4f47-961a-d02c04c52895-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-044440-79875748f5-mwv7c\" (UID: \"9027b7a9-7a43-4f47-961a-d02c04c52895\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-044440-79875748f5-mwv7c" Nov 01 21:09:27.958577 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:27.958459 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltdbb\" (UniqueName: \"kubernetes.io/projected/9027b7a9-7a43-4f47-961a-d02c04c52895-kube-api-access-ltdbb\") pod \"payload-pre-processing-e2e-inf-b-044440-79875748f5-mwv7c\" (UID: \"9027b7a9-7a43-4f47-961a-d02c04c52895\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-044440-79875748f5-mwv7c" Nov 01 21:09:28.018498 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:28.008501 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-044440-dockercfg-kfmzl\"" Nov 01 21:09:28.045076 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:28.019883 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-044440\"" Nov 01 21:09:28.065435 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:28.063982 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltdbb\" (UniqueName: \"kubernetes.io/projected/9027b7a9-7a43-4f47-961a-d02c04c52895-kube-api-access-ltdbb\") pod \"payload-pre-processing-e2e-inf-b-044440-79875748f5-mwv7c\" (UID: \"9027b7a9-7a43-4f47-961a-d02c04c52895\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-044440-79875748f5-mwv7c" Nov 01 21:09:28.065435 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:28.064348 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9027b7a9-7a43-4f47-961a-d02c04c52895-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-044440-79875748f5-mwv7c\" (UID: \"9027b7a9-7a43-4f47-961a-d02c04c52895\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-044440-79875748f5-mwv7c" Nov 01 21:09:28.069842 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:28.069813 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9027b7a9-7a43-4f47-961a-d02c04c52895-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-044440-79875748f5-mwv7c\" (UID: \"9027b7a9-7a43-4f47-961a-d02c04c52895\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-044440-79875748f5-mwv7c" Nov 01 21:09:28.136766 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:28.136729 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltdbb\" (UniqueName: \"kubernetes.io/projected/9027b7a9-7a43-4f47-961a-d02c04c52895-kube-api-access-ltdbb\") pod \"payload-pre-processing-e2e-inf-b-044440-79875748f5-mwv7c\" (UID: \"9027b7a9-7a43-4f47-961a-d02c04c52895\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-044440-79875748f5-mwv7c" Nov 01 21:09:28.207266 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:28.207228 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-044440-79875748f5-mwv7c" Nov 01 21:09:28.574588 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:28.574552 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-dd038b-58bf969f7c-5tbbt"] Nov 01 21:09:28.584721 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:09:28.584690 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9027b7a9_7a43_4f47_961a_d02c04c52895.slice/crio-28bfc598a4d84342a2c2a1d3bc7cfbe41d72d89bbd9c61fce9036deca2c4d4a8 WatchSource:0}: Error finding container 28bfc598a4d84342a2c2a1d3bc7cfbe41d72d89bbd9c61fce9036deca2c4d4a8: Status 404 returned error can't find the container with id 28bfc598a4d84342a2c2a1d3bc7cfbe41d72d89bbd9c61fce9036deca2c4d4a8 Nov 01 21:09:28.584842 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:28.584803 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-044440-79875748f5-mwv7c"] Nov 01 21:09:28.687460 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:28.687423 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-dd038b-c5cc45b55-nd59r"] Nov 01 21:09:28.966296 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:28.966203 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:09:29.132036 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:29.128584 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:09:29.181618 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:29.181572 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-044440-79875748f5-mwv7c" event={"ID":"9027b7a9-7a43-4f47-961a-d02c04c52895","Type":"ContainerStarted","Data":"fdb2510dd61330ae38d7cf624d911819d939d24b37fa214a7337b77d9c364c69"} Nov 01 21:09:29.181914 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:29.181881 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-044440-79875748f5-mwv7c" event={"ID":"9027b7a9-7a43-4f47-961a-d02c04c52895","Type":"ContainerStarted","Data":"28bfc598a4d84342a2c2a1d3bc7cfbe41d72d89bbd9c61fce9036deca2c4d4a8"} Nov 01 21:09:29.182116 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:29.182097 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-044440-79875748f5-mwv7c" Nov 01 21:09:29.298315 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:29.298266 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-044440-79875748f5-mwv7c" podStartSLOduration=2.298253619 podStartE2EDuration="2.298253619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:09:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:09:29.297495729 +0000 UTC m=+2320.075135865" watchObservedRunningTime="2026-11-01 21:09:29.298253619 +0000 UTC m=+2320.075893753" Nov 01 21:09:29.697359 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:29.697282 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:09:29.857980 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:29.857946 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:09:30.014047 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:30.013979 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:09:32.049780 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:32.049747 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-670df9-6b57976fb7-rgk2d" Nov 01 21:09:33.575853 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:33.575793 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-dd038b-58bf969f7c-5tbbt" podUID="9406d322-46c5-4c90-b984-46496f105414" containerName="payload-processing" containerID="cri-o://82f165a7468d8bb857f5da25cd812adb42a71208286cb149f2110085eb783ff7" gracePeriod=25 Nov 01 21:09:33.688321 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:33.688278 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-dd038b-c5cc45b55-nd59r" podUID="bbbfd5b3-4926-4ec2-beff-1b224fa49c5c" containerName="payload-pre-processing" containerID="cri-o://e943c9c1d9eed282da49ed07f1e09205e45dae2b070cc1a0a5a0d03c44e842b8" gracePeriod=25 Nov 01 21:09:34.076510 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:34.076473 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" podUID="e175b298-2c23-45d4-8f9c-0c9b9f6aab92" containerName="main" containerID="cri-o://955e23e9322ec7be3ec445fa9c82cdd1ba700e26a4d4c404cf4bdfa1f51bf859" gracePeriod=45 Nov 01 21:09:34.233738 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:34.233706 4484 generic.go:356] "Generic (PLEG): container finished" podID="e175b298-2c23-45d4-8f9c-0c9b9f6aab92" containerID="955e23e9322ec7be3ec445fa9c82cdd1ba700e26a4d4c404cf4bdfa1f51bf859" exitCode=0 Nov 01 21:09:34.233906 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:34.233755 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" event={"ID":"e175b298-2c23-45d4-8f9c-0c9b9f6aab92","Type":"ContainerDied","Data":"955e23e9322ec7be3ec445fa9c82cdd1ba700e26a4d4c404cf4bdfa1f51bf859"} Nov 01 21:09:34.325574 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:34.325446 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-dd038b-58bf969f7c-5tbbt" podUID="9406d322-46c5-4c90-b984-46496f105414" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 21:09:34.326137 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:34.326095 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-dd038b-c5cc45b55-nd59r" podUID="bbbfd5b3-4926-4ec2-beff-1b224fa49c5c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 21:09:34.383246 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:34.383214 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" Nov 01 21:09:34.499554 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:34.499398 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e175b298-2c23-45d4-8f9c-0c9b9f6aab92-tls-certs\" (UniqueName: \"kubernetes.io/secret/e175b298-2c23-45d4-8f9c-0c9b9f6aab92-tls-certs\") pod \"e175b298-2c23-45d4-8f9c-0c9b9f6aab92\" (UID: \"e175b298-2c23-45d4-8f9c-0c9b9f6aab92\") " Nov 01 21:09:34.499554 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:34.499510 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e175b298-2c23-45d4-8f9c-0c9b9f6aab92-home\" (UniqueName: \"kubernetes.io/empty-dir/e175b298-2c23-45d4-8f9c-0c9b9f6aab92-home\") pod \"e175b298-2c23-45d4-8f9c-0c9b9f6aab92\" (UID: \"e175b298-2c23-45d4-8f9c-0c9b9f6aab92\") " Nov 01 21:09:34.499554 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:34.499555 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e175b298-2c23-45d4-8f9c-0c9b9f6aab92-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e175b298-2c23-45d4-8f9c-0c9b9f6aab92-model-cache\") pod \"e175b298-2c23-45d4-8f9c-0c9b9f6aab92\" (UID: \"e175b298-2c23-45d4-8f9c-0c9b9f6aab92\") " Nov 01 21:09:34.499885 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:34.499652 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e175b298-2c23-45d4-8f9c-0c9b9f6aab92-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e175b298-2c23-45d4-8f9c-0c9b9f6aab92-tmp-dir\") pod \"e175b298-2c23-45d4-8f9c-0c9b9f6aab92\" (UID: \"e175b298-2c23-45d4-8f9c-0c9b9f6aab92\") " Nov 01 21:09:34.499885 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:34.499786 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e175b298-2c23-45d4-8f9c-0c9b9f6aab92-model-cache" pod "e175b298-2c23-45d4-8f9c-0c9b9f6aab92" (UID: "e175b298-2c23-45d4-8f9c-0c9b9f6aab92"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:09:34.499885 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:34.499829 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e175b298-2c23-45d4-8f9c-0c9b9f6aab92-home" pod "e175b298-2c23-45d4-8f9c-0c9b9f6aab92" (UID: "e175b298-2c23-45d4-8f9c-0c9b9f6aab92"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:09:34.500025 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:34.499909 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e175b298-2c23-45d4-8f9c-0c9b9f6aab92-tmp-dir" pod "e175b298-2c23-45d4-8f9c-0c9b9f6aab92" (UID: "e175b298-2c23-45d4-8f9c-0c9b9f6aab92"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:09:34.500096 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:34.500038 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e175b298-2c23-45d4-8f9c-0c9b9f6aab92-kube-api-access-w5nvb\" (UniqueName: \"kubernetes.io/projected/e175b298-2c23-45d4-8f9c-0c9b9f6aab92-kube-api-access-w5nvb\") pod \"e175b298-2c23-45d4-8f9c-0c9b9f6aab92\" (UID: \"e175b298-2c23-45d4-8f9c-0c9b9f6aab92\") " Nov 01 21:09:34.500157 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:34.500115 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e175b298-2c23-45d4-8f9c-0c9b9f6aab92-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e175b298-2c23-45d4-8f9c-0c9b9f6aab92-dshm\") pod \"e175b298-2c23-45d4-8f9c-0c9b9f6aab92\" (UID: \"e175b298-2c23-45d4-8f9c-0c9b9f6aab92\") " Nov 01 21:09:34.502864 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:34.501625 4484 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e175b298-2c23-45d4-8f9c-0c9b9f6aab92-tmp-dir\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:09:34.502864 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:34.501651 4484 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e175b298-2c23-45d4-8f9c-0c9b9f6aab92-home\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:09:34.502864 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:34.501668 4484 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e175b298-2c23-45d4-8f9c-0c9b9f6aab92-model-cache\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:09:34.502864 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:34.502535 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e175b298-2c23-45d4-8f9c-0c9b9f6aab92-tls-certs" pod "e175b298-2c23-45d4-8f9c-0c9b9f6aab92" (UID: "e175b298-2c23-45d4-8f9c-0c9b9f6aab92"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:09:34.503394 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:34.503361 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e175b298-2c23-45d4-8f9c-0c9b9f6aab92-dshm" pod "e175b298-2c23-45d4-8f9c-0c9b9f6aab92" (UID: "e175b298-2c23-45d4-8f9c-0c9b9f6aab92"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:09:34.503622 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:34.503601 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e175b298-2c23-45d4-8f9c-0c9b9f6aab92-kube-api-access-w5nvb" pod "e175b298-2c23-45d4-8f9c-0c9b9f6aab92" (UID: "e175b298-2c23-45d4-8f9c-0c9b9f6aab92"). InnerVolumeSpecName "kube-api-access-w5nvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:09:34.603381 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:34.603281 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5nvb\" (UniqueName: \"kubernetes.io/projected/e175b298-2c23-45d4-8f9c-0c9b9f6aab92-kube-api-access-w5nvb\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:09:34.603381 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:34.603326 4484 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e175b298-2c23-45d4-8f9c-0c9b9f6aab92-dshm\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:09:34.603381 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:34.603342 4484 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e175b298-2c23-45d4-8f9c-0c9b9f6aab92-tls-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:09:35.054201 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:35.054151 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:09:35.250210 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:35.250167 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" event={"ID":"e175b298-2c23-45d4-8f9c-0c9b9f6aab92","Type":"ContainerDied","Data":"81c64095598fdb4a7d88ed5cbbc49ff9e687b2f67fc0d5104c73ac933e48d14a"} Nov 01 21:09:35.250415 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:35.250231 4484 scope.go:122] "RemoveContainer" containerID="955e23e9322ec7be3ec445fa9c82cdd1ba700e26a4d4c404cf4bdfa1f51bf859" Nov 01 21:09:35.250415 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:35.250397 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq" Nov 01 21:09:35.465110 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:35.461049 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq"] Nov 01 21:09:35.476234 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:35.475888 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-dd038b/distinct-2-w6-dd038b-kserve-78fbb76987-ww2kq"] Nov 01 21:09:37.379153 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:37.379121 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e175b298-2c23-45d4-8f9c-0c9b9f6aab92" path="/var/lib/kubelet/pods/e175b298-2c23-45d4-8f9c-0c9b9f6aab92/volumes" Nov 01 21:09:39.942741 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:39.942704 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:09:40.196885 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:40.196807 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-044440-79875748f5-mwv7c" Nov 01 21:09:44.315898 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:44.315856 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-dd038b-58bf969f7c-5tbbt" podUID="9406d322-46c5-4c90-b984-46496f105414" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 21:09:44.316425 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:44.315871 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-dd038b-c5cc45b55-nd59r" podUID="bbbfd5b3-4926-4ec2-beff-1b224fa49c5c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 21:09:44.718639 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:44.718551 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:09:45.768108 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:09:45.768076 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:09:46.207656 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:46.207562 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:09:49.332320 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:49.332233 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:09:49.502653 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:49.502584 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:09:52.438621 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:52.438528 4484 generic.go:356] "Generic (PLEG): container finished" podID="78640135-a1da-48d7-8212-ecb337022dee" containerID="03cea7da2cbcb8c669251d370a2921fb6963efb834d18761f5e905ae83bffb7e" exitCode=1 Nov 01 21:09:52.438621 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:52.438600 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-gx48d" event={"ID":"78640135-a1da-48d7-8212-ecb337022dee","Type":"ContainerDied","Data":"03cea7da2cbcb8c669251d370a2921fb6963efb834d18761f5e905ae83bffb7e"} Nov 01 21:09:52.439184 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:52.439151 4484 scope.go:122] "RemoveContainer" containerID="03cea7da2cbcb8c669251d370a2921fb6963efb834d18761f5e905ae83bffb7e" Nov 01 21:09:53.447556 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:53.447524 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-gx48d" event={"ID":"78640135-a1da-48d7-8212-ecb337022dee","Type":"ContainerStarted","Data":"49a76b817aa86f434f5d38a00d54765e2000d3067458f8042e46eb764c2fdc46"} Nov 01 21:09:53.448025 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:53.447720 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-gx48d" Nov 01 21:09:53.868946 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:53.868919 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-dd038b-58bf969f7c-5tbbt" Nov 01 21:09:53.943809 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:53.943776 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9406d322-46c5-4c90-b984-46496f105414-kube-api-access-wwnk9\" (UniqueName: \"kubernetes.io/projected/9406d322-46c5-4c90-b984-46496f105414-kube-api-access-wwnk9\") pod \"9406d322-46c5-4c90-b984-46496f105414\" (UID: \"9406d322-46c5-4c90-b984-46496f105414\") " Nov 01 21:09:53.945288 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:53.943980 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9406d322-46c5-4c90-b984-46496f105414-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9406d322-46c5-4c90-b984-46496f105414-plugins-config-volume\") pod \"9406d322-46c5-4c90-b984-46496f105414\" (UID: \"9406d322-46c5-4c90-b984-46496f105414\") " Nov 01 21:09:53.945288 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:53.944475 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9406d322-46c5-4c90-b984-46496f105414-plugins-config-volume" pod "9406d322-46c5-4c90-b984-46496f105414" (UID: "9406d322-46c5-4c90-b984-46496f105414"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:09:53.946350 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:53.946320 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9406d322-46c5-4c90-b984-46496f105414-kube-api-access-wwnk9" pod "9406d322-46c5-4c90-b984-46496f105414" (UID: "9406d322-46c5-4c90-b984-46496f105414"). InnerVolumeSpecName "kube-api-access-wwnk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:09:53.970966 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:53.970912 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-dd038b-c5cc45b55-nd59r" Nov 01 21:09:54.044901 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:54.044870 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbbfd5b3-4926-4ec2-beff-1b224fa49c5c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbbfd5b3-4926-4ec2-beff-1b224fa49c5c-plugins-config-volume\") pod \"bbbfd5b3-4926-4ec2-beff-1b224fa49c5c\" (UID: \"bbbfd5b3-4926-4ec2-beff-1b224fa49c5c\") " Nov 01 21:09:54.045119 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:54.045104 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbbfd5b3-4926-4ec2-beff-1b224fa49c5c-kube-api-access-rkmgk\" (UniqueName: \"kubernetes.io/projected/bbbfd5b3-4926-4ec2-beff-1b224fa49c5c-kube-api-access-rkmgk\") pod \"bbbfd5b3-4926-4ec2-beff-1b224fa49c5c\" (UID: \"bbbfd5b3-4926-4ec2-beff-1b224fa49c5c\") " Nov 01 21:09:54.045284 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:54.045260 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbbfd5b3-4926-4ec2-beff-1b224fa49c5c-plugins-config-volume" pod "bbbfd5b3-4926-4ec2-beff-1b224fa49c5c" (UID: "bbbfd5b3-4926-4ec2-beff-1b224fa49c5c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:09:54.045578 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:54.045560 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwnk9\" (UniqueName: \"kubernetes.io/projected/9406d322-46c5-4c90-b984-46496f105414-kube-api-access-wwnk9\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:09:54.045660 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:54.045581 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9406d322-46c5-4c90-b984-46496f105414-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:09:54.045660 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:54.045590 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbbfd5b3-4926-4ec2-beff-1b224fa49c5c-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:09:54.047186 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:54.047163 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbbfd5b3-4926-4ec2-beff-1b224fa49c5c-kube-api-access-rkmgk" pod "bbbfd5b3-4926-4ec2-beff-1b224fa49c5c" (UID: "bbbfd5b3-4926-4ec2-beff-1b224fa49c5c"). InnerVolumeSpecName "kube-api-access-rkmgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:09:54.147273 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:54.147239 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkmgk\" (UniqueName: \"kubernetes.io/projected/bbbfd5b3-4926-4ec2-beff-1b224fa49c5c-kube-api-access-rkmgk\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:09:54.385024 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:54.383663 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:09:54.457794 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:54.457760 4484 generic.go:356] "Generic (PLEG): container finished" podID="9406d322-46c5-4c90-b984-46496f105414" containerID="82f165a7468d8bb857f5da25cd812adb42a71208286cb149f2110085eb783ff7" exitCode=0 Nov 01 21:09:54.458332 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:54.457806 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-dd038b-58bf969f7c-5tbbt" Nov 01 21:09:54.458332 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:54.457845 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-dd038b-58bf969f7c-5tbbt" event={"ID":"9406d322-46c5-4c90-b984-46496f105414","Type":"ContainerDied","Data":"82f165a7468d8bb857f5da25cd812adb42a71208286cb149f2110085eb783ff7"} Nov 01 21:09:54.458332 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:54.457892 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-dd038b-58bf969f7c-5tbbt" event={"ID":"9406d322-46c5-4c90-b984-46496f105414","Type":"ContainerDied","Data":"e234dffd0333c2468855c5f9839bcaba180946b1caeaeb3743a7d071e98de909"} Nov 01 21:09:54.458332 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:54.457916 4484 scope.go:122] "RemoveContainer" containerID="82f165a7468d8bb857f5da25cd812adb42a71208286cb149f2110085eb783ff7" Nov 01 21:09:54.459500 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:54.459474 4484 generic.go:356] "Generic (PLEG): container finished" podID="bbbfd5b3-4926-4ec2-beff-1b224fa49c5c" containerID="e943c9c1d9eed282da49ed07f1e09205e45dae2b070cc1a0a5a0d03c44e842b8" exitCode=0 Nov 01 21:09:54.459639 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:54.459510 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-dd038b-c5cc45b55-nd59r" Nov 01 21:09:54.459639 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:54.459517 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-dd038b-c5cc45b55-nd59r" event={"ID":"bbbfd5b3-4926-4ec2-beff-1b224fa49c5c","Type":"ContainerDied","Data":"e943c9c1d9eed282da49ed07f1e09205e45dae2b070cc1a0a5a0d03c44e842b8"} Nov 01 21:09:54.459639 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:54.459599 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-dd038b-c5cc45b55-nd59r" event={"ID":"bbbfd5b3-4926-4ec2-beff-1b224fa49c5c","Type":"ContainerDied","Data":"9f119e8f28a6e1233d7c417a75f45c0543d511f218ad30c22fb15370ed044769"} Nov 01 21:09:54.471500 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:54.471480 4484 scope.go:122] "RemoveContainer" containerID="82f165a7468d8bb857f5da25cd812adb42a71208286cb149f2110085eb783ff7" Nov 01 21:09:54.471814 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:09:54.471795 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82f165a7468d8bb857f5da25cd812adb42a71208286cb149f2110085eb783ff7\": container with ID starting with 82f165a7468d8bb857f5da25cd812adb42a71208286cb149f2110085eb783ff7 not found: ID does not exist" containerID="82f165a7468d8bb857f5da25cd812adb42a71208286cb149f2110085eb783ff7" Nov 01 21:09:54.471875 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:54.471825 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82f165a7468d8bb857f5da25cd812adb42a71208286cb149f2110085eb783ff7"} err="failed to get container status \"82f165a7468d8bb857f5da25cd812adb42a71208286cb149f2110085eb783ff7\": rpc error: code = NotFound desc = could not find container \"82f165a7468d8bb857f5da25cd812adb42a71208286cb149f2110085eb783ff7\": container with ID starting with 82f165a7468d8bb857f5da25cd812adb42a71208286cb149f2110085eb783ff7 not found: ID does not exist" Nov 01 21:09:54.471875 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:54.471844 4484 scope.go:122] "RemoveContainer" containerID="e943c9c1d9eed282da49ed07f1e09205e45dae2b070cc1a0a5a0d03c44e842b8" Nov 01 21:09:54.487450 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:54.487425 4484 scope.go:122] "RemoveContainer" containerID="e943c9c1d9eed282da49ed07f1e09205e45dae2b070cc1a0a5a0d03c44e842b8" Nov 01 21:09:54.487779 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:09:54.487754 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e943c9c1d9eed282da49ed07f1e09205e45dae2b070cc1a0a5a0d03c44e842b8\": container with ID starting with e943c9c1d9eed282da49ed07f1e09205e45dae2b070cc1a0a5a0d03c44e842b8 not found: ID does not exist" containerID="e943c9c1d9eed282da49ed07f1e09205e45dae2b070cc1a0a5a0d03c44e842b8" Nov 01 21:09:54.487883 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:54.487786 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e943c9c1d9eed282da49ed07f1e09205e45dae2b070cc1a0a5a0d03c44e842b8"} err="failed to get container status \"e943c9c1d9eed282da49ed07f1e09205e45dae2b070cc1a0a5a0d03c44e842b8\": rpc error: code = NotFound desc = could not find container \"e943c9c1d9eed282da49ed07f1e09205e45dae2b070cc1a0a5a0d03c44e842b8\": container with ID starting with e943c9c1d9eed282da49ed07f1e09205e45dae2b070cc1a0a5a0d03c44e842b8 not found: ID does not exist" Nov 01 21:09:54.625134 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:54.625099 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-dd038b-c5cc45b55-nd59r"] Nov 01 21:09:54.630259 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:54.630232 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-dd038b-c5cc45b55-nd59r"] Nov 01 21:09:54.743353 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:54.743322 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-dd038b-58bf969f7c-5tbbt"] Nov 01 21:09:54.746922 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:54.746900 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-dd038b-58bf969f7c-5tbbt"] Nov 01 21:09:55.373765 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:55.373731 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9406d322-46c5-4c90-b984-46496f105414" path="/var/lib/kubelet/pods/9406d322-46c5-4c90-b984-46496f105414/volumes" Nov 01 21:09:55.374227 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:55.374212 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbbfd5b3-4926-4ec2-beff-1b224fa49c5c" path="/var/lib/kubelet/pods/bbbfd5b3-4926-4ec2-beff-1b224fa49c5c/volumes" Nov 01 21:09:56.122563 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:56.122524 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:09:56.285050 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:56.284987 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:09:56.584157 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:56.584120 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:09:59.101493 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:09:59.101449 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:10:00.698858 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:00.698828 4484 scope.go:122] "RemoveContainer" containerID="b6d3070e0b2dfb4725aa64e643d90abec766cb2ca0f5992921c9662b89168ea1" Nov 01 21:10:00.711517 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:00.711492 4484 scope.go:122] "RemoveContainer" containerID="44cb9caf6cb1e23f4ad56089cfb37e048be8926bf43b1c9d72e95b0a932995fc" Nov 01 21:10:00.721568 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:00.721542 4484 scope.go:122] "RemoveContainer" containerID="b7e770acf075c92550fa312043aa00c3c5ce476e52b5ea1f3a9d178c4f0869f6" Nov 01 21:10:02.554175 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:02.554141 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:10:02.725393 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:02.725355 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:10:03.836521 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:03.836478 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:10:06.418739 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:06.418687 4484 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-35.us-west-1.compute.internal" podUID="f192393bd55f8e34f5db37f9db86763a" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded" Nov 01 21:10:08.610668 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:08.610538 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:10:13.228022 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:13.226676 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:10:13.389978 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:13.389935 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:10:14.253615 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:14.253573 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:10:14.443017 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:14.437290 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:10:14.461649 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:14.461622 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-gx48d" Nov 01 21:10:15.786948 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:10:15.786910 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:10:17.601807 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:17.601769 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d74f8c1e-sdwfd"] Nov 01 21:10:17.602668 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:17.602648 4484 cpu_manager.go:414] "containerMap: removing container" podUID="bbbfd5b3-4926-4ec2-beff-1b224fa49c5c" containerName="payload-pre-processing" Nov 01 21:10:17.602836 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:17.602817 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbbfd5b3-4926-4ec2-beff-1b224fa49c5c" containerName="payload-pre-processing" Nov 01 21:10:17.602926 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:17.602839 4484 cpu_manager.go:414] "containerMap: removing container" podUID="e175b298-2c23-45d4-8f9c-0c9b9f6aab92" containerName="main" Nov 01 21:10:17.602926 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:17.602847 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e175b298-2c23-45d4-8f9c-0c9b9f6aab92" containerName="main" Nov 01 21:10:17.602926 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:17.602907 4484 cpu_manager.go:414] "containerMap: removing container" podUID="9406d322-46c5-4c90-b984-46496f105414" containerName="payload-processing" Nov 01 21:10:17.602926 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:17.602916 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9406d322-46c5-4c90-b984-46496f105414" containerName="payload-processing" Nov 01 21:10:17.603173 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:17.603157 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbbfd5b3-4926-4ec2-beff-1b224fa49c5c" containerName="payload-pre-processing" Nov 01 21:10:17.603236 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:17.603176 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="e175b298-2c23-45d4-8f9c-0c9b9f6aab92" containerName="main" Nov 01 21:10:17.603236 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:17.603195 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="9406d322-46c5-4c90-b984-46496f105414" containerName="payload-processing" Nov 01 21:10:17.607178 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:17.607154 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74f8c1e-sdwfd" Nov 01 21:10:17.660046 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:17.660011 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9651f055-qb4vx"] Nov 01 21:10:17.672250 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:17.672216 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d74f8c1e-sdwfd"] Nov 01 21:10:17.672451 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:17.672340 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9651f055-qb4vx" Nov 01 21:10:17.704818 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:17.704787 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:10:17.704818 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:17.704795 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:10:17.707803 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:17.707772 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69b1f9a7-bdcc-4a23-937e-07957ceca362-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9651f055-qb4vx\" (UID: \"69b1f9a7-bdcc-4a23-937e-07957ceca362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9651f055-qb4vx" Nov 01 21:10:17.708026 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:17.707971 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69b1f9a7-bdcc-4a23-937e-07957ceca362-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9651f055-qb4vx\" (UID: \"69b1f9a7-bdcc-4a23-937e-07957ceca362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9651f055-qb4vx" Nov 01 21:10:17.708283 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:17.708261 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7n92\" (UniqueName: \"kubernetes.io/projected/69b1f9a7-bdcc-4a23-937e-07957ceca362-kube-api-access-n7n92\") pod \"maas-api-revoke-sub-9651f055-qb4vx\" (UID: \"69b1f9a7-bdcc-4a23-937e-07957ceca362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9651f055-qb4vx" Nov 01 21:10:17.708432 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:17.708417 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clp5z\" (UniqueName: \"kubernetes.io/projected/8b380568-48c5-4410-8ddc-3537d86ad229-kube-api-access-clp5z\") pod \"maas-api-revoke-sub-d74f8c1e-sdwfd\" (UID: \"8b380568-48c5-4410-8ddc-3537d86ad229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74f8c1e-sdwfd" Nov 01 21:10:17.708645 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:17.708631 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b380568-48c5-4410-8ddc-3537d86ad229-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d74f8c1e-sdwfd\" (UID: \"8b380568-48c5-4410-8ddc-3537d86ad229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74f8c1e-sdwfd" Nov 01 21:10:17.708853 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:17.708839 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b380568-48c5-4410-8ddc-3537d86ad229-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d74f8c1e-sdwfd\" (UID: \"8b380568-48c5-4410-8ddc-3537d86ad229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74f8c1e-sdwfd" Nov 01 21:10:17.710199 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:17.710141 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:10:17.710199 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:17.710160 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:10:17.720016 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:17.719958 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9651f055-qb4vx"] Nov 01 21:10:17.810722 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:17.810682 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clp5z\" (UniqueName: \"kubernetes.io/projected/8b380568-48c5-4410-8ddc-3537d86ad229-kube-api-access-clp5z\") pod \"maas-api-revoke-sub-d74f8c1e-sdwfd\" (UID: \"8b380568-48c5-4410-8ddc-3537d86ad229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74f8c1e-sdwfd" Nov 01 21:10:17.810948 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:17.810816 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b380568-48c5-4410-8ddc-3537d86ad229-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d74f8c1e-sdwfd\" (UID: \"8b380568-48c5-4410-8ddc-3537d86ad229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74f8c1e-sdwfd" Nov 01 21:10:17.810948 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:17.810856 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b380568-48c5-4410-8ddc-3537d86ad229-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d74f8c1e-sdwfd\" (UID: \"8b380568-48c5-4410-8ddc-3537d86ad229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74f8c1e-sdwfd" Nov 01 21:10:17.810948 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:17.810909 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69b1f9a7-bdcc-4a23-937e-07957ceca362-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9651f055-qb4vx\" (UID: \"69b1f9a7-bdcc-4a23-937e-07957ceca362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9651f055-qb4vx" Nov 01 21:10:17.811169 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:17.811020 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69b1f9a7-bdcc-4a23-937e-07957ceca362-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9651f055-qb4vx\" (UID: \"69b1f9a7-bdcc-4a23-937e-07957ceca362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9651f055-qb4vx" Nov 01 21:10:17.811169 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:17.811047 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7n92\" (UniqueName: \"kubernetes.io/projected/69b1f9a7-bdcc-4a23-937e-07957ceca362-kube-api-access-n7n92\") pod \"maas-api-revoke-sub-9651f055-qb4vx\" (UID: \"69b1f9a7-bdcc-4a23-937e-07957ceca362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9651f055-qb4vx" Nov 01 21:10:17.811652 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:17.811627 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b380568-48c5-4410-8ddc-3537d86ad229-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d74f8c1e-sdwfd\" (UID: \"8b380568-48c5-4410-8ddc-3537d86ad229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74f8c1e-sdwfd" Nov 01 21:10:17.812195 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:17.812170 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69b1f9a7-bdcc-4a23-937e-07957ceca362-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9651f055-qb4vx\" (UID: \"69b1f9a7-bdcc-4a23-937e-07957ceca362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9651f055-qb4vx" Nov 01 21:10:17.813923 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:17.813861 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b380568-48c5-4410-8ddc-3537d86ad229-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d74f8c1e-sdwfd\" (UID: \"8b380568-48c5-4410-8ddc-3537d86ad229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74f8c1e-sdwfd" Nov 01 21:10:17.813923 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:17.813860 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69b1f9a7-bdcc-4a23-937e-07957ceca362-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9651f055-qb4vx\" (UID: \"69b1f9a7-bdcc-4a23-937e-07957ceca362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9651f055-qb4vx" Nov 01 21:10:17.869634 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:17.869376 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7n92\" (UniqueName: \"kubernetes.io/projected/69b1f9a7-bdcc-4a23-937e-07957ceca362-kube-api-access-n7n92\") pod \"maas-api-revoke-sub-9651f055-qb4vx\" (UID: \"69b1f9a7-bdcc-4a23-937e-07957ceca362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9651f055-qb4vx" Nov 01 21:10:17.869634 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:17.869428 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clp5z\" (UniqueName: \"kubernetes.io/projected/8b380568-48c5-4410-8ddc-3537d86ad229-kube-api-access-clp5z\") pod \"maas-api-revoke-sub-d74f8c1e-sdwfd\" (UID: \"8b380568-48c5-4410-8ddc-3537d86ad229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74f8c1e-sdwfd" Nov 01 21:10:17.931438 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:17.931396 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74f8c1e-sdwfd" Nov 01 21:10:17.996484 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:17.996445 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9651f055-qb4vx" Nov 01 21:10:18.333635 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:10:18.333607 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b380568_48c5_4410_8ddc_3537d86ad229.slice/crio-359a851ecde11b0fc638530a3ffa0996966acf266b3a84f0cac03c013619bff5 WatchSource:0}: Error finding container 359a851ecde11b0fc638530a3ffa0996966acf266b3a84f0cac03c013619bff5: Status 404 returned error can't find the container with id 359a851ecde11b0fc638530a3ffa0996966acf266b3a84f0cac03c013619bff5 Nov 01 21:10:18.334596 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:18.334422 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d74f8c1e-sdwfd"] Nov 01 21:10:18.424226 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:18.424136 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:10:18.435431 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:10:18.435387 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69b1f9a7_bdcc_4a23_937e_07957ceca362.slice/crio-3e21d1bced3cacc54dd83c4bdba107b7f1bb0df7e1990e262a7e6c8f1feb59ab WatchSource:0}: Error finding container 3e21d1bced3cacc54dd83c4bdba107b7f1bb0df7e1990e262a7e6c8f1feb59ab: Status 404 returned error can't find the container with id 3e21d1bced3cacc54dd83c4bdba107b7f1bb0df7e1990e262a7e6c8f1feb59ab Nov 01 21:10:18.435890 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:18.435855 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9651f055-qb4vx"] Nov 01 21:10:18.731367 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:18.731332 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74f8c1e-sdwfd" event={"ID":"8b380568-48c5-4410-8ddc-3537d86ad229","Type":"ContainerStarted","Data":"e733ba9ef7e45667878e32f24afcc597b6b95710e2e7c89005ab21e89f843531"} Nov 01 21:10:18.731766 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:18.731400 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74f8c1e-sdwfd" event={"ID":"8b380568-48c5-4410-8ddc-3537d86ad229","Type":"ContainerStarted","Data":"359a851ecde11b0fc638530a3ffa0996966acf266b3a84f0cac03c013619bff5"} Nov 01 21:10:18.732675 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:18.732652 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9651f055-qb4vx" event={"ID":"69b1f9a7-bdcc-4a23-937e-07957ceca362","Type":"ContainerStarted","Data":"3e21d1bced3cacc54dd83c4bdba107b7f1bb0df7e1990e262a7e6c8f1feb59ab"} Nov 01 21:10:19.744033 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:19.743976 4484 generic.go:356] "Generic (PLEG): container finished" podID="8b380568-48c5-4410-8ddc-3537d86ad229" containerID="e733ba9ef7e45667878e32f24afcc597b6b95710e2e7c89005ab21e89f843531" exitCode=0 Nov 01 21:10:19.744506 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:19.744067 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74f8c1e-sdwfd" event={"ID":"8b380568-48c5-4410-8ddc-3537d86ad229","Type":"ContainerDied","Data":"e733ba9ef7e45667878e32f24afcc597b6b95710e2e7c89005ab21e89f843531"} Nov 01 21:10:19.746188 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:19.746166 4484 generic.go:356] "Generic (PLEG): container finished" podID="69b1f9a7-bdcc-4a23-937e-07957ceca362" containerID="ee0ed5bd5c45470bbf9bcf9498fd27943dc4ad13b355e1618edf0295167a51e6" exitCode=0 Nov 01 21:10:19.746320 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:19.746199 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9651f055-qb4vx" event={"ID":"69b1f9a7-bdcc-4a23-937e-07957ceca362","Type":"ContainerDied","Data":"ee0ed5bd5c45470bbf9bcf9498fd27943dc4ad13b355e1618edf0295167a51e6"} Nov 01 21:10:21.010464 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:21.010386 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9651f055-qb4vx" Nov 01 21:10:21.026369 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:21.026334 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74f8c1e-sdwfd" Nov 01 21:10:21.064557 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:21.064528 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b380568-48c5-4410-8ddc-3537d86ad229-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b380568-48c5-4410-8ddc-3537d86ad229-maas-api-openshift-ca\") pod \"8b380568-48c5-4410-8ddc-3537d86ad229\" (UID: \"8b380568-48c5-4410-8ddc-3537d86ad229\") " Nov 01 21:10:21.064745 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:21.064593 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69b1f9a7-bdcc-4a23-937e-07957ceca362-kube-api-access-n7n92\" (UniqueName: \"kubernetes.io/projected/69b1f9a7-bdcc-4a23-937e-07957ceca362-kube-api-access-n7n92\") pod \"69b1f9a7-bdcc-4a23-937e-07957ceca362\" (UID: \"69b1f9a7-bdcc-4a23-937e-07957ceca362\") " Nov 01 21:10:21.064745 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:21.064697 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69b1f9a7-bdcc-4a23-937e-07957ceca362-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69b1f9a7-bdcc-4a23-937e-07957ceca362-maas-api-openshift-ca\") pod \"69b1f9a7-bdcc-4a23-937e-07957ceca362\" (UID: \"69b1f9a7-bdcc-4a23-937e-07957ceca362\") " Nov 01 21:10:21.064745 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:21.064737 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69b1f9a7-bdcc-4a23-937e-07957ceca362-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69b1f9a7-bdcc-4a23-937e-07957ceca362-maas-api-xks-ca\") pod \"69b1f9a7-bdcc-4a23-937e-07957ceca362\" (UID: \"69b1f9a7-bdcc-4a23-937e-07957ceca362\") " Nov 01 21:10:21.064851 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:21.064803 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b380568-48c5-4410-8ddc-3537d86ad229-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b380568-48c5-4410-8ddc-3537d86ad229-maas-api-xks-ca\") pod \"8b380568-48c5-4410-8ddc-3537d86ad229\" (UID: \"8b380568-48c5-4410-8ddc-3537d86ad229\") " Nov 01 21:10:21.064851 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:21.064835 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b380568-48c5-4410-8ddc-3537d86ad229-kube-api-access-clp5z\" (UniqueName: \"kubernetes.io/projected/8b380568-48c5-4410-8ddc-3537d86ad229-kube-api-access-clp5z\") pod \"8b380568-48c5-4410-8ddc-3537d86ad229\" (UID: \"8b380568-48c5-4410-8ddc-3537d86ad229\") " Nov 01 21:10:21.065053 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:21.064891 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b380568-48c5-4410-8ddc-3537d86ad229-maas-api-openshift-ca" pod "8b380568-48c5-4410-8ddc-3537d86ad229" (UID: "8b380568-48c5-4410-8ddc-3537d86ad229"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:10:21.065178 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:21.065089 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69b1f9a7-bdcc-4a23-937e-07957ceca362-maas-api-openshift-ca" pod "69b1f9a7-bdcc-4a23-937e-07957ceca362" (UID: "69b1f9a7-bdcc-4a23-937e-07957ceca362"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:10:21.065471 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:21.065449 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69b1f9a7-bdcc-4a23-937e-07957ceca362-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:10:21.065597 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:21.065473 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b380568-48c5-4410-8ddc-3537d86ad229-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:10:21.067064 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:21.067016 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69b1f9a7-bdcc-4a23-937e-07957ceca362-kube-api-access-n7n92" pod "69b1f9a7-bdcc-4a23-937e-07957ceca362" (UID: "69b1f9a7-bdcc-4a23-937e-07957ceca362"). InnerVolumeSpecName "kube-api-access-n7n92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:10:21.067170 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:21.067137 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69b1f9a7-bdcc-4a23-937e-07957ceca362-maas-api-xks-ca" pod "69b1f9a7-bdcc-4a23-937e-07957ceca362" (UID: "69b1f9a7-bdcc-4a23-937e-07957ceca362"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:10:21.067225 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:21.067200 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b380568-48c5-4410-8ddc-3537d86ad229-maas-api-xks-ca" pod "8b380568-48c5-4410-8ddc-3537d86ad229" (UID: "8b380568-48c5-4410-8ddc-3537d86ad229"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:10:21.067286 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:21.067270 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b380568-48c5-4410-8ddc-3537d86ad229-kube-api-access-clp5z" pod "8b380568-48c5-4410-8ddc-3537d86ad229" (UID: "8b380568-48c5-4410-8ddc-3537d86ad229"). InnerVolumeSpecName "kube-api-access-clp5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:10:21.166443 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:21.166400 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b380568-48c5-4410-8ddc-3537d86ad229-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:10:21.166443 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:21.166435 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clp5z\" (UniqueName: \"kubernetes.io/projected/8b380568-48c5-4410-8ddc-3537d86ad229-kube-api-access-clp5z\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:10:21.166443 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:21.166446 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7n92\" (UniqueName: \"kubernetes.io/projected/69b1f9a7-bdcc-4a23-937e-07957ceca362-kube-api-access-n7n92\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:10:21.888130 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:21.166460 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69b1f9a7-bdcc-4a23-937e-07957ceca362-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:10:21.888130 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:21.768406 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74f8c1e-sdwfd" event={"ID":"8b380568-48c5-4410-8ddc-3537d86ad229","Type":"ContainerDied","Data":"359a851ecde11b0fc638530a3ffa0996966acf266b3a84f0cac03c013619bff5"} Nov 01 21:10:21.888130 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:21.768467 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="359a851ecde11b0fc638530a3ffa0996966acf266b3a84f0cac03c013619bff5" Nov 01 21:10:21.888130 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:21.768544 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74f8c1e-sdwfd" Nov 01 21:10:21.888130 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:21.771721 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9651f055-qb4vx" event={"ID":"69b1f9a7-bdcc-4a23-937e-07957ceca362","Type":"ContainerDied","Data":"3e21d1bced3cacc54dd83c4bdba107b7f1bb0df7e1990e262a7e6c8f1feb59ab"} Nov 01 21:10:21.888130 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:21.771777 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e21d1bced3cacc54dd83c4bdba107b7f1bb0df7e1990e262a7e6c8f1feb59ab" Nov 01 21:10:21.888130 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:21.771849 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9651f055-qb4vx" Nov 01 21:10:22.781645 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:22.781612 4484 generic.go:356] "Generic (PLEG): container finished" podID="297fe96b-76ad-499e-b9b7-f536d1851712" containerID="0bb6eab4f96a0a4c0b346f8fa050f8575cf18099997779b566985b9b7a370e1b" exitCode=0 Nov 01 21:10:22.782173 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:22.781687 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2" event={"ID":"297fe96b-76ad-499e-b9b7-f536d1851712","Type":"ContainerDied","Data":"0bb6eab4f96a0a4c0b346f8fa050f8575cf18099997779b566985b9b7a370e1b"} Nov 01 21:10:22.782264 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:22.782184 4484 scope.go:122] "RemoveContainer" containerID="0bb6eab4f96a0a4c0b346f8fa050f8575cf18099997779b566985b9b7a370e1b" Nov 01 21:10:22.783614 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:22.783588 4484 generic.go:356] "Generic (PLEG): container finished" podID="2d4f02fb-d74b-40e5-b470-f8d1e65b120c" containerID="ca2e831ab9700ed80a68b6bf74bd5f3b11c7f2b7bc066043094ff2be8b087e49" exitCode=0 Nov 01 21:10:22.783710 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:22.783634 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk" event={"ID":"2d4f02fb-d74b-40e5-b470-f8d1e65b120c","Type":"ContainerDied","Data":"ca2e831ab9700ed80a68b6bf74bd5f3b11c7f2b7bc066043094ff2be8b087e49"} Nov 01 21:10:22.784007 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:22.783983 4484 scope.go:122] "RemoveContainer" containerID="ca2e831ab9700ed80a68b6bf74bd5f3b11c7f2b7bc066043094ff2be8b087e49" Nov 01 21:10:22.784110 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:22.784085 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:10:22.784176 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:22.784163 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:10:22.784213 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:22.784201 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:10:22.785799 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:22.785762 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:10:22.785879 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:22.785852 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:10:22.785926 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:22.785910 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:10:23.004596 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:23.004563 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2" Nov 01 21:10:23.228166 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:23.228072 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:10:23.792965 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:23.792932 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2" event={"ID":"297fe96b-76ad-499e-b9b7-f536d1851712","Type":"ContainerStarted","Data":"c5598745e8a990a6fa1b4835b95fdb9aae63eeffbdac8f7b48ea236ade0a472f"} Nov 01 21:10:23.794855 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:23.794829 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk" event={"ID":"2d4f02fb-d74b-40e5-b470-f8d1e65b120c","Type":"ContainerStarted","Data":"bad8b92d50d5251531869bf9c2ff31eb420058dd763ac270d443dde84fbaec94"} Nov 01 21:10:24.734021 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:24.732449 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2" Nov 01 21:10:24.734021 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:24.732630 4484 patch_prober.go:29] interesting pod/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Nov 01 21:10:24.734021 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:24.732684 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2" podUID="297fe96b-76ad-499e-b9b7-f536d1851712" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Nov 01 21:10:24.780625 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:24.780590 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk" Nov 01 21:10:24.781474 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:24.781450 4484 patch_prober.go:29] interesting pod/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Nov 01 21:10:24.781567 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:24.781498 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk" podUID="2d4f02fb-d74b-40e5-b470-f8d1e65b120c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Nov 01 21:10:25.373533 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:25.373499 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk" Nov 01 21:10:25.735075 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:25.731430 4484 patch_prober.go:29] interesting pod/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Nov 01 21:10:25.735075 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:25.731496 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2" podUID="297fe96b-76ad-499e-b9b7-f536d1851712" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Nov 01 21:10:25.781218 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:25.781186 4484 patch_prober.go:29] interesting pod/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Nov 01 21:10:25.781391 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:25.781237 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk" podUID="2d4f02fb-d74b-40e5-b470-f8d1e65b120c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Nov 01 21:10:26.441019 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:26.440296 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:10:26.600973 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:26.600938 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:10:26.742077 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:26.731440 4484 patch_prober.go:29] interesting pod/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Nov 01 21:10:26.742077 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:26.731542 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2" podUID="297fe96b-76ad-499e-b9b7-f536d1851712" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Nov 01 21:10:26.781458 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:26.781413 4484 patch_prober.go:29] interesting pod/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Nov 01 21:10:26.781662 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:26.781479 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk" podUID="2d4f02fb-d74b-40e5-b470-f8d1e65b120c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Nov 01 21:10:27.735019 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:27.734489 4484 patch_prober.go:29] interesting pod/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Nov 01 21:10:27.735019 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:27.734590 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2" podUID="297fe96b-76ad-499e-b9b7-f536d1851712" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Nov 01 21:10:27.781416 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:27.781378 4484 patch_prober.go:29] interesting pod/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Nov 01 21:10:27.781598 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:27.781437 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk" podUID="2d4f02fb-d74b-40e5-b470-f8d1e65b120c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Nov 01 21:10:28.035019 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:28.034653 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:10:28.732520 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:28.732477 4484 patch_prober.go:29] interesting pod/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Nov 01 21:10:28.732834 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:28.732804 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2" podUID="297fe96b-76ad-499e-b9b7-f536d1851712" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Nov 01 21:10:28.781332 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:28.781292 4484 patch_prober.go:29] interesting pod/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Nov 01 21:10:28.781735 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:28.781367 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk" podUID="2d4f02fb-d74b-40e5-b470-f8d1e65b120c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Nov 01 21:10:29.735667 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:29.735631 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2" Nov 01 21:10:29.736123 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:29.736108 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2" Nov 01 21:10:29.736890 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:29.736871 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2" Nov 01 21:10:29.785837 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:29.785809 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk" Nov 01 21:10:29.786706 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:29.786688 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk" Nov 01 21:10:32.719067 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:32.718940 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:10:32.883422 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:32.879139 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:10:37.463777 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:37.463740 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:10:37.623241 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:37.623205 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:10:37.770750 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:37.770715 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:10:42.366347 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:42.366311 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:10:45.769443 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:10:45.769380 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:10:46.609058 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:46.608989 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:10:47.244942 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:47.244905 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:10:49.564048 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:49.564013 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:10:49.722294 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:49.722261 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:10:52.072886 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:52.072843 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:10:52.234731 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:52.234689 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:10:56.948467 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:10:56.948434 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:11:00.774182 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:00.774142 4484 scope.go:122] "RemoveContainer" containerID="134b52c382ff6c88f8c8b84568cb9cef76e0c4e878e6c7e02e62681a00c2422d" Nov 01 21:11:00.785034 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:00.784984 4484 scope.go:122] "RemoveContainer" containerID="58d73a0440a359ed30a4ac5c2e263fbce0a3143374d24f4eb3417f45464410b4" Nov 01 21:11:00.794983 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:00.794958 4484 scope.go:122] "RemoveContainer" containerID="d5c96cc4385de59c41be7420f068f3d4b5de59e9b76a9163c079eac500b05558" Nov 01 21:11:00.804353 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:00.804336 4484 scope.go:122] "RemoveContainer" containerID="90f190f7772e0c7f1ae7ca1c67437b5a9846a7c9039bac9872c2c9ed328a0312" Nov 01 21:11:00.814220 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:00.814202 4484 scope.go:122] "RemoveContainer" containerID="aaf96b0df739980fb14b5ec06637a2f9906b4ba05cc3c4ad9982d94dc25e51ad" Nov 01 21:11:01.664776 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:01.664743 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:11:06.433468 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:06.433428 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:11:08.706453 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:08.706414 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:11:10.059426 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:10.059395 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e0a6f5-4tvj9"] Nov 01 21:11:10.060189 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:10.060174 4484 cpu_manager.go:414] "containerMap: removing container" podUID="8b380568-48c5-4410-8ddc-3537d86ad229" containerName="revoke-keys" Nov 01 21:11:10.060246 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:10.060189 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b380568-48c5-4410-8ddc-3537d86ad229" containerName="revoke-keys" Nov 01 21:11:10.060246 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:10.060199 4484 cpu_manager.go:414] "containerMap: removing container" podUID="69b1f9a7-bdcc-4a23-937e-07957ceca362" containerName="revoke-keys" Nov 01 21:11:10.060246 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:10.060203 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69b1f9a7-bdcc-4a23-937e-07957ceca362" containerName="revoke-keys" Nov 01 21:11:10.060353 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:10.060306 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b380568-48c5-4410-8ddc-3537d86ad229" containerName="revoke-keys" Nov 01 21:11:10.060353 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:10.060313 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="69b1f9a7-bdcc-4a23-937e-07957ceca362" containerName="revoke-keys" Nov 01 21:11:10.067337 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:10.067319 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e0a6f5-4tvj9" Nov 01 21:11:10.131018 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:10.130858 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a0adb84-65af-4381-bbea-c4ed879ac334-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e0a6f5-4tvj9\" (UID: \"3a0adb84-65af-4381-bbea-c4ed879ac334\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e0a6f5-4tvj9" Nov 01 21:11:10.131229 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:10.131145 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a0adb84-65af-4381-bbea-c4ed879ac334-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e0a6f5-4tvj9\" (UID: \"3a0adb84-65af-4381-bbea-c4ed879ac334\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e0a6f5-4tvj9" Nov 01 21:11:10.131296 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:10.131239 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kwsl\" (UniqueName: \"kubernetes.io/projected/3a0adb84-65af-4381-bbea-c4ed879ac334-kube-api-access-2kwsl\") pod \"maas-api-revoke-keys-e2e-api-a-e0a6f5-4tvj9\" (UID: \"3a0adb84-65af-4381-bbea-c4ed879ac334\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e0a6f5-4tvj9" Nov 01 21:11:10.133308 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:10.133276 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e0a6f5-4tvj9"] Nov 01 21:11:10.175326 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:10.175284 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:11:10.175326 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:10.175284 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:11:10.175326 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:10.175294 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:11:10.175606 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:10.175344 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:11:10.232761 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:10.232718 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kwsl\" (UniqueName: \"kubernetes.io/projected/3a0adb84-65af-4381-bbea-c4ed879ac334-kube-api-access-2kwsl\") pod \"maas-api-revoke-keys-e2e-api-a-e0a6f5-4tvj9\" (UID: \"3a0adb84-65af-4381-bbea-c4ed879ac334\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e0a6f5-4tvj9" Nov 01 21:11:10.232949 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:10.232862 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a0adb84-65af-4381-bbea-c4ed879ac334-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e0a6f5-4tvj9\" (UID: \"3a0adb84-65af-4381-bbea-c4ed879ac334\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e0a6f5-4tvj9" Nov 01 21:11:10.233114 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:10.233096 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a0adb84-65af-4381-bbea-c4ed879ac334-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e0a6f5-4tvj9\" (UID: \"3a0adb84-65af-4381-bbea-c4ed879ac334\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e0a6f5-4tvj9" Nov 01 21:11:10.233547 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:10.233521 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a0adb84-65af-4381-bbea-c4ed879ac334-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e0a6f5-4tvj9\" (UID: \"3a0adb84-65af-4381-bbea-c4ed879ac334\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e0a6f5-4tvj9" Nov 01 21:11:10.235890 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:10.235860 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a0adb84-65af-4381-bbea-c4ed879ac334-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e0a6f5-4tvj9\" (UID: \"3a0adb84-65af-4381-bbea-c4ed879ac334\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e0a6f5-4tvj9" Nov 01 21:11:10.290592 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:10.290553 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kwsl\" (UniqueName: \"kubernetes.io/projected/3a0adb84-65af-4381-bbea-c4ed879ac334-kube-api-access-2kwsl\") pod \"maas-api-revoke-keys-e2e-api-a-e0a6f5-4tvj9\" (UID: \"3a0adb84-65af-4381-bbea-c4ed879ac334\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e0a6f5-4tvj9" Nov 01 21:11:10.382860 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:10.382777 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e0a6f5-4tvj9" Nov 01 21:11:10.790230 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:10.790198 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e0a6f5-4tvj9"] Nov 01 21:11:10.790643 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:11:10.790618 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a0adb84_65af_4381_bbea_c4ed879ac334.slice/crio-6927bd812e99104512201972b5d1e2d1dc419f7c2b2a19aabe11cfb585d6fc30 WatchSource:0}: Error finding container 6927bd812e99104512201972b5d1e2d1dc419f7c2b2a19aabe11cfb585d6fc30: Status 404 returned error can't find the container with id 6927bd812e99104512201972b5d1e2d1dc419f7c2b2a19aabe11cfb585d6fc30 Nov 01 21:11:11.246835 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:11.246805 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e0a6f5-4tvj9" event={"ID":"3a0adb84-65af-4381-bbea-c4ed879ac334","Type":"ContainerStarted","Data":"1a1ff6ff2d01bc10df37e5c5f158dd1424bee557006b80d116df54938fd33e56"} Nov 01 21:11:11.247285 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:11.246872 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e0a6f5-4tvj9" event={"ID":"3a0adb84-65af-4381-bbea-c4ed879ac334","Type":"ContainerStarted","Data":"6927bd812e99104512201972b5d1e2d1dc419f7c2b2a19aabe11cfb585d6fc30"} Nov 01 21:11:11.360911 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:11.360801 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e0a6f5-4tvj9" podStartSLOduration=1.3607873160000001 podStartE2EDuration="1.360787316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:11:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:11:11.360241528 +0000 UTC m=+2422.137881663" watchObservedRunningTime="2026-11-01 21:11:11.360787316 +0000 UTC m=+2422.138427489" Nov 01 21:11:11.730586 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:11.730446 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:11:12.255696 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:12.255661 4484 generic.go:356] "Generic (PLEG): container finished" podID="3a0adb84-65af-4381-bbea-c4ed879ac334" containerID="1a1ff6ff2d01bc10df37e5c5f158dd1424bee557006b80d116df54938fd33e56" exitCode=0 Nov 01 21:11:12.256111 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:12.255702 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e0a6f5-4tvj9" event={"ID":"3a0adb84-65af-4381-bbea-c4ed879ac334","Type":"ContainerDied","Data":"1a1ff6ff2d01bc10df37e5c5f158dd1424bee557006b80d116df54938fd33e56"} Nov 01 21:11:13.490058 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:13.490033 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e0a6f5-4tvj9" Nov 01 21:11:13.602142 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:13.602029 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a0adb84-65af-4381-bbea-c4ed879ac334-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a0adb84-65af-4381-bbea-c4ed879ac334-maas-api-openshift-ca\") pod \"3a0adb84-65af-4381-bbea-c4ed879ac334\" (UID: \"3a0adb84-65af-4381-bbea-c4ed879ac334\") " Nov 01 21:11:13.602325 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:13.602238 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a0adb84-65af-4381-bbea-c4ed879ac334-kube-api-access-2kwsl\" (UniqueName: \"kubernetes.io/projected/3a0adb84-65af-4381-bbea-c4ed879ac334-kube-api-access-2kwsl\") pod \"3a0adb84-65af-4381-bbea-c4ed879ac334\" (UID: \"3a0adb84-65af-4381-bbea-c4ed879ac334\") " Nov 01 21:11:13.602325 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:13.602289 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a0adb84-65af-4381-bbea-c4ed879ac334-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a0adb84-65af-4381-bbea-c4ed879ac334-maas-api-xks-ca\") pod \"3a0adb84-65af-4381-bbea-c4ed879ac334\" (UID: \"3a0adb84-65af-4381-bbea-c4ed879ac334\") " Nov 01 21:11:13.602457 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:13.602390 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a0adb84-65af-4381-bbea-c4ed879ac334-maas-api-openshift-ca" pod "3a0adb84-65af-4381-bbea-c4ed879ac334" (UID: "3a0adb84-65af-4381-bbea-c4ed879ac334"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:11:13.602804 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:13.602783 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a0adb84-65af-4381-bbea-c4ed879ac334-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:11:13.626165 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:13.626124 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a0adb84-65af-4381-bbea-c4ed879ac334-maas-api-xks-ca" pod "3a0adb84-65af-4381-bbea-c4ed879ac334" (UID: "3a0adb84-65af-4381-bbea-c4ed879ac334"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:11:13.637266 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:13.631545 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a0adb84-65af-4381-bbea-c4ed879ac334-kube-api-access-2kwsl" pod "3a0adb84-65af-4381-bbea-c4ed879ac334" (UID: "3a0adb84-65af-4381-bbea-c4ed879ac334"). InnerVolumeSpecName "kube-api-access-2kwsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:11:13.728224 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:13.728181 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kwsl\" (UniqueName: \"kubernetes.io/projected/3a0adb84-65af-4381-bbea-c4ed879ac334-kube-api-access-2kwsl\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:11:13.728224 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:13.728221 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a0adb84-65af-4381-bbea-c4ed879ac334-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:11:14.272575 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:14.272545 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e0a6f5-4tvj9" event={"ID":"3a0adb84-65af-4381-bbea-c4ed879ac334","Type":"ContainerDied","Data":"6927bd812e99104512201972b5d1e2d1dc419f7c2b2a19aabe11cfb585d6fc30"} Nov 01 21:11:14.272859 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:14.272836 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6927bd812e99104512201972b5d1e2d1dc419f7c2b2a19aabe11cfb585d6fc30" Nov 01 21:11:14.272859 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:14.272574 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e0a6f5-4tvj9" Nov 01 21:11:15.768378 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:11:15.768342 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:11:16.931610 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:16.931575 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:11:18.544921 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:18.544878 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:11:19.312781 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:19.312749 4484 generic.go:356] "Generic (PLEG): container finished" podID="8a5b2e73-9f47-47aa-9ed7-c75d17b1f658" containerID="4696272076f0d71ea68d0e91f545e2f24273c5a522684fc28be907d79aab81f7" exitCode=0 Nov 01 21:11:19.313055 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:19.312829 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk" event={"ID":"8a5b2e73-9f47-47aa-9ed7-c75d17b1f658","Type":"ContainerDied","Data":"4696272076f0d71ea68d0e91f545e2f24273c5a522684fc28be907d79aab81f7"} Nov 01 21:11:19.313341 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:19.313323 4484 scope.go:122] "RemoveContainer" containerID="4696272076f0d71ea68d0e91f545e2f24273c5a522684fc28be907d79aab81f7" Nov 01 21:11:19.314949 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:19.314919 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:11:19.315027 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:19.315008 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:11:19.315072 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:19.315043 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:11:20.339055 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:20.339012 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk" event={"ID":"8a5b2e73-9f47-47aa-9ed7-c75d17b1f658","Type":"ContainerStarted","Data":"40529fb5a9715b9494e878075c25a0959062796221255c40dfd8cac30095adee"} Nov 01 21:11:20.641721 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:20.641634 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk" Nov 01 21:11:20.641887 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:20.641798 4484 patch_prober.go:29] interesting pod/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body= Nov 01 21:11:20.641887 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:20.641842 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk" podUID="8a5b2e73-9f47-47aa-9ed7-c75d17b1f658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" Nov 01 21:11:21.555624 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:21.555587 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e0a6f5-56bbd4f6f7-5gtlm"] Nov 01 21:11:21.641689 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:21.641652 4484 patch_prober.go:29] interesting pod/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body= Nov 01 21:11:21.641889 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:21.641717 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk" podUID="8a5b2e73-9f47-47aa-9ed7-c75d17b1f658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" Nov 01 21:11:21.680115 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:21.680076 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e0a6f5-68b97cb76c-sm88c"] Nov 01 21:11:21.941214 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:21.941137 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:11:22.126726 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:22.126682 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:11:22.640917 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:22.640885 4484 patch_prober.go:29] interesting pod/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body= Nov 01 21:11:22.641438 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:22.640948 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk" podUID="8a5b2e73-9f47-47aa-9ed7-c75d17b1f658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" Nov 01 21:11:23.640905 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:23.640872 4484 patch_prober.go:29] interesting pod/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body= Nov 01 21:11:23.641107 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:23.640934 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk" podUID="8a5b2e73-9f47-47aa-9ed7-c75d17b1f658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" Nov 01 21:11:24.642152 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:24.642118 4484 patch_prober.go:29] interesting pod/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body= Nov 01 21:11:24.642618 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:24.642173 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk" podUID="8a5b2e73-9f47-47aa-9ed7-c75d17b1f658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" Nov 01 21:11:25.641721 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:25.641678 4484 patch_prober.go:29] interesting pod/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body= Nov 01 21:11:25.641919 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:25.641760 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk" podUID="8a5b2e73-9f47-47aa-9ed7-c75d17b1f658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" Nov 01 21:11:26.556831 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:26.556780 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-e0a6f5-56bbd4f6f7-5gtlm" podUID="6c9c289a-bbe6-4ef2-a11d-e87f0bf77480" containerName="payload-processing" containerID="cri-o://d5df541a80bf0a13bf7e56fcee990f1f157ff469a0a25f14e468a9c0ca089688" gracePeriod=25 Nov 01 21:11:26.691017 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:26.682065 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e0a6f5-68b97cb76c-sm88c" podUID="a007bf8c-6aab-4b78-87eb-3bbd63998634" containerName="payload-pre-processing" containerID="cri-o://d83793fddda42fd98e1a8a16a6b71e745cf300461144ae7582a42c4680b3fe13" gracePeriod=25 Nov 01 21:11:26.779037 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:26.769850 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk" Nov 01 21:11:26.779037 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:26.772134 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk" Nov 01 21:11:26.779037 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:26.773338 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk" Nov 01 21:11:26.967675 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:26.967640 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e0a6f5-56bbd4f6f7-5gtlm" Nov 01 21:11:27.070670 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:27.070602 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c9c289a-bbe6-4ef2-a11d-e87f0bf77480-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c9c289a-bbe6-4ef2-a11d-e87f0bf77480-plugins-config-volume\") pod \"6c9c289a-bbe6-4ef2-a11d-e87f0bf77480\" (UID: \"6c9c289a-bbe6-4ef2-a11d-e87f0bf77480\") " Nov 01 21:11:27.071053 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:27.071017 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c9c289a-bbe6-4ef2-a11d-e87f0bf77480-kube-api-access-f8m5g\" (UniqueName: \"kubernetes.io/projected/6c9c289a-bbe6-4ef2-a11d-e87f0bf77480-kube-api-access-f8m5g\") pod \"6c9c289a-bbe6-4ef2-a11d-e87f0bf77480\" (UID: \"6c9c289a-bbe6-4ef2-a11d-e87f0bf77480\") " Nov 01 21:11:27.072594 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:27.072561 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c9c289a-bbe6-4ef2-a11d-e87f0bf77480-plugins-config-volume" pod "6c9c289a-bbe6-4ef2-a11d-e87f0bf77480" (UID: "6c9c289a-bbe6-4ef2-a11d-e87f0bf77480"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:11:27.081646 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:27.081606 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c9c289a-bbe6-4ef2-a11d-e87f0bf77480-kube-api-access-f8m5g" pod "6c9c289a-bbe6-4ef2-a11d-e87f0bf77480" (UID: "6c9c289a-bbe6-4ef2-a11d-e87f0bf77480"). InnerVolumeSpecName "kube-api-access-f8m5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:11:27.104829 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:27.104798 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e0a6f5-68b97cb76c-sm88c" Nov 01 21:11:27.177269 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:27.173534 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a007bf8c-6aab-4b78-87eb-3bbd63998634-kube-api-access-cs5fr\" (UniqueName: \"kubernetes.io/projected/a007bf8c-6aab-4b78-87eb-3bbd63998634-kube-api-access-cs5fr\") pod \"a007bf8c-6aab-4b78-87eb-3bbd63998634\" (UID: \"a007bf8c-6aab-4b78-87eb-3bbd63998634\") " Nov 01 21:11:27.177269 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:27.173911 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a007bf8c-6aab-4b78-87eb-3bbd63998634-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a007bf8c-6aab-4b78-87eb-3bbd63998634-plugins-config-volume\") pod \"a007bf8c-6aab-4b78-87eb-3bbd63998634\" (UID: \"a007bf8c-6aab-4b78-87eb-3bbd63998634\") " Nov 01 21:11:27.177269 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:27.176195 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c9c289a-bbe6-4ef2-a11d-e87f0bf77480-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:11:27.177269 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:27.176222 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8m5g\" (UniqueName: \"kubernetes.io/projected/6c9c289a-bbe6-4ef2-a11d-e87f0bf77480-kube-api-access-f8m5g\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:11:27.177269 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:27.176558 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a007bf8c-6aab-4b78-87eb-3bbd63998634-plugins-config-volume" pod "a007bf8c-6aab-4b78-87eb-3bbd63998634" (UID: "a007bf8c-6aab-4b78-87eb-3bbd63998634"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:11:27.180043 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:27.179922 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a007bf8c-6aab-4b78-87eb-3bbd63998634-kube-api-access-cs5fr" pod "a007bf8c-6aab-4b78-87eb-3bbd63998634" (UID: "a007bf8c-6aab-4b78-87eb-3bbd63998634"). InnerVolumeSpecName "kube-api-access-cs5fr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:11:27.277830 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:27.277793 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs5fr\" (UniqueName: \"kubernetes.io/projected/a007bf8c-6aab-4b78-87eb-3bbd63998634-kube-api-access-cs5fr\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:11:27.277830 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:27.277826 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a007bf8c-6aab-4b78-87eb-3bbd63998634-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:11:27.400896 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:27.400860 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:11:27.504855 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:27.504818 4484 generic.go:356] "Generic (PLEG): container finished" podID="a007bf8c-6aab-4b78-87eb-3bbd63998634" containerID="d83793fddda42fd98e1a8a16a6b71e745cf300461144ae7582a42c4680b3fe13" exitCode=0 Nov 01 21:11:27.505092 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:27.504869 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e0a6f5-68b97cb76c-sm88c" Nov 01 21:11:27.505092 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:27.504900 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e0a6f5-68b97cb76c-sm88c" event={"ID":"a007bf8c-6aab-4b78-87eb-3bbd63998634","Type":"ContainerDied","Data":"d83793fddda42fd98e1a8a16a6b71e745cf300461144ae7582a42c4680b3fe13"} Nov 01 21:11:27.505092 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:27.504954 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e0a6f5-68b97cb76c-sm88c" event={"ID":"a007bf8c-6aab-4b78-87eb-3bbd63998634","Type":"ContainerDied","Data":"71e92f302e0b0180b25871ddcdb105db6578c8e46954cdb7970c66b6a2e0755b"} Nov 01 21:11:27.505092 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:27.504976 4484 scope.go:122] "RemoveContainer" containerID="d83793fddda42fd98e1a8a16a6b71e745cf300461144ae7582a42c4680b3fe13" Nov 01 21:11:27.506676 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:27.506652 4484 generic.go:356] "Generic (PLEG): container finished" podID="6c9c289a-bbe6-4ef2-a11d-e87f0bf77480" containerID="d5df541a80bf0a13bf7e56fcee990f1f157ff469a0a25f14e468a9c0ca089688" exitCode=0 Nov 01 21:11:27.506836 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:27.506702 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e0a6f5-56bbd4f6f7-5gtlm" Nov 01 21:11:27.506836 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:27.506704 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e0a6f5-56bbd4f6f7-5gtlm" event={"ID":"6c9c289a-bbe6-4ef2-a11d-e87f0bf77480","Type":"ContainerDied","Data":"d5df541a80bf0a13bf7e56fcee990f1f157ff469a0a25f14e468a9c0ca089688"} Nov 01 21:11:27.506836 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:27.506761 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e0a6f5-56bbd4f6f7-5gtlm" event={"ID":"6c9c289a-bbe6-4ef2-a11d-e87f0bf77480","Type":"ContainerDied","Data":"ab5ca9655fac3228cc480b0ebe6e63a5f80b36b22630b15028e22f168bcd9cda"} Nov 01 21:11:27.516426 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:27.516400 4484 scope.go:122] "RemoveContainer" containerID="d83793fddda42fd98e1a8a16a6b71e745cf300461144ae7582a42c4680b3fe13" Nov 01 21:11:27.516748 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:11:27.516728 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d83793fddda42fd98e1a8a16a6b71e745cf300461144ae7582a42c4680b3fe13\": container with ID starting with d83793fddda42fd98e1a8a16a6b71e745cf300461144ae7582a42c4680b3fe13 not found: ID does not exist" containerID="d83793fddda42fd98e1a8a16a6b71e745cf300461144ae7582a42c4680b3fe13" Nov 01 21:11:27.516811 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:27.516758 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d83793fddda42fd98e1a8a16a6b71e745cf300461144ae7582a42c4680b3fe13"} err="failed to get container status \"d83793fddda42fd98e1a8a16a6b71e745cf300461144ae7582a42c4680b3fe13\": rpc error: code = NotFound desc = could not find container \"d83793fddda42fd98e1a8a16a6b71e745cf300461144ae7582a42c4680b3fe13\": container with ID starting with d83793fddda42fd98e1a8a16a6b71e745cf300461144ae7582a42c4680b3fe13 not found: ID does not exist" Nov 01 21:11:27.516811 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:27.516776 4484 scope.go:122] "RemoveContainer" containerID="d5df541a80bf0a13bf7e56fcee990f1f157ff469a0a25f14e468a9c0ca089688" Nov 01 21:11:27.526578 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:27.526553 4484 scope.go:122] "RemoveContainer" containerID="d5df541a80bf0a13bf7e56fcee990f1f157ff469a0a25f14e468a9c0ca089688" Nov 01 21:11:27.526896 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:11:27.526875 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5df541a80bf0a13bf7e56fcee990f1f157ff469a0a25f14e468a9c0ca089688\": container with ID starting with d5df541a80bf0a13bf7e56fcee990f1f157ff469a0a25f14e468a9c0ca089688 not found: ID does not exist" containerID="d5df541a80bf0a13bf7e56fcee990f1f157ff469a0a25f14e468a9c0ca089688" Nov 01 21:11:27.526958 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:27.526906 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5df541a80bf0a13bf7e56fcee990f1f157ff469a0a25f14e468a9c0ca089688"} err="failed to get container status \"d5df541a80bf0a13bf7e56fcee990f1f157ff469a0a25f14e468a9c0ca089688\": rpc error: code = NotFound desc = could not find container \"d5df541a80bf0a13bf7e56fcee990f1f157ff469a0a25f14e468a9c0ca089688\": container with ID starting with d5df541a80bf0a13bf7e56fcee990f1f157ff469a0a25f14e468a9c0ca089688 not found: ID does not exist" Nov 01 21:11:27.673240 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:27.673152 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e0a6f5-68b97cb76c-sm88c"] Nov 01 21:11:27.679377 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:27.679349 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e0a6f5-68b97cb76c-sm88c"] Nov 01 21:11:27.788766 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:27.788733 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e0a6f5-56bbd4f6f7-5gtlm"] Nov 01 21:11:27.792432 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:27.792408 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e0a6f5-56bbd4f6f7-5gtlm"] Nov 01 21:11:29.374936 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:29.374903 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c9c289a-bbe6-4ef2-a11d-e87f0bf77480" path="/var/lib/kubelet/pods/6c9c289a-bbe6-4ef2-a11d-e87f0bf77480/volumes" Nov 01 21:11:29.375424 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:29.375378 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a007bf8c-6aab-4b78-87eb-3bbd63998634" path="/var/lib/kubelet/pods/a007bf8c-6aab-4b78-87eb-3bbd63998634/volumes" Nov 01 21:11:29.846348 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:29.846315 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3ab520-jfqg5"] Nov 01 21:11:29.847116 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:29.847101 4484 cpu_manager.go:414] "containerMap: removing container" podUID="a007bf8c-6aab-4b78-87eb-3bbd63998634" containerName="payload-pre-processing" Nov 01 21:11:29.847184 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:29.847117 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a007bf8c-6aab-4b78-87eb-3bbd63998634" containerName="payload-pre-processing" Nov 01 21:11:29.847184 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:29.847136 4484 cpu_manager.go:414] "containerMap: removing container" podUID="3a0adb84-65af-4381-bbea-c4ed879ac334" containerName="revoke-keys" Nov 01 21:11:29.847184 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:29.847141 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a0adb84-65af-4381-bbea-c4ed879ac334" containerName="revoke-keys" Nov 01 21:11:29.847184 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:29.847158 4484 cpu_manager.go:414] "containerMap: removing container" podUID="6c9c289a-bbe6-4ef2-a11d-e87f0bf77480" containerName="payload-processing" Nov 01 21:11:29.847184 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:29.847163 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c9c289a-bbe6-4ef2-a11d-e87f0bf77480" containerName="payload-processing" Nov 01 21:11:29.847353 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:29.847281 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="a007bf8c-6aab-4b78-87eb-3bbd63998634" containerName="payload-pre-processing" Nov 01 21:11:29.847353 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:29.847292 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a0adb84-65af-4381-bbea-c4ed879ac334" containerName="revoke-keys" Nov 01 21:11:29.847353 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:29.847302 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c9c289a-bbe6-4ef2-a11d-e87f0bf77480" containerName="payload-processing" Nov 01 21:11:29.850657 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:29.850640 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3ab520-jfqg5" Nov 01 21:11:29.907305 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:29.907270 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3ab520-jfqg5"] Nov 01 21:11:29.913760 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:29.913730 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a19f1a9a-b9d2-4ee1-8336-4425230b87b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3ab520-jfqg5\" (UID: \"a19f1a9a-b9d2-4ee1-8336-4425230b87b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3ab520-jfqg5" Nov 01 21:11:29.913938 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:29.913782 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28xdn\" (UniqueName: \"kubernetes.io/projected/a19f1a9a-b9d2-4ee1-8336-4425230b87b0-kube-api-access-28xdn\") pod \"maas-api-revoke-keys-e2e-inf-a-3ab520-jfqg5\" (UID: \"a19f1a9a-b9d2-4ee1-8336-4425230b87b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3ab520-jfqg5" Nov 01 21:11:29.914051 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:29.913944 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a19f1a9a-b9d2-4ee1-8336-4425230b87b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3ab520-jfqg5\" (UID: \"a19f1a9a-b9d2-4ee1-8336-4425230b87b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3ab520-jfqg5" Nov 01 21:11:29.953290 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:29.953237 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:11:29.953290 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:29.953269 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:11:29.953290 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:29.953244 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:11:29.953537 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:29.953314 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:11:30.015761 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:30.015726 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a19f1a9a-b9d2-4ee1-8336-4425230b87b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3ab520-jfqg5\" (UID: \"a19f1a9a-b9d2-4ee1-8336-4425230b87b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3ab520-jfqg5" Nov 01 21:11:30.015945 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:30.015777 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28xdn\" (UniqueName: \"kubernetes.io/projected/a19f1a9a-b9d2-4ee1-8336-4425230b87b0-kube-api-access-28xdn\") pod \"maas-api-revoke-keys-e2e-inf-a-3ab520-jfqg5\" (UID: \"a19f1a9a-b9d2-4ee1-8336-4425230b87b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3ab520-jfqg5" Nov 01 21:11:30.015945 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:30.015819 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a19f1a9a-b9d2-4ee1-8336-4425230b87b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3ab520-jfqg5\" (UID: \"a19f1a9a-b9d2-4ee1-8336-4425230b87b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3ab520-jfqg5" Nov 01 21:11:30.016531 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:30.016501 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a19f1a9a-b9d2-4ee1-8336-4425230b87b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3ab520-jfqg5\" (UID: \"a19f1a9a-b9d2-4ee1-8336-4425230b87b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3ab520-jfqg5" Nov 01 21:11:30.018381 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:30.018364 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a19f1a9a-b9d2-4ee1-8336-4425230b87b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3ab520-jfqg5\" (UID: \"a19f1a9a-b9d2-4ee1-8336-4425230b87b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3ab520-jfqg5" Nov 01 21:11:30.072902 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:30.072862 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28xdn\" (UniqueName: \"kubernetes.io/projected/a19f1a9a-b9d2-4ee1-8336-4425230b87b0-kube-api-access-28xdn\") pod \"maas-api-revoke-keys-e2e-inf-a-3ab520-jfqg5\" (UID: \"a19f1a9a-b9d2-4ee1-8336-4425230b87b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3ab520-jfqg5" Nov 01 21:11:30.163839 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:30.163752 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3ab520-jfqg5" Nov 01 21:11:30.546553 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:11:30.546525 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda19f1a9a_b9d2_4ee1_8336_4425230b87b0.slice/crio-9c60bd05c02526e3e271463bfae760dab570a05d6c08c342f7fc582fc0ef4517 WatchSource:0}: Error finding container 9c60bd05c02526e3e271463bfae760dab570a05d6c08c342f7fc582fc0ef4517: Status 404 returned error can't find the container with id 9c60bd05c02526e3e271463bfae760dab570a05d6c08c342f7fc582fc0ef4517 Nov 01 21:11:30.546944 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:30.546917 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3ab520-jfqg5"] Nov 01 21:11:31.542332 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:31.542292 4484 generic.go:356] "Generic (PLEG): container finished" podID="a19f1a9a-b9d2-4ee1-8336-4425230b87b0" containerID="f6043b38f21aee17b7bb3e2955efccd8f3fab033b9c70ab7834d63c3ce9988da" exitCode=0 Nov 01 21:11:31.542541 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:31.542329 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3ab520-jfqg5" event={"ID":"a19f1a9a-b9d2-4ee1-8336-4425230b87b0","Type":"ContainerDied","Data":"f6043b38f21aee17b7bb3e2955efccd8f3fab033b9c70ab7834d63c3ce9988da"} Nov 01 21:11:31.542541 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:31.542390 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3ab520-jfqg5" event={"ID":"a19f1a9a-b9d2-4ee1-8336-4425230b87b0","Type":"ContainerStarted","Data":"9c60bd05c02526e3e271463bfae760dab570a05d6c08c342f7fc582fc0ef4517"} Nov 01 21:11:32.676191 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:32.676154 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:11:32.746344 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:32.746307 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3ab520-jfqg5" Nov 01 21:11:32.855724 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:32.855629 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a19f1a9a-b9d2-4ee1-8336-4425230b87b0-kube-api-access-28xdn\" (UniqueName: \"kubernetes.io/projected/a19f1a9a-b9d2-4ee1-8336-4425230b87b0-kube-api-access-28xdn\") pod \"a19f1a9a-b9d2-4ee1-8336-4425230b87b0\" (UID: \"a19f1a9a-b9d2-4ee1-8336-4425230b87b0\") " Nov 01 21:11:32.855724 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:32.855674 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a19f1a9a-b9d2-4ee1-8336-4425230b87b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a19f1a9a-b9d2-4ee1-8336-4425230b87b0-maas-api-openshift-ca\") pod \"a19f1a9a-b9d2-4ee1-8336-4425230b87b0\" (UID: \"a19f1a9a-b9d2-4ee1-8336-4425230b87b0\") " Nov 01 21:11:32.855724 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:32.855701 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a19f1a9a-b9d2-4ee1-8336-4425230b87b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a19f1a9a-b9d2-4ee1-8336-4425230b87b0-maas-api-xks-ca\") pod \"a19f1a9a-b9d2-4ee1-8336-4425230b87b0\" (UID: \"a19f1a9a-b9d2-4ee1-8336-4425230b87b0\") " Nov 01 21:11:32.856158 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:32.856129 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a19f1a9a-b9d2-4ee1-8336-4425230b87b0-maas-api-openshift-ca" pod "a19f1a9a-b9d2-4ee1-8336-4425230b87b0" (UID: "a19f1a9a-b9d2-4ee1-8336-4425230b87b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:11:32.856381 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:32.856366 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a19f1a9a-b9d2-4ee1-8336-4425230b87b0-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:11:32.857841 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:32.857815 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a19f1a9a-b9d2-4ee1-8336-4425230b87b0-maas-api-xks-ca" pod "a19f1a9a-b9d2-4ee1-8336-4425230b87b0" (UID: "a19f1a9a-b9d2-4ee1-8336-4425230b87b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:11:32.857940 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:32.857854 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a19f1a9a-b9d2-4ee1-8336-4425230b87b0-kube-api-access-28xdn" pod "a19f1a9a-b9d2-4ee1-8336-4425230b87b0" (UID: "a19f1a9a-b9d2-4ee1-8336-4425230b87b0"). InnerVolumeSpecName "kube-api-access-28xdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:11:32.957396 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:32.957355 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28xdn\" (UniqueName: \"kubernetes.io/projected/a19f1a9a-b9d2-4ee1-8336-4425230b87b0-kube-api-access-28xdn\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:11:32.957396 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:32.957390 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a19f1a9a-b9d2-4ee1-8336-4425230b87b0-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:11:33.566083 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:33.566041 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3ab520-jfqg5" event={"ID":"a19f1a9a-b9d2-4ee1-8336-4425230b87b0","Type":"ContainerDied","Data":"9c60bd05c02526e3e271463bfae760dab570a05d6c08c342f7fc582fc0ef4517"} Nov 01 21:11:33.566331 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:33.566125 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3ab520-jfqg5" Nov 01 21:11:33.566493 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:33.566399 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c60bd05c02526e3e271463bfae760dab570a05d6c08c342f7fc582fc0ef4517" Nov 01 21:11:36.058436 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:36.058401 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44594b37-7m92d"] Nov 01 21:11:36.061978 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:36.061952 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c004f1b-mfht8"] Nov 01 21:11:36.065219 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:36.065199 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44594b37-7m92d"] Nov 01 21:11:36.068254 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:36.068229 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c004f1b-mfht8"] Nov 01 21:11:37.372748 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:37.372716 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5126f2ca-9d9e-47c5-ac5c-917192929e84" path="/var/lib/kubelet/pods/5126f2ca-9d9e-47c5-ac5c-917192929e84/volumes" Nov 01 21:11:37.373207 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:37.373192 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f40c85e9-90d4-4d9a-a2b4-fcddafc3aa43" path="/var/lib/kubelet/pods/f40c85e9-90d4-4d9a-a2b4-fcddafc3aa43/volumes" Nov 01 21:11:37.714946 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:37.714845 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:11:41.343552 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:41.343507 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3ab520-7b7f4ff56c-vszhw"] Nov 01 21:11:41.459258 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:41.459220 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3ab520-848dc5bd6c-k9zhf"] Nov 01 21:11:41.739795 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:41.739760 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:11:41.896020 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:41.891628 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:11:42.746549 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:42.746511 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:11:45.769458 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:11:45.769415 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:11:46.343986 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:46.343940 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-3ab520-7b7f4ff56c-vszhw" podUID="45202ae4-063e-42eb-a800-a7118450a9e5" containerName="payload-processing" containerID="cri-o://b97410525adb1cafdd39591d28b4635178788a1df7feadf4271d7c396146e005" gracePeriod=25 Nov 01 21:11:46.460731 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:46.460673 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3ab520-848dc5bd6c-k9zhf" podUID="23aa6e10-cab2-459f-8bfe-f1f4772e343a" containerName="payload-pre-processing" containerID="cri-o://06ba08968a62529995fd28cd07703495d8ea44fb8309af02d4adc98023b2239e" gracePeriod=25 Nov 01 21:11:46.658211 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:46.658180 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3ab520-7b7f4ff56c-vszhw" Nov 01 21:11:46.678914 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:46.678877 4484 generic.go:356] "Generic (PLEG): container finished" podID="45202ae4-063e-42eb-a800-a7118450a9e5" containerID="b97410525adb1cafdd39591d28b4635178788a1df7feadf4271d7c396146e005" exitCode=0 Nov 01 21:11:46.679204 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:46.679183 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3ab520-7b7f4ff56c-vszhw" event={"ID":"45202ae4-063e-42eb-a800-a7118450a9e5","Type":"ContainerDied","Data":"b97410525adb1cafdd39591d28b4635178788a1df7feadf4271d7c396146e005"} Nov 01 21:11:46.679291 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:46.679242 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3ab520-7b7f4ff56c-vszhw" event={"ID":"45202ae4-063e-42eb-a800-a7118450a9e5","Type":"ContainerDied","Data":"54072412428de32458a273e4488ad793d57b42be42c890c31ced7506622248ee"} Nov 01 21:11:46.679291 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:46.679274 4484 scope.go:122] "RemoveContainer" containerID="b97410525adb1cafdd39591d28b4635178788a1df7feadf4271d7c396146e005" Nov 01 21:11:46.679457 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:46.679443 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3ab520-7b7f4ff56c-vszhw" Nov 01 21:11:46.686202 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:46.685332 4484 generic.go:356] "Generic (PLEG): container finished" podID="23aa6e10-cab2-459f-8bfe-f1f4772e343a" containerID="06ba08968a62529995fd28cd07703495d8ea44fb8309af02d4adc98023b2239e" exitCode=0 Nov 01 21:11:46.686202 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:46.685434 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3ab520-848dc5bd6c-k9zhf" event={"ID":"23aa6e10-cab2-459f-8bfe-f1f4772e343a","Type":"ContainerDied","Data":"06ba08968a62529995fd28cd07703495d8ea44fb8309af02d4adc98023b2239e"} Nov 01 21:11:46.690297 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:46.690273 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45202ae4-063e-42eb-a800-a7118450a9e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45202ae4-063e-42eb-a800-a7118450a9e5-plugins-config-volume\") pod \"45202ae4-063e-42eb-a800-a7118450a9e5\" (UID: \"45202ae4-063e-42eb-a800-a7118450a9e5\") " Nov 01 21:11:46.690418 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:46.690394 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45202ae4-063e-42eb-a800-a7118450a9e5-kube-api-access-qbjtk\" (UniqueName: \"kubernetes.io/projected/45202ae4-063e-42eb-a800-a7118450a9e5-kube-api-access-qbjtk\") pod \"45202ae4-063e-42eb-a800-a7118450a9e5\" (UID: \"45202ae4-063e-42eb-a800-a7118450a9e5\") " Nov 01 21:11:46.690722 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:46.690696 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45202ae4-063e-42eb-a800-a7118450a9e5-plugins-config-volume" pod "45202ae4-063e-42eb-a800-a7118450a9e5" (UID: "45202ae4-063e-42eb-a800-a7118450a9e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:11:46.691167 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:46.691141 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45202ae4-063e-42eb-a800-a7118450a9e5-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:11:46.693476 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:46.693442 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45202ae4-063e-42eb-a800-a7118450a9e5-kube-api-access-qbjtk" pod "45202ae4-063e-42eb-a800-a7118450a9e5" (UID: "45202ae4-063e-42eb-a800-a7118450a9e5"). InnerVolumeSpecName "kube-api-access-qbjtk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:11:46.693616 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:46.693596 4484 scope.go:122] "RemoveContainer" containerID="b97410525adb1cafdd39591d28b4635178788a1df7feadf4271d7c396146e005" Nov 01 21:11:46.693980 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:11:46.693930 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b97410525adb1cafdd39591d28b4635178788a1df7feadf4271d7c396146e005\": container with ID starting with b97410525adb1cafdd39591d28b4635178788a1df7feadf4271d7c396146e005 not found: ID does not exist" containerID="b97410525adb1cafdd39591d28b4635178788a1df7feadf4271d7c396146e005" Nov 01 21:11:46.693980 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:46.693963 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b97410525adb1cafdd39591d28b4635178788a1df7feadf4271d7c396146e005"} err="failed to get container status \"b97410525adb1cafdd39591d28b4635178788a1df7feadf4271d7c396146e005\": rpc error: code = NotFound desc = could not find container \"b97410525adb1cafdd39591d28b4635178788a1df7feadf4271d7c396146e005\": container with ID starting with b97410525adb1cafdd39591d28b4635178788a1df7feadf4271d7c396146e005 not found: ID does not exist" Nov 01 21:11:46.731842 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:46.731818 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3ab520-848dc5bd6c-k9zhf" Nov 01 21:11:46.792143 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:46.792108 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23aa6e10-cab2-459f-8bfe-f1f4772e343a-kube-api-access-bgt58\" (UniqueName: \"kubernetes.io/projected/23aa6e10-cab2-459f-8bfe-f1f4772e343a-kube-api-access-bgt58\") pod \"23aa6e10-cab2-459f-8bfe-f1f4772e343a\" (UID: \"23aa6e10-cab2-459f-8bfe-f1f4772e343a\") " Nov 01 21:11:46.792645 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:46.792255 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23aa6e10-cab2-459f-8bfe-f1f4772e343a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23aa6e10-cab2-459f-8bfe-f1f4772e343a-plugins-config-volume\") pod \"23aa6e10-cab2-459f-8bfe-f1f4772e343a\" (UID: \"23aa6e10-cab2-459f-8bfe-f1f4772e343a\") " Nov 01 21:11:46.792736 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:46.792657 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23aa6e10-cab2-459f-8bfe-f1f4772e343a-plugins-config-volume" pod "23aa6e10-cab2-459f-8bfe-f1f4772e343a" (UID: "23aa6e10-cab2-459f-8bfe-f1f4772e343a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:11:46.793071 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:46.793051 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23aa6e10-cab2-459f-8bfe-f1f4772e343a-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:11:46.793071 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:46.793070 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbjtk\" (UniqueName: \"kubernetes.io/projected/45202ae4-063e-42eb-a800-a7118450a9e5-kube-api-access-qbjtk\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:11:46.795027 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:46.794983 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23aa6e10-cab2-459f-8bfe-f1f4772e343a-kube-api-access-bgt58" pod "23aa6e10-cab2-459f-8bfe-f1f4772e343a" (UID: "23aa6e10-cab2-459f-8bfe-f1f4772e343a"). InnerVolumeSpecName "kube-api-access-bgt58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:11:46.894159 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:46.894084 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgt58\" (UniqueName: \"kubernetes.io/projected/23aa6e10-cab2-459f-8bfe-f1f4772e343a-kube-api-access-bgt58\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:11:46.933050 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:46.928619 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:11:47.106317 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:47.106283 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3ab520-7b7f4ff56c-vszhw"] Nov 01 21:11:47.110017 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:47.109974 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3ab520-7b7f4ff56c-vszhw"] Nov 01 21:11:47.379603 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:47.377961 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45202ae4-063e-42eb-a800-a7118450a9e5" path="/var/lib/kubelet/pods/45202ae4-063e-42eb-a800-a7118450a9e5/volumes" Nov 01 21:11:47.536503 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:47.536464 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:11:47.693847 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:47.693753 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3ab520-848dc5bd6c-k9zhf" Nov 01 21:11:47.693847 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:47.693771 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3ab520-848dc5bd6c-k9zhf" event={"ID":"23aa6e10-cab2-459f-8bfe-f1f4772e343a","Type":"ContainerDied","Data":"7b261bd949d427bd9325ddac45111c207e012d1ead8898128c95368feacea34b"} Nov 01 21:11:47.693847 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:47.693825 4484 scope.go:122] "RemoveContainer" containerID="06ba08968a62529995fd28cd07703495d8ea44fb8309af02d4adc98023b2239e" Nov 01 21:11:47.861690 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:47.861651 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3ab520-848dc5bd6c-k9zhf"] Nov 01 21:11:47.912184 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:47.912149 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3ab520-848dc5bd6c-k9zhf"] Nov 01 21:11:48.402089 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:48.402054 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4ac1ef-gtngk"] Nov 01 21:11:48.402845 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:48.402823 4484 cpu_manager.go:414] "containerMap: removing container" podUID="45202ae4-063e-42eb-a800-a7118450a9e5" containerName="payload-processing" Nov 01 21:11:48.402980 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:48.402846 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45202ae4-063e-42eb-a800-a7118450a9e5" containerName="payload-processing" Nov 01 21:11:48.402980 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:48.402863 4484 cpu_manager.go:414] "containerMap: removing container" podUID="a19f1a9a-b9d2-4ee1-8336-4425230b87b0" containerName="revoke-keys" Nov 01 21:11:48.402980 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:48.402871 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a19f1a9a-b9d2-4ee1-8336-4425230b87b0" containerName="revoke-keys" Nov 01 21:11:48.402980 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:48.402898 4484 cpu_manager.go:414] "containerMap: removing container" podUID="23aa6e10-cab2-459f-8bfe-f1f4772e343a" containerName="payload-pre-processing" Nov 01 21:11:48.402980 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:48.402905 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23aa6e10-cab2-459f-8bfe-f1f4772e343a" containerName="payload-pre-processing" Nov 01 21:11:48.403298 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:48.403110 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="45202ae4-063e-42eb-a800-a7118450a9e5" containerName="payload-processing" Nov 01 21:11:48.403298 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:48.403131 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="a19f1a9a-b9d2-4ee1-8336-4425230b87b0" containerName="revoke-keys" Nov 01 21:11:48.403298 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:48.403145 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="23aa6e10-cab2-459f-8bfe-f1f4772e343a" containerName="payload-pre-processing" Nov 01 21:11:48.408161 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:48.408140 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4ac1ef-gtngk" Nov 01 21:11:48.462765 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:48.462719 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:11:48.462938 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:48.462772 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:11:48.464044 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:48.464026 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4ac1ef-gtngk"] Nov 01 21:11:48.510281 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:48.510250 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:11:48.510469 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:48.510249 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:11:48.516468 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:48.516437 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f599828-d9f3-4e90-ba56-1561e8d577e2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4ac1ef-gtngk\" (UID: \"0f599828-d9f3-4e90-ba56-1561e8d577e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4ac1ef-gtngk" Nov 01 21:11:48.516560 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:48.516539 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg9sv\" (UniqueName: \"kubernetes.io/projected/0f599828-d9f3-4e90-ba56-1561e8d577e2-kube-api-access-mg9sv\") pod \"maas-api-revoke-keys-e2e-inf-a-4ac1ef-gtngk\" (UID: \"0f599828-d9f3-4e90-ba56-1561e8d577e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4ac1ef-gtngk" Nov 01 21:11:48.516698 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:48.516681 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f599828-d9f3-4e90-ba56-1561e8d577e2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4ac1ef-gtngk\" (UID: \"0f599828-d9f3-4e90-ba56-1561e8d577e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4ac1ef-gtngk" Nov 01 21:11:48.617904 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:48.617868 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f599828-d9f3-4e90-ba56-1561e8d577e2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4ac1ef-gtngk\" (UID: \"0f599828-d9f3-4e90-ba56-1561e8d577e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4ac1ef-gtngk" Nov 01 21:11:48.618099 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:48.618012 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f599828-d9f3-4e90-ba56-1561e8d577e2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4ac1ef-gtngk\" (UID: \"0f599828-d9f3-4e90-ba56-1561e8d577e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4ac1ef-gtngk" Nov 01 21:11:48.618099 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:48.618056 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg9sv\" (UniqueName: \"kubernetes.io/projected/0f599828-d9f3-4e90-ba56-1561e8d577e2-kube-api-access-mg9sv\") pod \"maas-api-revoke-keys-e2e-inf-a-4ac1ef-gtngk\" (UID: \"0f599828-d9f3-4e90-ba56-1561e8d577e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4ac1ef-gtngk" Nov 01 21:11:48.618644 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:48.618624 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f599828-d9f3-4e90-ba56-1561e8d577e2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4ac1ef-gtngk\" (UID: \"0f599828-d9f3-4e90-ba56-1561e8d577e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4ac1ef-gtngk" Nov 01 21:11:48.620600 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:48.620569 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f599828-d9f3-4e90-ba56-1561e8d577e2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4ac1ef-gtngk\" (UID: \"0f599828-d9f3-4e90-ba56-1561e8d577e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4ac1ef-gtngk" Nov 01 21:11:48.676333 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:48.676260 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg9sv\" (UniqueName: \"kubernetes.io/projected/0f599828-d9f3-4e90-ba56-1561e8d577e2-kube-api-access-mg9sv\") pod \"maas-api-revoke-keys-e2e-inf-a-4ac1ef-gtngk\" (UID: \"0f599828-d9f3-4e90-ba56-1561e8d577e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4ac1ef-gtngk" Nov 01 21:11:48.722078 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:48.722042 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4ac1ef-gtngk" Nov 01 21:11:49.102788 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:11:49.102751 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f599828_d9f3_4e90_ba56_1561e8d577e2.slice/crio-58430fd65bdf09e1ad3a02d1fdd7e72ecf613fc7433acb5cf8abf4211d290ec3 WatchSource:0}: Error finding container 58430fd65bdf09e1ad3a02d1fdd7e72ecf613fc7433acb5cf8abf4211d290ec3: Status 404 returned error can't find the container with id 58430fd65bdf09e1ad3a02d1fdd7e72ecf613fc7433acb5cf8abf4211d290ec3 Nov 01 21:11:49.103469 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:49.103446 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4ac1ef-gtngk"] Nov 01 21:11:49.375648 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:49.375571 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23aa6e10-cab2-459f-8bfe-f1f4772e343a" path="/var/lib/kubelet/pods/23aa6e10-cab2-459f-8bfe-f1f4772e343a/volumes" Nov 01 21:11:49.712772 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:49.712678 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4ac1ef-gtngk" event={"ID":"0f599828-d9f3-4e90-ba56-1561e8d577e2","Type":"ContainerStarted","Data":"d265ed2597a7b992c73050b65dbc231df667281560d1fdf482377cfa3294b8ee"} Nov 01 21:11:49.712772 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:49.712740 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4ac1ef-gtngk" event={"ID":"0f599828-d9f3-4e90-ba56-1561e8d577e2","Type":"ContainerStarted","Data":"58430fd65bdf09e1ad3a02d1fdd7e72ecf613fc7433acb5cf8abf4211d290ec3"} Nov 01 21:11:49.829204 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:49.829143 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4ac1ef-gtngk" podStartSLOduration=1.829126721 podStartE2EDuration="1.829126721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:11:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:11:49.827335565 +0000 UTC m=+2460.604975699" watchObservedRunningTime="2026-11-01 21:11:49.829126721 +0000 UTC m=+2460.606766855" Nov 01 21:11:50.724030 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:50.723980 4484 generic.go:356] "Generic (PLEG): container finished" podID="0f599828-d9f3-4e90-ba56-1561e8d577e2" containerID="d265ed2597a7b992c73050b65dbc231df667281560d1fdf482377cfa3294b8ee" exitCode=0 Nov 01 21:11:50.724546 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:50.724026 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4ac1ef-gtngk" event={"ID":"0f599828-d9f3-4e90-ba56-1561e8d577e2","Type":"ContainerDied","Data":"d265ed2597a7b992c73050b65dbc231df667281560d1fdf482377cfa3294b8ee"} Nov 01 21:11:52.049328 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:52.049300 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4ac1ef-gtngk" Nov 01 21:11:52.179036 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:52.173686 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f599828-d9f3-4e90-ba56-1561e8d577e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f599828-d9f3-4e90-ba56-1561e8d577e2-maas-api-xks-ca\") pod \"0f599828-d9f3-4e90-ba56-1561e8d577e2\" (UID: \"0f599828-d9f3-4e90-ba56-1561e8d577e2\") " Nov 01 21:11:52.179036 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:52.173758 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f599828-d9f3-4e90-ba56-1561e8d577e2-kube-api-access-mg9sv\" (UniqueName: \"kubernetes.io/projected/0f599828-d9f3-4e90-ba56-1561e8d577e2-kube-api-access-mg9sv\") pod \"0f599828-d9f3-4e90-ba56-1561e8d577e2\" (UID: \"0f599828-d9f3-4e90-ba56-1561e8d577e2\") " Nov 01 21:11:52.179036 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:52.174124 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f599828-d9f3-4e90-ba56-1561e8d577e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f599828-d9f3-4e90-ba56-1561e8d577e2-maas-api-openshift-ca\") pod \"0f599828-d9f3-4e90-ba56-1561e8d577e2\" (UID: \"0f599828-d9f3-4e90-ba56-1561e8d577e2\") " Nov 01 21:11:52.179036 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:52.174353 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f599828-d9f3-4e90-ba56-1561e8d577e2-maas-api-openshift-ca" pod "0f599828-d9f3-4e90-ba56-1561e8d577e2" (UID: "0f599828-d9f3-4e90-ba56-1561e8d577e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:11:52.179036 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:52.174984 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f599828-d9f3-4e90-ba56-1561e8d577e2-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:11:52.179036 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:52.178575 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f599828-d9f3-4e90-ba56-1561e8d577e2-kube-api-access-mg9sv" pod "0f599828-d9f3-4e90-ba56-1561e8d577e2" (UID: "0f599828-d9f3-4e90-ba56-1561e8d577e2"). InnerVolumeSpecName "kube-api-access-mg9sv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:11:52.179036 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:52.178865 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f599828-d9f3-4e90-ba56-1561e8d577e2-maas-api-xks-ca" pod "0f599828-d9f3-4e90-ba56-1561e8d577e2" (UID: "0f599828-d9f3-4e90-ba56-1561e8d577e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:11:52.275939 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:52.275855 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f599828-d9f3-4e90-ba56-1561e8d577e2-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:11:52.275939 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:52.275888 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg9sv\" (UniqueName: \"kubernetes.io/projected/0f599828-d9f3-4e90-ba56-1561e8d577e2-kube-api-access-mg9sv\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:11:52.694866 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:52.694774 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:11:52.760017 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:52.759306 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4ac1ef-gtngk" Nov 01 21:11:52.760017 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:52.759328 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4ac1ef-gtngk" event={"ID":"0f599828-d9f3-4e90-ba56-1561e8d577e2","Type":"ContainerDied","Data":"58430fd65bdf09e1ad3a02d1fdd7e72ecf613fc7433acb5cf8abf4211d290ec3"} Nov 01 21:11:52.760017 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:52.759382 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58430fd65bdf09e1ad3a02d1fdd7e72ecf613fc7433acb5cf8abf4211d290ec3" Nov 01 21:11:57.225842 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:57.225808 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:11:59.111417 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.111381 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:11:59.590320 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.590273 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:11:59.596696 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.596669 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh"] Nov 01 21:11:59.597472 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.597444 4484 cpu_manager.go:414] "containerMap: removing container" podUID="0f599828-d9f3-4e90-ba56-1561e8d577e2" containerName="revoke-keys" Nov 01 21:11:59.597472 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.597474 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f599828-d9f3-4e90-ba56-1561e8d577e2" containerName="revoke-keys" Nov 01 21:11:59.597588 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.597578 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f599828-d9f3-4e90-ba56-1561e8d577e2" containerName="revoke-keys" Nov 01 21:11:59.601163 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.601147 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh" Nov 01 21:11:59.647602 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.647566 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td"] Nov 01 21:11:59.651981 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.651962 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td" Nov 01 21:11:59.662041 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.657930 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-7091fd-openshift-default-dockercfg-pms7j\"" Nov 01 21:11:59.681205 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.681171 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs8bf\" (UniqueName: \"kubernetes.io/projected/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-kube-api-access-qs8bf\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh\" (UID: \"d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh" Nov 01 21:11:59.681355 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.681209 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-istio-data\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh\" (UID: \"d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh" Nov 01 21:11:59.681355 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.681243 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-istiod-ca-cert\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh\" (UID: \"d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh" Nov 01 21:11:59.681438 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.681363 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-workload-certs\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh\" (UID: \"d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh" Nov 01 21:11:59.681438 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.681390 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-istio-token\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh\" (UID: \"d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh" Nov 01 21:11:59.681513 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.681490 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-istio-podinfo\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh\" (UID: \"d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh" Nov 01 21:11:59.681553 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.681540 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-credential-socket\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh\" (UID: \"d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh" Nov 01 21:11:59.681613 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.681600 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-workload-socket\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh\" (UID: \"d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh" Nov 01 21:11:59.681653 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.681630 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-istio-envoy\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh\" (UID: \"d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh" Nov 01 21:11:59.702573 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.702533 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh"] Nov 01 21:11:59.753058 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.753023 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td"] Nov 01 21:11:59.783125 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.783076 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs8bf\" (UniqueName: \"kubernetes.io/projected/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-kube-api-access-qs8bf\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh\" (UID: \"d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh" Nov 01 21:11:59.783328 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.783135 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-istio-data\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td\" (UID: \"f2ddd589-df2c-4d9e-a504-dd6029e7f99b\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td" Nov 01 21:11:59.783328 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.783169 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-istio-data\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh\" (UID: \"d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh" Nov 01 21:11:59.783328 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.783220 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-istiod-ca-cert\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh\" (UID: \"d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh" Nov 01 21:11:59.783328 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.783263 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-workload-socket\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td\" (UID: \"f2ddd589-df2c-4d9e-a504-dd6029e7f99b\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td" Nov 01 21:11:59.783328 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.783312 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-istio-envoy\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td\" (UID: \"f2ddd589-df2c-4d9e-a504-dd6029e7f99b\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td" Nov 01 21:11:59.783631 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.783373 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-workload-certs\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh\" (UID: \"d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh" Nov 01 21:11:59.783631 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.783402 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-istio-token\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh\" (UID: \"d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh" Nov 01 21:11:59.783631 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.783432 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-istiod-ca-cert\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td\" (UID: \"f2ddd589-df2c-4d9e-a504-dd6029e7f99b\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td" Nov 01 21:11:59.783798 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.783630 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-istio-podinfo\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td\" (UID: \"f2ddd589-df2c-4d9e-a504-dd6029e7f99b\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td" Nov 01 21:11:59.783798 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.783689 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-istio-podinfo\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh\" (UID: \"d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh" Nov 01 21:11:59.783798 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.783764 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-credential-socket\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh\" (UID: \"d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh" Nov 01 21:11:59.783949 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.783815 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-workload-socket\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh\" (UID: \"d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh" Nov 01 21:11:59.783949 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.783845 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-workload-certs\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh\" (UID: \"d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh" Nov 01 21:11:59.783949 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.783868 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-istio-envoy\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh\" (UID: \"d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh" Nov 01 21:11:59.784169 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.783956 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-istio-data\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh\" (UID: \"d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh" Nov 01 21:11:59.784169 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.784053 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-credential-socket\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td\" (UID: \"f2ddd589-df2c-4d9e-a504-dd6029e7f99b\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td" Nov 01 21:11:59.784169 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.784133 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9f6j\" (UniqueName: \"kubernetes.io/projected/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-kube-api-access-q9f6j\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td\" (UID: \"f2ddd589-df2c-4d9e-a504-dd6029e7f99b\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td" Nov 01 21:11:59.784340 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.784228 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-workload-certs\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td\" (UID: \"f2ddd589-df2c-4d9e-a504-dd6029e7f99b\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td" Nov 01 21:11:59.784340 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.784283 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-istio-token\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td\" (UID: \"f2ddd589-df2c-4d9e-a504-dd6029e7f99b\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td" Nov 01 21:11:59.784464 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.784373 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-credential-socket\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh\" (UID: \"d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh" Nov 01 21:11:59.784609 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.784589 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-istiod-ca-cert\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh\" (UID: \"d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh" Nov 01 21:11:59.784847 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.784760 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-workload-socket\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh\" (UID: \"d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh" Nov 01 21:11:59.790036 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.787187 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-istio-envoy\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh\" (UID: \"d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh" Nov 01 21:11:59.790036 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.787203 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-istio-podinfo\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh\" (UID: \"d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh" Nov 01 21:11:59.844401 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.844322 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-istio-token\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh\" (UID: \"d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh" Nov 01 21:11:59.844661 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.844628 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs8bf\" (UniqueName: \"kubernetes.io/projected/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-kube-api-access-qs8bf\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh\" (UID: \"d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh" Nov 01 21:11:59.870974 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.870929 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:11:59.885647 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.885616 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-workload-certs\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td\" (UID: \"f2ddd589-df2c-4d9e-a504-dd6029e7f99b\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td" Nov 01 21:11:59.885824 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.885655 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-istio-token\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td\" (UID: \"f2ddd589-df2c-4d9e-a504-dd6029e7f99b\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td" Nov 01 21:11:59.885824 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.885732 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-istio-data\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td\" (UID: \"f2ddd589-df2c-4d9e-a504-dd6029e7f99b\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td" Nov 01 21:11:59.885824 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.885793 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-workload-socket\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td\" (UID: \"f2ddd589-df2c-4d9e-a504-dd6029e7f99b\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td" Nov 01 21:11:59.886058 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.885837 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-istio-envoy\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td\" (UID: \"f2ddd589-df2c-4d9e-a504-dd6029e7f99b\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td" Nov 01 21:11:59.886058 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.885905 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-istiod-ca-cert\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td\" (UID: \"f2ddd589-df2c-4d9e-a504-dd6029e7f99b\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td" Nov 01 21:11:59.886181 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.886087 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-workload-certs\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td\" (UID: \"f2ddd589-df2c-4d9e-a504-dd6029e7f99b\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td" Nov 01 21:11:59.886181 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.886107 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-istio-data\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td\" (UID: \"f2ddd589-df2c-4d9e-a504-dd6029e7f99b\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td" Nov 01 21:11:59.886439 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.886413 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-workload-socket\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td\" (UID: \"f2ddd589-df2c-4d9e-a504-dd6029e7f99b\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td" Nov 01 21:11:59.886536 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.886490 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-istio-podinfo\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td\" (UID: \"f2ddd589-df2c-4d9e-a504-dd6029e7f99b\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td" Nov 01 21:11:59.886671 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.886648 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-credential-socket\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td\" (UID: \"f2ddd589-df2c-4d9e-a504-dd6029e7f99b\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td" Nov 01 21:11:59.886765 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.886697 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9f6j\" (UniqueName: \"kubernetes.io/projected/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-kube-api-access-q9f6j\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td\" (UID: \"f2ddd589-df2c-4d9e-a504-dd6029e7f99b\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td" Nov 01 21:11:59.886765 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.886709 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-istiod-ca-cert\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td\" (UID: \"f2ddd589-df2c-4d9e-a504-dd6029e7f99b\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td" Nov 01 21:11:59.886898 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.886874 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-credential-socket\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td\" (UID: \"f2ddd589-df2c-4d9e-a504-dd6029e7f99b\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td" Nov 01 21:11:59.888408 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.888388 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-istio-envoy\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td\" (UID: \"f2ddd589-df2c-4d9e-a504-dd6029e7f99b\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td" Nov 01 21:11:59.888964 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.888934 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-istio-podinfo\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td\" (UID: \"f2ddd589-df2c-4d9e-a504-dd6029e7f99b\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td" Nov 01 21:11:59.916573 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.916541 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh" Nov 01 21:11:59.943588 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.943556 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-istio-token\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td\" (UID: \"f2ddd589-df2c-4d9e-a504-dd6029e7f99b\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td" Nov 01 21:11:59.944607 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.944579 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9f6j\" (UniqueName: \"kubernetes.io/projected/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-kube-api-access-q9f6j\") pod \"e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td\" (UID: \"f2ddd589-df2c-4d9e-a504-dd6029e7f99b\") " pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td" Nov 01 21:11:59.967626 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:11:59.967593 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td" Nov 01 21:12:00.190769 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:00.190686 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-dd038b-px89s"] Nov 01 21:12:00.195106 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:00.195076 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-dd038b-px89s"] Nov 01 21:12:00.353885 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:00.353855 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh"] Nov 01 21:12:00.354108 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:12:00.354056 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1daf54e_deaf_48ab_ab8c_e16cdd7ef0fe.slice/crio-e8adba4108b07a1c74c51d463eb9a8d586d5ee19ba28432f036f8316e49b52b2 WatchSource:0}: Error finding container e8adba4108b07a1c74c51d463eb9a8d586d5ee19ba28432f036f8316e49b52b2: Status 404 returned error can't find the container with id e8adba4108b07a1c74c51d463eb9a8d586d5ee19ba28432f036f8316e49b52b2 Nov 01 21:12:00.356896 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:00.356862 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:12:00.356988 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:00.356929 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:12:00.356988 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:00.356958 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:12:00.407624 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:00.407591 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td"] Nov 01 21:12:00.407773 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:12:00.407597 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2ddd589_df2c_4d9e_a504_dd6029e7f99b.slice/crio-abf1703672cff47df88d572269847b9f2a5fcd4c2ef558d0e84266639412aede WatchSource:0}: Error finding container abf1703672cff47df88d572269847b9f2a5fcd4c2ef558d0e84266639412aede: Status 404 returned error can't find the container with id abf1703672cff47df88d572269847b9f2a5fcd4c2ef558d0e84266639412aede Nov 01 21:12:00.410555 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:00.410517 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:12:00.410658 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:00.410588 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:12:00.410658 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:00.410634 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:12:00.608898 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:00.608866 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:12:00.821309 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:00.821274 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh" event={"ID":"d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe","Type":"ContainerStarted","Data":"c4bda6656f8f23bccaf4362114fd946e89fd87aed8cc7825ea9178be76944b79"} Nov 01 21:12:00.821555 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:00.821337 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh" event={"ID":"d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe","Type":"ContainerStarted","Data":"e8adba4108b07a1c74c51d463eb9a8d586d5ee19ba28432f036f8316e49b52b2"} Nov 01 21:12:00.823732 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:00.823699 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td" event={"ID":"f2ddd589-df2c-4d9e-a504-dd6029e7f99b","Type":"ContainerStarted","Data":"1367aa64dd18542dca6efa2a415391c8daf86cb47e3ffaf953b8767ac5144a63"} Nov 01 21:12:00.823865 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:00.823759 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td" event={"ID":"f2ddd589-df2c-4d9e-a504-dd6029e7f99b","Type":"ContainerStarted","Data":"abf1703672cff47df88d572269847b9f2a5fcd4c2ef558d0e84266639412aede"} Nov 01 21:12:00.863617 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:00.863581 4484 scope.go:122] "RemoveContainer" containerID="c492d331e125a38faf3d6f62d30cb040649d013ff87b2b5002ed81e25dab6623" Nov 01 21:12:00.877933 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:00.877905 4484 scope.go:122] "RemoveContainer" containerID="ea01f477ba1394ecc23d38e1cc46a4f862614977326f4f219c5069088449f225" Nov 01 21:12:00.917341 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:00.917316 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh" Nov 01 21:12:00.968511 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:00.968485 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td" Nov 01 21:12:01.068563 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:01.068509 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh" podStartSLOduration=2.06849431 podStartE2EDuration="2.06849431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:11:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:12:00.950296928 +0000 UTC m=+2471.727937080" watchObservedRunningTime="2026-11-01 21:12:01.06849431 +0000 UTC m=+2471.846134445" Nov 01 21:12:01.071555 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:01.071468 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td" podStartSLOduration=2.071455327 podStartE2EDuration="2.071455327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:11:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:12:01.067725997 +0000 UTC m=+2471.845366151" watchObservedRunningTime="2026-11-01 21:12:01.071455327 +0000 UTC m=+2471.849095461" Nov 01 21:12:01.376643 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:01.376575 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11dcc6c3-7e00-40d3-8df5-f9b93d2dbc60" path="/var/lib/kubelet/pods/11dcc6c3-7e00-40d3-8df5-f9b93d2dbc60/volumes" Nov 01 21:12:01.737639 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:01.737527 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4ac1ef-5bddb89485-dpbt4"] Nov 01 21:12:01.992934 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:01.902171 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4ac1ef-db4769dc-ffkcl"] Nov 01 21:12:01.992934 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:01.926138 4484 patch_prober.go:29] interesting pod/e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body= Nov 01 21:12:01.992934 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:01.992882 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh" podUID="d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" Nov 01 21:12:01.992934 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:01.969984 4484 patch_prober.go:29] interesting pod/e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body= Nov 01 21:12:01.993316 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:01.992949 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td" podUID="f2ddd589-df2c-4d9e-a504-dd6029e7f99b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" Nov 01 21:12:02.160067 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:02.154882 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:12:02.458963 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:02.458852 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:12:02.917517 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:02.917473 4484 patch_prober.go:29] interesting pod/e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body= Nov 01 21:12:02.917824 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:02.917791 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh" podUID="d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" Nov 01 21:12:02.968044 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:02.967985 4484 patch_prober.go:29] interesting pod/e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body= Nov 01 21:12:02.968232 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:02.968083 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td" podUID="f2ddd589-df2c-4d9e-a504-dd6029e7f99b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" Nov 01 21:12:03.926020 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:03.920494 4484 patch_prober.go:29] interesting pod/e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body= Nov 01 21:12:03.926020 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:03.920563 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh" podUID="d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" Nov 01 21:12:03.968813 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:03.968688 4484 patch_prober.go:29] interesting pod/e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body= Nov 01 21:12:03.968813 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:03.968753 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td" podUID="f2ddd589-df2c-4d9e-a504-dd6029e7f99b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" Nov 01 21:12:04.377066 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:04.377029 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:12:04.932057 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:04.931852 4484 patch_prober.go:29] interesting pod/e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body= Nov 01 21:12:04.932057 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:04.931910 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh" podUID="d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" Nov 01 21:12:04.968409 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:04.968365 4484 patch_prober.go:29] interesting pod/e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body= Nov 01 21:12:04.968600 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:04.968440 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td" podUID="f2ddd589-df2c-4d9e-a504-dd6029e7f99b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" Nov 01 21:12:05.157357 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:05.157307 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb"] Nov 01 21:12:05.157566 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:05.157536 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb" podUID="da4ca8d2-0b86-4eba-8920-d9633474d6d2" containerName="istio-proxy" containerID="cri-o://1cdf7483f836ce3134a9ad04106b8bff6818378c4298844fab81ad0dfa577a41" gracePeriod=30 Nov 01 21:12:05.921020 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:05.920484 4484 patch_prober.go:29] interesting pod/e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body= Nov 01 21:12:05.921020 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:05.920550 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh" podUID="d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" Nov 01 21:12:05.968702 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:05.968669 4484 patch_prober.go:29] interesting pod/e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body= Nov 01 21:12:05.969132 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:05.969088 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td" podUID="f2ddd589-df2c-4d9e-a504-dd6029e7f99b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" Nov 01 21:12:06.330678 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:06.330586 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:12:06.497715 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:06.497682 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:12:06.662145 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:06.662055 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:12:06.740112 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:06.740067 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-4ac1ef-5bddb89485-dpbt4" podUID="74f275b8-4451-479f-9a16-99cb725261b3" containerName="payload-processing" containerID="cri-o://3936314ce0f913719d75613f969434fefbce2855f29d5087890f7375e6a68f6d" gracePeriod=25 Nov 01 21:12:06.930093 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:06.928864 4484 patch_prober.go:29] interesting pod/e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body= Nov 01 21:12:06.930093 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:06.928956 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh" podUID="d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" Nov 01 21:12:06.971028 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:06.969652 4484 patch_prober.go:29] interesting pod/e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body= Nov 01 21:12:06.971028 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:06.969740 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td" podUID="f2ddd589-df2c-4d9e-a504-dd6029e7f99b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" Nov 01 21:12:06.994667 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:06.993622 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4ac1ef-db4769dc-ffkcl" podUID="c5429352-860e-44aa-a7c0-2ad3dcdf9267" containerName="payload-pre-processing" containerID="cri-o://819f27af16c160ec2178c1679cf290305cf397680cbe51a9d2ddb88dc806c669" gracePeriod=25 Nov 01 21:12:07.043024 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:07.036584 4484 generic.go:356] "Generic (PLEG): container finished" podID="74f275b8-4451-479f-9a16-99cb725261b3" containerID="3936314ce0f913719d75613f969434fefbce2855f29d5087890f7375e6a68f6d" exitCode=0 Nov 01 21:12:07.043024 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:07.036757 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4ac1ef-5bddb89485-dpbt4" event={"ID":"74f275b8-4451-479f-9a16-99cb725261b3","Type":"ContainerDied","Data":"3936314ce0f913719d75613f969434fefbce2855f29d5087890f7375e6a68f6d"} Nov 01 21:12:07.198056 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:07.197885 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4ac1ef-5bddb89485-dpbt4" Nov 01 21:12:07.294329 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:07.294071 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74f275b8-4451-479f-9a16-99cb725261b3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74f275b8-4451-479f-9a16-99cb725261b3-plugins-config-volume\") pod \"74f275b8-4451-479f-9a16-99cb725261b3\" (UID: \"74f275b8-4451-479f-9a16-99cb725261b3\") " Nov 01 21:12:07.294329 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:07.294178 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74f275b8-4451-479f-9a16-99cb725261b3-kube-api-access-nprmb\" (UniqueName: \"kubernetes.io/projected/74f275b8-4451-479f-9a16-99cb725261b3-kube-api-access-nprmb\") pod \"74f275b8-4451-479f-9a16-99cb725261b3\" (UID: \"74f275b8-4451-479f-9a16-99cb725261b3\") " Nov 01 21:12:07.295145 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:07.295116 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74f275b8-4451-479f-9a16-99cb725261b3-plugins-config-volume" pod "74f275b8-4451-479f-9a16-99cb725261b3" (UID: "74f275b8-4451-479f-9a16-99cb725261b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:12:07.295617 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:07.295588 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74f275b8-4451-479f-9a16-99cb725261b3-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:12:07.301737 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:07.301700 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74f275b8-4451-479f-9a16-99cb725261b3-kube-api-access-nprmb" pod "74f275b8-4451-479f-9a16-99cb725261b3" (UID: "74f275b8-4451-479f-9a16-99cb725261b3"). InnerVolumeSpecName "kube-api-access-nprmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:12:07.361239 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:07.361208 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4ac1ef-db4769dc-ffkcl" Nov 01 21:12:07.396593 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:07.396557 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5429352-860e-44aa-a7c0-2ad3dcdf9267-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5429352-860e-44aa-a7c0-2ad3dcdf9267-plugins-config-volume\") pod \"c5429352-860e-44aa-a7c0-2ad3dcdf9267\" (UID: \"c5429352-860e-44aa-a7c0-2ad3dcdf9267\") " Nov 01 21:12:07.396792 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:07.396642 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5429352-860e-44aa-a7c0-2ad3dcdf9267-kube-api-access-dndhm\" (UniqueName: \"kubernetes.io/projected/c5429352-860e-44aa-a7c0-2ad3dcdf9267-kube-api-access-dndhm\") pod \"c5429352-860e-44aa-a7c0-2ad3dcdf9267\" (UID: \"c5429352-860e-44aa-a7c0-2ad3dcdf9267\") " Nov 01 21:12:07.397017 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:07.396966 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5429352-860e-44aa-a7c0-2ad3dcdf9267-plugins-config-volume" pod "c5429352-860e-44aa-a7c0-2ad3dcdf9267" (UID: "c5429352-860e-44aa-a7c0-2ad3dcdf9267"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:12:07.397765 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:07.397738 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5429352-860e-44aa-a7c0-2ad3dcdf9267-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:12:07.397893 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:07.397767 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nprmb\" (UniqueName: \"kubernetes.io/projected/74f275b8-4451-479f-9a16-99cb725261b3-kube-api-access-nprmb\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:12:07.399301 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:07.399268 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5429352-860e-44aa-a7c0-2ad3dcdf9267-kube-api-access-dndhm" pod "c5429352-860e-44aa-a7c0-2ad3dcdf9267" (UID: "c5429352-860e-44aa-a7c0-2ad3dcdf9267"). InnerVolumeSpecName "kube-api-access-dndhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:12:07.509045 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:07.500109 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dndhm\" (UniqueName: \"kubernetes.io/projected/c5429352-860e-44aa-a7c0-2ad3dcdf9267-kube-api-access-dndhm\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:12:07.922276 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:07.922245 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh" Nov 01 21:12:07.922562 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:07.922542 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh" Nov 01 21:12:07.923431 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:07.923411 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh" Nov 01 21:12:07.974352 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:07.974320 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td" Nov 01 21:12:07.974801 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:07.974766 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td" Nov 01 21:12:07.975767 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:07.975746 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td" Nov 01 21:12:08.046223 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:08.046147 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4ac1ef-5bddb89485-dpbt4" Nov 01 21:12:08.046223 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:08.046160 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4ac1ef-5bddb89485-dpbt4" event={"ID":"74f275b8-4451-479f-9a16-99cb725261b3","Type":"ContainerDied","Data":"4fffdcaba6b10a137f591d815dc582c12f91eb85dffb5a6b000e039e9cb73949"} Nov 01 21:12:08.046445 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:08.046224 4484 scope.go:122] "RemoveContainer" containerID="3936314ce0f913719d75613f969434fefbce2855f29d5087890f7375e6a68f6d" Nov 01 21:12:08.047816 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:08.047788 4484 generic.go:356] "Generic (PLEG): container finished" podID="c5429352-860e-44aa-a7c0-2ad3dcdf9267" containerID="819f27af16c160ec2178c1679cf290305cf397680cbe51a9d2ddb88dc806c669" exitCode=0 Nov 01 21:12:08.047937 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:08.047839 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4ac1ef-db4769dc-ffkcl" Nov 01 21:12:08.047937 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:08.047878 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4ac1ef-db4769dc-ffkcl" event={"ID":"c5429352-860e-44aa-a7c0-2ad3dcdf9267","Type":"ContainerDied","Data":"819f27af16c160ec2178c1679cf290305cf397680cbe51a9d2ddb88dc806c669"} Nov 01 21:12:08.047937 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:08.047927 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4ac1ef-db4769dc-ffkcl" event={"ID":"c5429352-860e-44aa-a7c0-2ad3dcdf9267","Type":"ContainerDied","Data":"dbff33f5f1caaafc8033e7d1ae5b2c68d6a8ec051942f88c94008891ec32f93d"} Nov 01 21:12:08.058272 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:08.058255 4484 scope.go:122] "RemoveContainer" containerID="819f27af16c160ec2178c1679cf290305cf397680cbe51a9d2ddb88dc806c669" Nov 01 21:12:08.067966 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:08.067945 4484 scope.go:122] "RemoveContainer" containerID="819f27af16c160ec2178c1679cf290305cf397680cbe51a9d2ddb88dc806c669" Nov 01 21:12:08.068249 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:12:08.068230 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"819f27af16c160ec2178c1679cf290305cf397680cbe51a9d2ddb88dc806c669\": container with ID starting with 819f27af16c160ec2178c1679cf290305cf397680cbe51a9d2ddb88dc806c669 not found: ID does not exist" containerID="819f27af16c160ec2178c1679cf290305cf397680cbe51a9d2ddb88dc806c669" Nov 01 21:12:08.068301 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:08.068257 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"819f27af16c160ec2178c1679cf290305cf397680cbe51a9d2ddb88dc806c669"} err="failed to get container status \"819f27af16c160ec2178c1679cf290305cf397680cbe51a9d2ddb88dc806c669\": rpc error: code = NotFound desc = could not find container \"819f27af16c160ec2178c1679cf290305cf397680cbe51a9d2ddb88dc806c669\": container with ID starting with 819f27af16c160ec2178c1679cf290305cf397680cbe51a9d2ddb88dc806c669 not found: ID does not exist" Nov 01 21:12:08.400452 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:08.400331 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4ac1ef-db4769dc-ffkcl"] Nov 01 21:12:08.404768 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:08.404743 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4ac1ef-db4769dc-ffkcl"] Nov 01 21:12:08.506873 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:08.506833 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4ac1ef-5bddb89485-dpbt4"] Nov 01 21:12:08.511081 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:08.511052 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4ac1ef-5bddb89485-dpbt4"] Nov 01 21:12:09.373829 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:09.373798 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74f275b8-4451-479f-9a16-99cb725261b3" path="/var/lib/kubelet/pods/74f275b8-4451-479f-9a16-99cb725261b3/volumes" Nov 01 21:12:09.374349 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:09.374334 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5429352-860e-44aa-a7c0-2ad3dcdf9267" path="/var/lib/kubelet/pods/c5429352-860e-44aa-a7c0-2ad3dcdf9267/volumes" Nov 01 21:12:09.529760 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:09.529722 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:12:10.443730 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:10.443652 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb" Nov 01 21:12:10.545287 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:10.545250 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da4ca8d2-0b86-4eba-8920-d9633474d6d2-istio-token\" (UniqueName: \"kubernetes.io/projected/da4ca8d2-0b86-4eba-8920-d9633474d6d2-istio-token\") pod \"da4ca8d2-0b86-4eba-8920-d9633474d6d2\" (UID: \"da4ca8d2-0b86-4eba-8920-d9633474d6d2\") " Nov 01 21:12:10.545469 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:10.545320 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da4ca8d2-0b86-4eba-8920-d9633474d6d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da4ca8d2-0b86-4eba-8920-d9633474d6d2-istio-data\") pod \"da4ca8d2-0b86-4eba-8920-d9633474d6d2\" (UID: \"da4ca8d2-0b86-4eba-8920-d9633474d6d2\") " Nov 01 21:12:10.545469 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:10.545377 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da4ca8d2-0b86-4eba-8920-d9633474d6d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4ca8d2-0b86-4eba-8920-d9633474d6d2-workload-socket\") pod \"da4ca8d2-0b86-4eba-8920-d9633474d6d2\" (UID: \"da4ca8d2-0b86-4eba-8920-d9633474d6d2\") " Nov 01 21:12:10.545469 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:10.545410 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da4ca8d2-0b86-4eba-8920-d9633474d6d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da4ca8d2-0b86-4eba-8920-d9633474d6d2-istio-podinfo\") pod \"da4ca8d2-0b86-4eba-8920-d9633474d6d2\" (UID: \"da4ca8d2-0b86-4eba-8920-d9633474d6d2\") " Nov 01 21:12:10.545611 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:10.545475 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da4ca8d2-0b86-4eba-8920-d9633474d6d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da4ca8d2-0b86-4eba-8920-d9633474d6d2-workload-certs\") pod \"da4ca8d2-0b86-4eba-8920-d9633474d6d2\" (UID: \"da4ca8d2-0b86-4eba-8920-d9633474d6d2\") " Nov 01 21:12:10.545611 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:10.545526 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da4ca8d2-0b86-4eba-8920-d9633474d6d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da4ca8d2-0b86-4eba-8920-d9633474d6d2-istio-envoy\") pod \"da4ca8d2-0b86-4eba-8920-d9633474d6d2\" (UID: \"da4ca8d2-0b86-4eba-8920-d9633474d6d2\") " Nov 01 21:12:10.545611 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:10.545600 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da4ca8d2-0b86-4eba-8920-d9633474d6d2-kube-api-access-dqwxz\" (UniqueName: \"kubernetes.io/projected/da4ca8d2-0b86-4eba-8920-d9633474d6d2-kube-api-access-dqwxz\") pod \"da4ca8d2-0b86-4eba-8920-d9633474d6d2\" (UID: \"da4ca8d2-0b86-4eba-8920-d9633474d6d2\") " Nov 01 21:12:10.545790 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:10.545661 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da4ca8d2-0b86-4eba-8920-d9633474d6d2-istio-data" pod "da4ca8d2-0b86-4eba-8920-d9633474d6d2" (UID: "da4ca8d2-0b86-4eba-8920-d9633474d6d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:12:10.545790 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:10.545707 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da4ca8d2-0b86-4eba-8920-d9633474d6d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4ca8d2-0b86-4eba-8920-d9633474d6d2-credential-socket\") pod \"da4ca8d2-0b86-4eba-8920-d9633474d6d2\" (UID: \"da4ca8d2-0b86-4eba-8920-d9633474d6d2\") " Nov 01 21:12:10.545790 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:10.545709 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da4ca8d2-0b86-4eba-8920-d9633474d6d2-workload-socket" pod "da4ca8d2-0b86-4eba-8920-d9633474d6d2" (UID: "da4ca8d2-0b86-4eba-8920-d9633474d6d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:12:10.545790 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:10.545766 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da4ca8d2-0b86-4eba-8920-d9633474d6d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da4ca8d2-0b86-4eba-8920-d9633474d6d2-istiod-ca-cert\") pod \"da4ca8d2-0b86-4eba-8920-d9633474d6d2\" (UID: \"da4ca8d2-0b86-4eba-8920-d9633474d6d2\") " Nov 01 21:12:10.546024 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:10.545798 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da4ca8d2-0b86-4eba-8920-d9633474d6d2-workload-certs" pod "da4ca8d2-0b86-4eba-8920-d9633474d6d2" (UID: "da4ca8d2-0b86-4eba-8920-d9633474d6d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:12:10.546024 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:10.545953 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da4ca8d2-0b86-4eba-8920-d9633474d6d2-credential-socket" pod "da4ca8d2-0b86-4eba-8920-d9633474d6d2" (UID: "da4ca8d2-0b86-4eba-8920-d9633474d6d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:12:10.546298 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:10.546268 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da4ca8d2-0b86-4eba-8920-d9633474d6d2-istiod-ca-cert" pod "da4ca8d2-0b86-4eba-8920-d9633474d6d2" (UID: "da4ca8d2-0b86-4eba-8920-d9633474d6d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:12:10.546488 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:10.546466 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4ca8d2-0b86-4eba-8920-d9633474d6d2-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:12:10.546540 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:10.546497 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da4ca8d2-0b86-4eba-8920-d9633474d6d2-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:12:10.546540 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:10.546512 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4ca8d2-0b86-4eba-8920-d9633474d6d2-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:12:10.546540 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:10.546526 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da4ca8d2-0b86-4eba-8920-d9633474d6d2-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:12:10.546659 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:10.546541 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da4ca8d2-0b86-4eba-8920-d9633474d6d2-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:12:10.547874 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:10.547844 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da4ca8d2-0b86-4eba-8920-d9633474d6d2-kube-api-access-dqwxz" pod "da4ca8d2-0b86-4eba-8920-d9633474d6d2" (UID: "da4ca8d2-0b86-4eba-8920-d9633474d6d2"). InnerVolumeSpecName "kube-api-access-dqwxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:12:10.548037 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:10.547905 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da4ca8d2-0b86-4eba-8920-d9633474d6d2-istio-envoy" pod "da4ca8d2-0b86-4eba-8920-d9633474d6d2" (UID: "da4ca8d2-0b86-4eba-8920-d9633474d6d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:12:10.548037 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:10.548023 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da4ca8d2-0b86-4eba-8920-d9633474d6d2-istio-token" pod "da4ca8d2-0b86-4eba-8920-d9633474d6d2" (UID: "da4ca8d2-0b86-4eba-8920-d9633474d6d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:12:10.548166 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:10.548030 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da4ca8d2-0b86-4eba-8920-d9633474d6d2-istio-podinfo" pod "da4ca8d2-0b86-4eba-8920-d9633474d6d2" (UID: "da4ca8d2-0b86-4eba-8920-d9633474d6d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:12:10.647821 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:10.647789 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da4ca8d2-0b86-4eba-8920-d9633474d6d2-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:12:10.647821 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:10.647818 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqwxz\" (UniqueName: \"kubernetes.io/projected/da4ca8d2-0b86-4eba-8920-d9633474d6d2-kube-api-access-dqwxz\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:12:10.648077 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:10.647830 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da4ca8d2-0b86-4eba-8920-d9633474d6d2-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:12:10.648077 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:10.647843 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da4ca8d2-0b86-4eba-8920-d9633474d6d2-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:12:11.074406 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:11.074373 4484 generic.go:356] "Generic (PLEG): container finished" podID="da4ca8d2-0b86-4eba-8920-d9633474d6d2" containerID="1cdf7483f836ce3134a9ad04106b8bff6818378c4298844fab81ad0dfa577a41" exitCode=0 Nov 01 21:12:11.074595 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:11.074427 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb" Nov 01 21:12:11.074595 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:11.074454 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb" event={"ID":"da4ca8d2-0b86-4eba-8920-d9633474d6d2","Type":"ContainerDied","Data":"1cdf7483f836ce3134a9ad04106b8bff6818378c4298844fab81ad0dfa577a41"} Nov 01 21:12:11.074595 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:11.074532 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb" event={"ID":"da4ca8d2-0b86-4eba-8920-d9633474d6d2","Type":"ContainerDied","Data":"291edce2e389684abbce9438b178f721ac37a310f38c78610ba9a1d6164dd0d4"} Nov 01 21:12:11.074595 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:11.074553 4484 scope.go:122] "RemoveContainer" containerID="1cdf7483f836ce3134a9ad04106b8bff6818378c4298844fab81ad0dfa577a41" Nov 01 21:12:11.086278 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:11.086260 4484 scope.go:122] "RemoveContainer" containerID="1cdf7483f836ce3134a9ad04106b8bff6818378c4298844fab81ad0dfa577a41" Nov 01 21:12:11.086637 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:12:11.086610 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cdf7483f836ce3134a9ad04106b8bff6818378c4298844fab81ad0dfa577a41\": container with ID starting with 1cdf7483f836ce3134a9ad04106b8bff6818378c4298844fab81ad0dfa577a41 not found: ID does not exist" containerID="1cdf7483f836ce3134a9ad04106b8bff6818378c4298844fab81ad0dfa577a41" Nov 01 21:12:11.086704 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:11.086649 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cdf7483f836ce3134a9ad04106b8bff6818378c4298844fab81ad0dfa577a41"} err="failed to get container status \"1cdf7483f836ce3134a9ad04106b8bff6818378c4298844fab81ad0dfa577a41\": rpc error: code = NotFound desc = could not find container \"1cdf7483f836ce3134a9ad04106b8bff6818378c4298844fab81ad0dfa577a41\": container with ID starting with 1cdf7483f836ce3134a9ad04106b8bff6818378c4298844fab81ad0dfa577a41 not found: ID does not exist" Nov 01 21:12:11.242694 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:11.242657 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb"] Nov 01 21:12:11.293365 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:11.293330 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-1a1e9e-openshift-default-56c9685797-nsqsb"] Nov 01 21:12:11.377926 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:11.377828 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da4ca8d2-0b86-4eba-8920-d9633474d6d2" path="/var/lib/kubelet/pods/da4ca8d2-0b86-4eba-8920-d9633474d6d2/volumes" Nov 01 21:12:14.327752 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:14.327717 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:12:15.768410 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:12:15.768369 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:12:17.679696 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:17.679665 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-deff5e-cbgm9"] Nov 01 21:12:17.680386 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:17.680370 4484 cpu_manager.go:414] "containerMap: removing container" podUID="da4ca8d2-0b86-4eba-8920-d9633474d6d2" containerName="istio-proxy" Nov 01 21:12:17.680439 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:17.680387 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da4ca8d2-0b86-4eba-8920-d9633474d6d2" containerName="istio-proxy" Nov 01 21:12:17.680439 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:17.680403 4484 cpu_manager.go:414] "containerMap: removing container" podUID="74f275b8-4451-479f-9a16-99cb725261b3" containerName="payload-processing" Nov 01 21:12:17.680439 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:17.680408 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74f275b8-4451-479f-9a16-99cb725261b3" containerName="payload-processing" Nov 01 21:12:17.680534 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:17.680450 4484 cpu_manager.go:414] "containerMap: removing container" podUID="c5429352-860e-44aa-a7c0-2ad3dcdf9267" containerName="payload-pre-processing" Nov 01 21:12:17.680534 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:17.680455 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5429352-860e-44aa-a7c0-2ad3dcdf9267" containerName="payload-pre-processing" Nov 01 21:12:17.680604 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:17.680564 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="da4ca8d2-0b86-4eba-8920-d9633474d6d2" containerName="istio-proxy" Nov 01 21:12:17.680604 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:17.680574 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="74f275b8-4451-479f-9a16-99cb725261b3" containerName="payload-processing" Nov 01 21:12:17.680604 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:17.680587 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5429352-860e-44aa-a7c0-2ad3dcdf9267" containerName="payload-pre-processing" Nov 01 21:12:17.684201 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:17.684185 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-deff5e-cbgm9" Nov 01 21:12:17.740607 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:17.740576 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-deff5e-cbgm9"] Nov 01 21:12:17.745933 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:17.745904 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b63c3f9-f156-46dd-a0ee-854b5b9c5350-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-deff5e-cbgm9\" (UID: \"7b63c3f9-f156-46dd-a0ee-854b5b9c5350\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-deff5e-cbgm9" Nov 01 21:12:17.746134 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:17.746107 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7ddf\" (UniqueName: \"kubernetes.io/projected/7b63c3f9-f156-46dd-a0ee-854b5b9c5350-kube-api-access-r7ddf\") pod \"maas-api-revoke-keys-e2e-api-a-deff5e-cbgm9\" (UID: \"7b63c3f9-f156-46dd-a0ee-854b5b9c5350\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-deff5e-cbgm9" Nov 01 21:12:17.746275 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:17.746154 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b63c3f9-f156-46dd-a0ee-854b5b9c5350-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-deff5e-cbgm9\" (UID: \"7b63c3f9-f156-46dd-a0ee-854b5b9c5350\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-deff5e-cbgm9" Nov 01 21:12:17.786635 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:17.786601 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:12:17.786635 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:17.786605 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:12:17.786635 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:17.786605 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:12:17.786886 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:17.786669 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:12:17.847588 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:17.847552 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7ddf\" (UniqueName: \"kubernetes.io/projected/7b63c3f9-f156-46dd-a0ee-854b5b9c5350-kube-api-access-r7ddf\") pod \"maas-api-revoke-keys-e2e-api-a-deff5e-cbgm9\" (UID: \"7b63c3f9-f156-46dd-a0ee-854b5b9c5350\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-deff5e-cbgm9" Nov 01 21:12:17.847588 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:17.847597 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b63c3f9-f156-46dd-a0ee-854b5b9c5350-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-deff5e-cbgm9\" (UID: \"7b63c3f9-f156-46dd-a0ee-854b5b9c5350\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-deff5e-cbgm9" Nov 01 21:12:17.847816 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:17.847712 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b63c3f9-f156-46dd-a0ee-854b5b9c5350-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-deff5e-cbgm9\" (UID: \"7b63c3f9-f156-46dd-a0ee-854b5b9c5350\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-deff5e-cbgm9" Nov 01 21:12:17.848404 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:17.848373 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b63c3f9-f156-46dd-a0ee-854b5b9c5350-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-deff5e-cbgm9\" (UID: \"7b63c3f9-f156-46dd-a0ee-854b5b9c5350\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-deff5e-cbgm9" Nov 01 21:12:17.849958 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:17.849940 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b63c3f9-f156-46dd-a0ee-854b5b9c5350-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-deff5e-cbgm9\" (UID: \"7b63c3f9-f156-46dd-a0ee-854b5b9c5350\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-deff5e-cbgm9" Nov 01 21:12:17.905054 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:17.905017 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7ddf\" (UniqueName: \"kubernetes.io/projected/7b63c3f9-f156-46dd-a0ee-854b5b9c5350-kube-api-access-r7ddf\") pod \"maas-api-revoke-keys-e2e-api-a-deff5e-cbgm9\" (UID: \"7b63c3f9-f156-46dd-a0ee-854b5b9c5350\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-deff5e-cbgm9" Nov 01 21:12:17.998231 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:17.998190 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-deff5e-cbgm9" Nov 01 21:12:18.378796 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:18.378766 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-deff5e-cbgm9"] Nov 01 21:12:18.379384 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:12:18.379362 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b63c3f9_f156_46dd_a0ee_854b5b9c5350.slice/crio-d5f6e86c68d9e0f64ad3903fe7337fe2b9084bbca3d4b736d2aa9a60ff02866b WatchSource:0}: Error finding container d5f6e86c68d9e0f64ad3903fe7337fe2b9084bbca3d4b736d2aa9a60ff02866b: Status 404 returned error can't find the container with id d5f6e86c68d9e0f64ad3903fe7337fe2b9084bbca3d4b736d2aa9a60ff02866b Nov 01 21:12:19.139070 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:19.139032 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-deff5e-cbgm9" event={"ID":"7b63c3f9-f156-46dd-a0ee-854b5b9c5350","Type":"ContainerStarted","Data":"0c9be017ab8b64cb30115390bd04c820f7634ea9d8f9d310ac130c4a67860617"} Nov 01 21:12:19.139473 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:19.139090 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-deff5e-cbgm9" event={"ID":"7b63c3f9-f156-46dd-a0ee-854b5b9c5350","Type":"ContainerStarted","Data":"d5f6e86c68d9e0f64ad3903fe7337fe2b9084bbca3d4b736d2aa9a60ff02866b"} Nov 01 21:12:19.257686 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:19.257382 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-deff5e-cbgm9" podStartSLOduration=2.257364287 podStartE2EDuration="2.257364287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:12:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:12:19.253410113 +0000 UTC m=+2490.031050240" watchObservedRunningTime="2026-11-01 21:12:19.257364287 +0000 UTC m=+2490.035004424" Nov 01 21:12:19.399777 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:19.399690 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:12:20.148295 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:20.148260 4484 generic.go:356] "Generic (PLEG): container finished" podID="7b63c3f9-f156-46dd-a0ee-854b5b9c5350" containerID="0c9be017ab8b64cb30115390bd04c820f7634ea9d8f9d310ac130c4a67860617" exitCode=0 Nov 01 21:12:20.148690 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:20.148344 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-deff5e-cbgm9" event={"ID":"7b63c3f9-f156-46dd-a0ee-854b5b9c5350","Type":"ContainerDied","Data":"0c9be017ab8b64cb30115390bd04c820f7634ea9d8f9d310ac130c4a67860617"} Nov 01 21:12:21.313122 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:21.313095 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-deff5e-cbgm9" Nov 01 21:12:21.334427 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:21.334399 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b63c3f9-f156-46dd-a0ee-854b5b9c5350-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b63c3f9-f156-46dd-a0ee-854b5b9c5350-maas-api-xks-ca\") pod \"7b63c3f9-f156-46dd-a0ee-854b5b9c5350\" (UID: \"7b63c3f9-f156-46dd-a0ee-854b5b9c5350\") " Nov 01 21:12:21.334573 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:21.334463 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b63c3f9-f156-46dd-a0ee-854b5b9c5350-kube-api-access-r7ddf\" (UniqueName: \"kubernetes.io/projected/7b63c3f9-f156-46dd-a0ee-854b5b9c5350-kube-api-access-r7ddf\") pod \"7b63c3f9-f156-46dd-a0ee-854b5b9c5350\" (UID: \"7b63c3f9-f156-46dd-a0ee-854b5b9c5350\") " Nov 01 21:12:21.334573 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:21.334491 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b63c3f9-f156-46dd-a0ee-854b5b9c5350-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b63c3f9-f156-46dd-a0ee-854b5b9c5350-maas-api-openshift-ca\") pod \"7b63c3f9-f156-46dd-a0ee-854b5b9c5350\" (UID: \"7b63c3f9-f156-46dd-a0ee-854b5b9c5350\") " Nov 01 21:12:21.334885 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:21.334862 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b63c3f9-f156-46dd-a0ee-854b5b9c5350-maas-api-openshift-ca" pod "7b63c3f9-f156-46dd-a0ee-854b5b9c5350" (UID: "7b63c3f9-f156-46dd-a0ee-854b5b9c5350"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:12:21.334987 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:21.334969 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b63c3f9-f156-46dd-a0ee-854b5b9c5350-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:12:21.336803 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:21.336777 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b63c3f9-f156-46dd-a0ee-854b5b9c5350-kube-api-access-r7ddf" pod "7b63c3f9-f156-46dd-a0ee-854b5b9c5350" (UID: "7b63c3f9-f156-46dd-a0ee-854b5b9c5350"). InnerVolumeSpecName "kube-api-access-r7ddf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:12:21.336934 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:21.336886 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b63c3f9-f156-46dd-a0ee-854b5b9c5350-maas-api-xks-ca" pod "7b63c3f9-f156-46dd-a0ee-854b5b9c5350" (UID: "7b63c3f9-f156-46dd-a0ee-854b5b9c5350"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:12:21.436400 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:21.436298 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7ddf\" (UniqueName: \"kubernetes.io/projected/7b63c3f9-f156-46dd-a0ee-854b5b9c5350-kube-api-access-r7ddf\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:12:21.436400 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:21.436338 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b63c3f9-f156-46dd-a0ee-854b5b9c5350-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:12:22.166379 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:22.166346 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-deff5e-cbgm9" event={"ID":"7b63c3f9-f156-46dd-a0ee-854b5b9c5350","Type":"ContainerDied","Data":"d5f6e86c68d9e0f64ad3903fe7337fe2b9084bbca3d4b736d2aa9a60ff02866b"} Nov 01 21:12:22.166379 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:22.166369 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-deff5e-cbgm9" Nov 01 21:12:22.166613 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:22.166400 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5f6e86c68d9e0f64ad3903fe7337fe2b9084bbca3d4b736d2aa9a60ff02866b" Nov 01 21:12:24.425305 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:24.425267 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:12:27.560348 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:27.560305 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b9305-wvt2d"] Nov 01 21:12:27.561138 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:27.561119 4484 cpu_manager.go:414] "containerMap: removing container" podUID="7b63c3f9-f156-46dd-a0ee-854b5b9c5350" containerName="revoke-keys" Nov 01 21:12:27.561191 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:27.561139 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b63c3f9-f156-46dd-a0ee-854b5b9c5350" containerName="revoke-keys" Nov 01 21:12:27.561296 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:27.561286 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b63c3f9-f156-46dd-a0ee-854b5b9c5350" containerName="revoke-keys" Nov 01 21:12:27.565139 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:27.565115 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b9305-wvt2d" Nov 01 21:12:27.620387 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:27.620355 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b9305-wvt2d"] Nov 01 21:12:27.622303 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:27.622272 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32959521-121f-43c1-939d-a0f22e0d1f58-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6b9305-wvt2d\" (UID: \"32959521-121f-43c1-939d-a0f22e0d1f58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b9305-wvt2d" Nov 01 21:12:27.622604 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:27.622567 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs9xh\" (UniqueName: \"kubernetes.io/projected/32959521-121f-43c1-939d-a0f22e0d1f58-kube-api-access-vs9xh\") pod \"maas-api-revoke-keys-e2e-inf-a-6b9305-wvt2d\" (UID: \"32959521-121f-43c1-939d-a0f22e0d1f58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b9305-wvt2d" Nov 01 21:12:27.622714 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:27.622687 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32959521-121f-43c1-939d-a0f22e0d1f58-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6b9305-wvt2d\" (UID: \"32959521-121f-43c1-939d-a0f22e0d1f58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b9305-wvt2d" Nov 01 21:12:27.666403 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:27.666368 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:12:27.666403 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:27.666377 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:12:27.666403 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:27.666403 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:12:27.666793 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:27.666470 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:12:27.723986 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:27.723947 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs9xh\" (UniqueName: \"kubernetes.io/projected/32959521-121f-43c1-939d-a0f22e0d1f58-kube-api-access-vs9xh\") pod \"maas-api-revoke-keys-e2e-inf-a-6b9305-wvt2d\" (UID: \"32959521-121f-43c1-939d-a0f22e0d1f58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b9305-wvt2d" Nov 01 21:12:27.724188 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:27.724049 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32959521-121f-43c1-939d-a0f22e0d1f58-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6b9305-wvt2d\" (UID: \"32959521-121f-43c1-939d-a0f22e0d1f58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b9305-wvt2d" Nov 01 21:12:27.724188 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:27.724095 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32959521-121f-43c1-939d-a0f22e0d1f58-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6b9305-wvt2d\" (UID: \"32959521-121f-43c1-939d-a0f22e0d1f58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b9305-wvt2d" Nov 01 21:12:27.724721 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:27.724702 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32959521-121f-43c1-939d-a0f22e0d1f58-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6b9305-wvt2d\" (UID: \"32959521-121f-43c1-939d-a0f22e0d1f58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b9305-wvt2d" Nov 01 21:12:27.726899 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:27.726866 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32959521-121f-43c1-939d-a0f22e0d1f58-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6b9305-wvt2d\" (UID: \"32959521-121f-43c1-939d-a0f22e0d1f58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b9305-wvt2d" Nov 01 21:12:27.781243 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:27.781208 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs9xh\" (UniqueName: \"kubernetes.io/projected/32959521-121f-43c1-939d-a0f22e0d1f58-kube-api-access-vs9xh\") pod \"maas-api-revoke-keys-e2e-inf-a-6b9305-wvt2d\" (UID: \"32959521-121f-43c1-939d-a0f22e0d1f58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b9305-wvt2d" Nov 01 21:12:27.879829 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:27.879749 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b9305-wvt2d" Nov 01 21:12:28.258570 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:28.258538 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b9305-wvt2d"] Nov 01 21:12:28.258735 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:12:28.258612 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32959521_121f_43c1_939d_a0f22e0d1f58.slice/crio-e7d0a63edac67e8c218bfd63d063deb5e7648155f6b0d37e38e2c2d11e816cbb WatchSource:0}: Error finding container e7d0a63edac67e8c218bfd63d063deb5e7648155f6b0d37e38e2c2d11e816cbb: Status 404 returned error can't find the container with id e7d0a63edac67e8c218bfd63d063deb5e7648155f6b0d37e38e2c2d11e816cbb Nov 01 21:12:28.624844 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:28.624815 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-deff5e-bd664cbf7-wvr5w"] Nov 01 21:12:28.731805 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:28.731763 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-deff5e-77db945ff9-jtzp7"] Nov 01 21:12:29.012966 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:29.012931 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:12:29.206398 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:29.206353 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:12:29.219179 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:29.219144 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b9305-wvt2d" event={"ID":"32959521-121f-43c1-939d-a0f22e0d1f58","Type":"ContainerStarted","Data":"e7c95b69061038095e1d78802157374152db76b31d40dbd0b47ce97c9123c7b1"} Nov 01 21:12:29.219359 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:29.219197 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b9305-wvt2d" event={"ID":"32959521-121f-43c1-939d-a0f22e0d1f58","Type":"ContainerStarted","Data":"e7d0a63edac67e8c218bfd63d063deb5e7648155f6b0d37e38e2c2d11e816cbb"} Nov 01 21:12:29.337074 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:29.336972 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b9305-wvt2d" podStartSLOduration=2.336958089 podStartE2EDuration="2.336958089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:12:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:12:29.333696057 +0000 UTC m=+2500.111336192" watchObservedRunningTime="2026-11-01 21:12:29.336958089 +0000 UTC m=+2500.114598273" Nov 01 21:12:29.534973 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:29.534935 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:12:30.228010 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:30.227962 4484 generic.go:356] "Generic (PLEG): container finished" podID="32959521-121f-43c1-939d-a0f22e0d1f58" containerID="e7c95b69061038095e1d78802157374152db76b31d40dbd0b47ce97c9123c7b1" exitCode=0 Nov 01 21:12:30.228484 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:30.228050 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b9305-wvt2d" event={"ID":"32959521-121f-43c1-939d-a0f22e0d1f58","Type":"ContainerDied","Data":"e7c95b69061038095e1d78802157374152db76b31d40dbd0b47ce97c9123c7b1"} Nov 01 21:12:31.387440 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:31.387412 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b9305-wvt2d" Nov 01 21:12:31.459882 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:31.459842 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32959521-121f-43c1-939d-a0f22e0d1f58-kube-api-access-vs9xh\" (UniqueName: \"kubernetes.io/projected/32959521-121f-43c1-939d-a0f22e0d1f58-kube-api-access-vs9xh\") pod \"32959521-121f-43c1-939d-a0f22e0d1f58\" (UID: \"32959521-121f-43c1-939d-a0f22e0d1f58\") " Nov 01 21:12:31.460133 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:31.459902 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32959521-121f-43c1-939d-a0f22e0d1f58-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32959521-121f-43c1-939d-a0f22e0d1f58-maas-api-xks-ca\") pod \"32959521-121f-43c1-939d-a0f22e0d1f58\" (UID: \"32959521-121f-43c1-939d-a0f22e0d1f58\") " Nov 01 21:12:31.460133 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:31.459960 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32959521-121f-43c1-939d-a0f22e0d1f58-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32959521-121f-43c1-939d-a0f22e0d1f58-maas-api-openshift-ca\") pod \"32959521-121f-43c1-939d-a0f22e0d1f58\" (UID: \"32959521-121f-43c1-939d-a0f22e0d1f58\") " Nov 01 21:12:31.460382 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:31.460359 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32959521-121f-43c1-939d-a0f22e0d1f58-maas-api-openshift-ca" pod "32959521-121f-43c1-939d-a0f22e0d1f58" (UID: "32959521-121f-43c1-939d-a0f22e0d1f58"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:12:31.460601 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:31.460581 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32959521-121f-43c1-939d-a0f22e0d1f58-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:12:31.462205 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:31.462176 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32959521-121f-43c1-939d-a0f22e0d1f58-maas-api-xks-ca" pod "32959521-121f-43c1-939d-a0f22e0d1f58" (UID: "32959521-121f-43c1-939d-a0f22e0d1f58"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:12:31.462303 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:31.462264 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32959521-121f-43c1-939d-a0f22e0d1f58-kube-api-access-vs9xh" pod "32959521-121f-43c1-939d-a0f22e0d1f58" (UID: "32959521-121f-43c1-939d-a0f22e0d1f58"). InnerVolumeSpecName "kube-api-access-vs9xh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:12:31.561672 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:31.561631 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs9xh\" (UniqueName: \"kubernetes.io/projected/32959521-121f-43c1-939d-a0f22e0d1f58-kube-api-access-vs9xh\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:12:31.561672 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:31.561665 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32959521-121f-43c1-939d-a0f22e0d1f58-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:12:32.245209 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:32.245177 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b9305-wvt2d" Nov 01 21:12:32.245393 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:32.245212 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b9305-wvt2d" event={"ID":"32959521-121f-43c1-939d-a0f22e0d1f58","Type":"ContainerDied","Data":"e7d0a63edac67e8c218bfd63d063deb5e7648155f6b0d37e38e2c2d11e816cbb"} Nov 01 21:12:32.245393 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:32.245261 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7d0a63edac67e8c218bfd63d063deb5e7648155f6b0d37e38e2c2d11e816cbb" Nov 01 21:12:33.625944 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:33.625905 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-deff5e-bd664cbf7-wvr5w" podUID="9ef99ac7-1947-4ffc-98ad-4a167eecba8c" containerName="payload-processing" containerID="cri-o://79fad4827ba26033adc4030db87771feea93adc731d7d8114624b186596d43b5" gracePeriod=25 Nov 01 21:12:33.732583 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:33.732539 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-deff5e-77db945ff9-jtzp7" podUID="8d19b68e-4561-480b-bcd4-3cc376211ff3" containerName="payload-pre-processing" containerID="cri-o://d176456610e05c06599d5c7fcf74edec940c77e8294a2b88b31f6f75b2386c84" gracePeriod=25 Nov 01 21:12:33.908498 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:33.908472 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-deff5e-bd664cbf7-wvr5w" Nov 01 21:12:33.993051 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:33.992984 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ef99ac7-1947-4ffc-98ad-4a167eecba8c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ef99ac7-1947-4ffc-98ad-4a167eecba8c-plugins-config-volume\") pod \"9ef99ac7-1947-4ffc-98ad-4a167eecba8c\" (UID: \"9ef99ac7-1947-4ffc-98ad-4a167eecba8c\") " Nov 01 21:12:33.993233 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:33.993076 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ef99ac7-1947-4ffc-98ad-4a167eecba8c-kube-api-access-8g7gz\" (UniqueName: \"kubernetes.io/projected/9ef99ac7-1947-4ffc-98ad-4a167eecba8c-kube-api-access-8g7gz\") pod \"9ef99ac7-1947-4ffc-98ad-4a167eecba8c\" (UID: \"9ef99ac7-1947-4ffc-98ad-4a167eecba8c\") " Nov 01 21:12:33.993369 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:33.993345 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ef99ac7-1947-4ffc-98ad-4a167eecba8c-plugins-config-volume" pod "9ef99ac7-1947-4ffc-98ad-4a167eecba8c" (UID: "9ef99ac7-1947-4ffc-98ad-4a167eecba8c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:12:33.993567 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:33.993551 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ef99ac7-1947-4ffc-98ad-4a167eecba8c-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:12:33.995301 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:33.995276 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ef99ac7-1947-4ffc-98ad-4a167eecba8c-kube-api-access-8g7gz" pod "9ef99ac7-1947-4ffc-98ad-4a167eecba8c" (UID: "9ef99ac7-1947-4ffc-98ad-4a167eecba8c"). InnerVolumeSpecName "kube-api-access-8g7gz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:12:34.017062 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:34.017037 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-deff5e-77db945ff9-jtzp7" Nov 01 21:12:34.094380 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:34.094345 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d19b68e-4561-480b-bcd4-3cc376211ff3-kube-api-access-75nbc\" (UniqueName: \"kubernetes.io/projected/8d19b68e-4561-480b-bcd4-3cc376211ff3-kube-api-access-75nbc\") pod \"8d19b68e-4561-480b-bcd4-3cc376211ff3\" (UID: \"8d19b68e-4561-480b-bcd4-3cc376211ff3\") " Nov 01 21:12:34.094555 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:34.094444 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d19b68e-4561-480b-bcd4-3cc376211ff3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d19b68e-4561-480b-bcd4-3cc376211ff3-plugins-config-volume\") pod \"8d19b68e-4561-480b-bcd4-3cc376211ff3\" (UID: \"8d19b68e-4561-480b-bcd4-3cc376211ff3\") " Nov 01 21:12:34.094809 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:34.094785 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d19b68e-4561-480b-bcd4-3cc376211ff3-plugins-config-volume" pod "8d19b68e-4561-480b-bcd4-3cc376211ff3" (UID: "8d19b68e-4561-480b-bcd4-3cc376211ff3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:12:34.094962 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:34.094950 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g7gz\" (UniqueName: \"kubernetes.io/projected/9ef99ac7-1947-4ffc-98ad-4a167eecba8c-kube-api-access-8g7gz\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:12:34.095021 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:34.094964 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d19b68e-4561-480b-bcd4-3cc376211ff3-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:12:34.096510 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:34.096489 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d19b68e-4561-480b-bcd4-3cc376211ff3-kube-api-access-75nbc" pod "8d19b68e-4561-480b-bcd4-3cc376211ff3" (UID: "8d19b68e-4561-480b-bcd4-3cc376211ff3"). InnerVolumeSpecName "kube-api-access-75nbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:12:34.196222 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:34.196144 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75nbc\" (UniqueName: \"kubernetes.io/projected/8d19b68e-4561-480b-bcd4-3cc376211ff3-kube-api-access-75nbc\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:12:34.262247 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:34.262213 4484 generic.go:356] "Generic (PLEG): container finished" podID="8d19b68e-4561-480b-bcd4-3cc376211ff3" containerID="d176456610e05c06599d5c7fcf74edec940c77e8294a2b88b31f6f75b2386c84" exitCode=0 Nov 01 21:12:34.262440 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:34.262262 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-deff5e-77db945ff9-jtzp7" Nov 01 21:12:34.262440 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:34.262297 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-deff5e-77db945ff9-jtzp7" event={"ID":"8d19b68e-4561-480b-bcd4-3cc376211ff3","Type":"ContainerDied","Data":"d176456610e05c06599d5c7fcf74edec940c77e8294a2b88b31f6f75b2386c84"} Nov 01 21:12:34.262440 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:34.262352 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-deff5e-77db945ff9-jtzp7" event={"ID":"8d19b68e-4561-480b-bcd4-3cc376211ff3","Type":"ContainerDied","Data":"c4aa699a240fc79414edf17d046409ee3f2fc730111588579aa07e5c7428f31d"} Nov 01 21:12:34.262440 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:34.262374 4484 scope.go:122] "RemoveContainer" containerID="d176456610e05c06599d5c7fcf74edec940c77e8294a2b88b31f6f75b2386c84" Nov 01 21:12:34.263861 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:34.263838 4484 generic.go:356] "Generic (PLEG): container finished" podID="9ef99ac7-1947-4ffc-98ad-4a167eecba8c" containerID="79fad4827ba26033adc4030db87771feea93adc731d7d8114624b186596d43b5" exitCode=0 Nov 01 21:12:34.264080 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:34.263881 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-deff5e-bd664cbf7-wvr5w" Nov 01 21:12:34.264080 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:34.263925 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-deff5e-bd664cbf7-wvr5w" event={"ID":"9ef99ac7-1947-4ffc-98ad-4a167eecba8c","Type":"ContainerDied","Data":"79fad4827ba26033adc4030db87771feea93adc731d7d8114624b186596d43b5"} Nov 01 21:12:34.264080 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:34.263975 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-deff5e-bd664cbf7-wvr5w" event={"ID":"9ef99ac7-1947-4ffc-98ad-4a167eecba8c","Type":"ContainerDied","Data":"cc42eb0d8fdd747be28450f10f4f0f9215cb55bdf59d3abe3cff1c9e799977a0"} Nov 01 21:12:34.274699 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:34.274680 4484 scope.go:122] "RemoveContainer" containerID="d176456610e05c06599d5c7fcf74edec940c77e8294a2b88b31f6f75b2386c84" Nov 01 21:12:34.275159 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:12:34.275132 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d176456610e05c06599d5c7fcf74edec940c77e8294a2b88b31f6f75b2386c84\": container with ID starting with d176456610e05c06599d5c7fcf74edec940c77e8294a2b88b31f6f75b2386c84 not found: ID does not exist" containerID="d176456610e05c06599d5c7fcf74edec940c77e8294a2b88b31f6f75b2386c84" Nov 01 21:12:34.275277 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:34.275169 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d176456610e05c06599d5c7fcf74edec940c77e8294a2b88b31f6f75b2386c84"} err="failed to get container status \"d176456610e05c06599d5c7fcf74edec940c77e8294a2b88b31f6f75b2386c84\": rpc error: code = NotFound desc = could not find container \"d176456610e05c06599d5c7fcf74edec940c77e8294a2b88b31f6f75b2386c84\": container with ID starting with d176456610e05c06599d5c7fcf74edec940c77e8294a2b88b31f6f75b2386c84 not found: ID does not exist" Nov 01 21:12:34.275277 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:34.275189 4484 scope.go:122] "RemoveContainer" containerID="79fad4827ba26033adc4030db87771feea93adc731d7d8114624b186596d43b5" Nov 01 21:12:34.284974 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:34.284956 4484 scope.go:122] "RemoveContainer" containerID="79fad4827ba26033adc4030db87771feea93adc731d7d8114624b186596d43b5" Nov 01 21:12:34.285290 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:12:34.285265 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79fad4827ba26033adc4030db87771feea93adc731d7d8114624b186596d43b5\": container with ID starting with 79fad4827ba26033adc4030db87771feea93adc731d7d8114624b186596d43b5 not found: ID does not exist" containerID="79fad4827ba26033adc4030db87771feea93adc731d7d8114624b186596d43b5" Nov 01 21:12:34.285341 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:34.285300 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79fad4827ba26033adc4030db87771feea93adc731d7d8114624b186596d43b5"} err="failed to get container status \"79fad4827ba26033adc4030db87771feea93adc731d7d8114624b186596d43b5\": rpc error: code = NotFound desc = could not find container \"79fad4827ba26033adc4030db87771feea93adc731d7d8114624b186596d43b5\": container with ID starting with 79fad4827ba26033adc4030db87771feea93adc731d7d8114624b186596d43b5 not found: ID does not exist" Nov 01 21:12:34.430719 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:34.430683 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-deff5e-77db945ff9-jtzp7"] Nov 01 21:12:34.435881 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:34.435853 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-deff5e-77db945ff9-jtzp7"] Nov 01 21:12:34.463885 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:34.463807 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:12:34.551676 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:34.551642 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-deff5e-bd664cbf7-wvr5w"] Nov 01 21:12:34.555201 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:34.555174 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-deff5e-bd664cbf7-wvr5w"] Nov 01 21:12:35.372750 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:35.372714 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d19b68e-4561-480b-bcd4-3cc376211ff3" path="/var/lib/kubelet/pods/8d19b68e-4561-480b-bcd4-3cc376211ff3/volumes" Nov 01 21:12:35.373207 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:35.373192 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ef99ac7-1947-4ffc-98ad-4a167eecba8c" path="/var/lib/kubelet/pods/9ef99ac7-1947-4ffc-98ad-4a167eecba8c/volumes" Nov 01 21:12:39.052722 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:39.052684 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6b9305-6964565579-hwlhv"] Nov 01 21:12:39.164864 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:39.164829 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6b9305-864cc97d9f-dcsdn"] Nov 01 21:12:39.346770 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:39.346678 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:12:39.528026 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:39.525053 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:12:39.678540 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:39.678460 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:12:44.053183 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:44.053113 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-6b9305-6964565579-hwlhv" podUID="64f18742-ed7c-43e2-bd65-1bde7fe2a2c6" containerName="payload-processing" containerID="cri-o://902674b97e02d910395e0957f4baaa91cebf162221f610eadde4d013237772cd" gracePeriod=25 Nov 01 21:12:44.162237 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:44.162167 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:12:44.171704 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:44.171480 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b9305-864cc97d9f-dcsdn" podUID="24468902-76d5-4e4e-b459-8833f95e525d" containerName="payload-pre-processing" containerID="cri-o://8e7e09e3c52d3fe2dda04a21dd1997abe5fd41832f78f1d38cd1943ddc11462d" gracePeriod=25 Nov 01 21:12:44.344636 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:44.344603 4484 generic.go:356] "Generic (PLEG): container finished" podID="64f18742-ed7c-43e2-bd65-1bde7fe2a2c6" containerID="902674b97e02d910395e0957f4baaa91cebf162221f610eadde4d013237772cd" exitCode=0 Nov 01 21:12:44.344815 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:44.344673 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6b9305-6964565579-hwlhv" event={"ID":"64f18742-ed7c-43e2-bd65-1bde7fe2a2c6","Type":"ContainerDied","Data":"902674b97e02d910395e0957f4baaa91cebf162221f610eadde4d013237772cd"} Nov 01 21:12:44.347271 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:44.347239 4484 generic.go:356] "Generic (PLEG): container finished" podID="24468902-76d5-4e4e-b459-8833f95e525d" containerID="8e7e09e3c52d3fe2dda04a21dd1997abe5fd41832f78f1d38cd1943ddc11462d" exitCode=0 Nov 01 21:12:44.347412 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:44.347328 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b9305-864cc97d9f-dcsdn" event={"ID":"24468902-76d5-4e4e-b459-8833f95e525d","Type":"ContainerDied","Data":"8e7e09e3c52d3fe2dda04a21dd1997abe5fd41832f78f1d38cd1943ddc11462d"} Nov 01 21:12:44.369280 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:44.369256 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6b9305-6964565579-hwlhv" Nov 01 21:12:44.466525 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:44.466493 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64f18742-ed7c-43e2-bd65-1bde7fe2a2c6-kube-api-access-j8klh\" (UniqueName: \"kubernetes.io/projected/64f18742-ed7c-43e2-bd65-1bde7fe2a2c6-kube-api-access-j8klh\") pod \"64f18742-ed7c-43e2-bd65-1bde7fe2a2c6\" (UID: \"64f18742-ed7c-43e2-bd65-1bde7fe2a2c6\") " Nov 01 21:12:44.466744 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:44.466726 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64f18742-ed7c-43e2-bd65-1bde7fe2a2c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64f18742-ed7c-43e2-bd65-1bde7fe2a2c6-plugins-config-volume\") pod \"64f18742-ed7c-43e2-bd65-1bde7fe2a2c6\" (UID: \"64f18742-ed7c-43e2-bd65-1bde7fe2a2c6\") " Nov 01 21:12:44.467294 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:44.467238 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64f18742-ed7c-43e2-bd65-1bde7fe2a2c6-plugins-config-volume" pod "64f18742-ed7c-43e2-bd65-1bde7fe2a2c6" (UID: "64f18742-ed7c-43e2-bd65-1bde7fe2a2c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:12:44.469178 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:44.469155 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64f18742-ed7c-43e2-bd65-1bde7fe2a2c6-kube-api-access-j8klh" pod "64f18742-ed7c-43e2-bd65-1bde7fe2a2c6" (UID: "64f18742-ed7c-43e2-bd65-1bde7fe2a2c6"). InnerVolumeSpecName "kube-api-access-j8klh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:12:44.470700 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:44.470683 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b9305-864cc97d9f-dcsdn" Nov 01 21:12:44.568441 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:44.568354 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24468902-76d5-4e4e-b459-8833f95e525d-kube-api-access-jc9nl\" (UniqueName: \"kubernetes.io/projected/24468902-76d5-4e4e-b459-8833f95e525d-kube-api-access-jc9nl\") pod \"24468902-76d5-4e4e-b459-8833f95e525d\" (UID: \"24468902-76d5-4e4e-b459-8833f95e525d\") " Nov 01 21:12:44.568606 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:44.568496 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24468902-76d5-4e4e-b459-8833f95e525d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24468902-76d5-4e4e-b459-8833f95e525d-plugins-config-volume\") pod \"24468902-76d5-4e4e-b459-8833f95e525d\" (UID: \"24468902-76d5-4e4e-b459-8833f95e525d\") " Nov 01 21:12:44.568883 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:44.568855 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24468902-76d5-4e4e-b459-8833f95e525d-plugins-config-volume" pod "24468902-76d5-4e4e-b459-8833f95e525d" (UID: "24468902-76d5-4e4e-b459-8833f95e525d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:12:44.569038 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:44.568967 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8klh\" (UniqueName: \"kubernetes.io/projected/64f18742-ed7c-43e2-bd65-1bde7fe2a2c6-kube-api-access-j8klh\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:12:44.569038 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:44.568980 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64f18742-ed7c-43e2-bd65-1bde7fe2a2c6-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:12:44.569038 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:44.568991 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24468902-76d5-4e4e-b459-8833f95e525d-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:12:44.570518 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:44.570495 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24468902-76d5-4e4e-b459-8833f95e525d-kube-api-access-jc9nl" pod "24468902-76d5-4e4e-b459-8833f95e525d" (UID: "24468902-76d5-4e4e-b459-8833f95e525d"). InnerVolumeSpecName "kube-api-access-jc9nl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:12:44.670389 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:44.670355 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc9nl\" (UniqueName: \"kubernetes.io/projected/24468902-76d5-4e4e-b459-8833f95e525d-kube-api-access-jc9nl\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:12:45.356936 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:45.356846 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6b9305-6964565579-hwlhv" Nov 01 21:12:45.356936 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:45.356878 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6b9305-6964565579-hwlhv" event={"ID":"64f18742-ed7c-43e2-bd65-1bde7fe2a2c6","Type":"ContainerDied","Data":"4084453bc02659c09d2b48d00ae3871eda8b20976873eb9074289bdf27ffb588"} Nov 01 21:12:45.357524 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:45.356944 4484 scope.go:122] "RemoveContainer" containerID="902674b97e02d910395e0957f4baaa91cebf162221f610eadde4d013237772cd" Nov 01 21:12:45.358498 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:45.358477 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b9305-864cc97d9f-dcsdn" Nov 01 21:12:45.358619 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:45.358542 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b9305-864cc97d9f-dcsdn" event={"ID":"24468902-76d5-4e4e-b459-8833f95e525d","Type":"ContainerDied","Data":"6ea842b4c9befd3d305fd831594223b92005b010442269927a5c46823c26ad63"} Nov 01 21:12:45.372062 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:45.372026 4484 scope.go:122] "RemoveContainer" containerID="8e7e09e3c52d3fe2dda04a21dd1997abe5fd41832f78f1d38cd1943ddc11462d" Nov 01 21:12:45.526165 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:45.526126 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6b9305-6964565579-hwlhv"] Nov 01 21:12:45.576137 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:45.576104 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6b9305-6964565579-hwlhv"] Nov 01 21:12:45.686846 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:45.686760 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6b9305-864cc97d9f-dcsdn"] Nov 01 21:12:45.690653 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:45.690625 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6b9305-864cc97d9f-dcsdn"] Nov 01 21:12:45.767520 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:12:45.767484 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:12:47.266719 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:47.266686 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:12:47.373303 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:47.373269 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24468902-76d5-4e4e-b459-8833f95e525d" path="/var/lib/kubelet/pods/24468902-76d5-4e4e-b459-8833f95e525d/volumes" Nov 01 21:12:47.374048 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:47.374029 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64f18742-ed7c-43e2-bd65-1bde7fe2a2c6" path="/var/lib/kubelet/pods/64f18742-ed7c-43e2-bd65-1bde7fe2a2c6/volumes" Nov 01 21:12:48.936648 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:48.936618 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:12:53.083216 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:53.083178 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:12:54.605682 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:54.605645 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:12:54.845289 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:54.845259 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:12:54.997430 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:54.997400 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:12:55.082069 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:55.082035 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3ab520-jfqg5"] Nov 01 21:12:55.132959 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:55.132914 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3ab520-jfqg5"] Nov 01 21:12:55.372809 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:55.372731 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a19f1a9a-b9d2-4ee1-8336-4425230b87b0" path="/var/lib/kubelet/pods/a19f1a9a-b9d2-4ee1-8336-4425230b87b0/volumes" Nov 01 21:12:55.506764 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:55.506721 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:12:56.910386 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:56.910354 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:12:59.410300 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:12:59.410255 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:00.970410 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:00.970376 4484 scope.go:122] "RemoveContainer" containerID="8ac3ed47881de2adbcc17d46222b9e82328111e0fcc6bc407b30aead717af64a" Nov 01 21:13:00.980380 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:00.980358 4484 scope.go:122] "RemoveContainer" containerID="68209950f22a2dd368f2af4e2c4d23b9631553de463a12d28c6e5c1069c9b34f" Nov 01 21:13:00.989672 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:00.989649 4484 scope.go:122] "RemoveContainer" containerID="d35f26d2950ee85f4722a7efcde33d33c9c238b38ae871e4b3af23187002913f" Nov 01 21:13:01.110168 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:01.110135 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:01.264835 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:01.264806 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:01.415809 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:01.415771 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk"] Nov 01 21:13:01.416022 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:01.415982 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk" podUID="2d4f02fb-d74b-40e5-b470-f8d1e65b120c" containerName="istio-proxy" containerID="cri-o://bad8b92d50d5251531869bf9c2ff31eb420058dd763ac270d443dde84fbaec94" gracePeriod=30 Nov 01 21:13:01.426177 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:01.426149 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:01.586077 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:01.585503 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:01.670717 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:01.670679 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4ac1ef-gtngk"] Nov 01 21:13:01.674717 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:01.674689 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4ac1ef-gtngk"] Nov 01 21:13:02.074575 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:02.074539 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:03.151340 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:03.151301 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:03.373870 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:03.373838 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f599828-d9f3-4e90-ba56-1561e8d577e2" path="/var/lib/kubelet/pods/0f599828-d9f3-4e90-ba56-1561e8d577e2/volumes" Nov 01 21:13:03.597939 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:03.597901 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-cd248f-ffrxv"] Nov 01 21:13:03.598725 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:03.598706 4484 cpu_manager.go:414] "containerMap: removing container" podUID="9ef99ac7-1947-4ffc-98ad-4a167eecba8c" containerName="payload-processing" Nov 01 21:13:03.598808 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:03.598726 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ef99ac7-1947-4ffc-98ad-4a167eecba8c" containerName="payload-processing" Nov 01 21:13:03.598808 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:03.598747 4484 cpu_manager.go:414] "containerMap: removing container" podUID="8d19b68e-4561-480b-bcd4-3cc376211ff3" containerName="payload-pre-processing" Nov 01 21:13:03.598808 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:03.598753 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d19b68e-4561-480b-bcd4-3cc376211ff3" containerName="payload-pre-processing" Nov 01 21:13:03.598808 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:03.598774 4484 cpu_manager.go:414] "containerMap: removing container" podUID="64f18742-ed7c-43e2-bd65-1bde7fe2a2c6" containerName="payload-processing" Nov 01 21:13:03.598808 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:03.598779 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64f18742-ed7c-43e2-bd65-1bde7fe2a2c6" containerName="payload-processing" Nov 01 21:13:03.598808 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:03.598786 4484 cpu_manager.go:414] "containerMap: removing container" podUID="32959521-121f-43c1-939d-a0f22e0d1f58" containerName="revoke-keys" Nov 01 21:13:03.598808 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:03.598791 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32959521-121f-43c1-939d-a0f22e0d1f58" containerName="revoke-keys" Nov 01 21:13:03.599155 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:03.598815 4484 cpu_manager.go:414] "containerMap: removing container" podUID="24468902-76d5-4e4e-b459-8833f95e525d" containerName="payload-pre-processing" Nov 01 21:13:03.599155 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:03.598820 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24468902-76d5-4e4e-b459-8833f95e525d" containerName="payload-pre-processing" Nov 01 21:13:03.599155 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:03.598948 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d19b68e-4561-480b-bcd4-3cc376211ff3" containerName="payload-pre-processing" Nov 01 21:13:03.599155 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:03.598962 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="64f18742-ed7c-43e2-bd65-1bde7fe2a2c6" containerName="payload-processing" Nov 01 21:13:03.599155 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:03.598970 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="32959521-121f-43c1-939d-a0f22e0d1f58" containerName="revoke-keys" Nov 01 21:13:03.599155 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:03.598980 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="24468902-76d5-4e4e-b459-8833f95e525d" containerName="payload-pre-processing" Nov 01 21:13:03.599155 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:03.598991 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ef99ac7-1947-4ffc-98ad-4a167eecba8c" containerName="payload-processing" Nov 01 21:13:03.604855 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:03.604833 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-cd248f-ffrxv" Nov 01 21:13:03.658547 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:03.658510 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-cd248f-ffrxv"] Nov 01 21:13:03.702388 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:03.702354 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4339ad7f-a07a-46b9-a9c2-a7d28a3f5cf0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-cd248f-ffrxv\" (UID: \"4339ad7f-a07a-46b9-a9c2-a7d28a3f5cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-cd248f-ffrxv" Nov 01 21:13:03.702621 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:03.702563 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4339ad7f-a07a-46b9-a9c2-a7d28a3f5cf0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-cd248f-ffrxv\" (UID: \"4339ad7f-a07a-46b9-a9c2-a7d28a3f5cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-cd248f-ffrxv" Nov 01 21:13:03.702696 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:03.702682 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25w92\" (UniqueName: \"kubernetes.io/projected/4339ad7f-a07a-46b9-a9c2-a7d28a3f5cf0-kube-api-access-25w92\") pod \"maas-api-revoke-keys-e2e-worker-w2-cd248f-ffrxv\" (UID: \"4339ad7f-a07a-46b9-a9c2-a7d28a3f5cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-cd248f-ffrxv" Nov 01 21:13:03.704398 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:03.704374 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:13:03.704516 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:03.704396 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:13:03.708976 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:03.708953 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:13:03.709091 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:03.709021 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:13:03.804554 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:03.804518 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25w92\" (UniqueName: \"kubernetes.io/projected/4339ad7f-a07a-46b9-a9c2-a7d28a3f5cf0-kube-api-access-25w92\") pod \"maas-api-revoke-keys-e2e-worker-w2-cd248f-ffrxv\" (UID: \"4339ad7f-a07a-46b9-a9c2-a7d28a3f5cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-cd248f-ffrxv" Nov 01 21:13:03.804746 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:03.804633 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4339ad7f-a07a-46b9-a9c2-a7d28a3f5cf0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-cd248f-ffrxv\" (UID: \"4339ad7f-a07a-46b9-a9c2-a7d28a3f5cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-cd248f-ffrxv" Nov 01 21:13:03.804795 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:03.804777 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4339ad7f-a07a-46b9-a9c2-a7d28a3f5cf0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-cd248f-ffrxv\" (UID: \"4339ad7f-a07a-46b9-a9c2-a7d28a3f5cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-cd248f-ffrxv" Nov 01 21:13:03.805484 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:03.805451 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4339ad7f-a07a-46b9-a9c2-a7d28a3f5cf0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-cd248f-ffrxv\" (UID: \"4339ad7f-a07a-46b9-a9c2-a7d28a3f5cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-cd248f-ffrxv" Nov 01 21:13:03.807072 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:03.807049 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4339ad7f-a07a-46b9-a9c2-a7d28a3f5cf0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-cd248f-ffrxv\" (UID: \"4339ad7f-a07a-46b9-a9c2-a7d28a3f5cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-cd248f-ffrxv" Nov 01 21:13:03.861729 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:03.861649 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25w92\" (UniqueName: \"kubernetes.io/projected/4339ad7f-a07a-46b9-a9c2-a7d28a3f5cf0-kube-api-access-25w92\") pod \"maas-api-revoke-keys-e2e-worker-w2-cd248f-ffrxv\" (UID: \"4339ad7f-a07a-46b9-a9c2-a7d28a3f5cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-cd248f-ffrxv" Nov 01 21:13:03.919878 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:03.919841 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-cd248f-ffrxv" Nov 01 21:13:04.296247 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:04.296214 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-cd248f-ffrxv"] Nov 01 21:13:04.296614 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:13:04.296272 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4339ad7f_a07a_46b9_a9c2_a7d28a3f5cf0.slice/crio-8fa488f5e605ba1438236257fb04966c3b2948e1678e32889bf563050854408d WatchSource:0}: Error finding container 8fa488f5e605ba1438236257fb04966c3b2948e1678e32889bf563050854408d: Status 404 returned error can't find the container with id 8fa488f5e605ba1438236257fb04966c3b2948e1678e32889bf563050854408d Nov 01 21:13:04.572936 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:04.572840 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-cd248f-ffrxv" event={"ID":"4339ad7f-a07a-46b9-a9c2-a7d28a3f5cf0","Type":"ContainerStarted","Data":"8fa488f5e605ba1438236257fb04966c3b2948e1678e32889bf563050854408d"} Nov 01 21:13:05.582031 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:05.581982 4484 generic.go:356] "Generic (PLEG): container finished" podID="4339ad7f-a07a-46b9-a9c2-a7d28a3f5cf0" containerID="dc1c3c34e21cbbfb2932534448e3e950c228f511dc50741739e51bc79f550cc3" exitCode=0 Nov 01 21:13:05.582421 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:05.582066 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-cd248f-ffrxv" event={"ID":"4339ad7f-a07a-46b9-a9c2-a7d28a3f5cf0","Type":"ContainerDied","Data":"dc1c3c34e21cbbfb2932534448e3e950c228f511dc50741739e51bc79f550cc3"} Nov 01 21:13:06.400118 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:06.400083 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:06.592906 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:06.592874 4484 generic.go:356] "Generic (PLEG): container finished" podID="2d4f02fb-d74b-40e5-b470-f8d1e65b120c" containerID="bad8b92d50d5251531869bf9c2ff31eb420058dd763ac270d443dde84fbaec94" exitCode=0 Nov 01 21:13:06.593362 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:06.592942 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk" event={"ID":"2d4f02fb-d74b-40e5-b470-f8d1e65b120c","Type":"ContainerDied","Data":"bad8b92d50d5251531869bf9c2ff31eb420058dd763ac270d443dde84fbaec94"} Nov 01 21:13:06.593362 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:06.593033 4484 scope.go:122] "RemoveContainer" containerID="ca2e831ab9700ed80a68b6bf74bd5f3b11c7f2b7bc066043094ff2be8b087e49" Nov 01 21:13:06.734856 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:06.734820 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk" Nov 01 21:13:06.802662 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:06.802637 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-cd248f-ffrxv" Nov 01 21:13:06.846179 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:06.846152 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4339ad7f-a07a-46b9-a9c2-a7d28a3f5cf0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4339ad7f-a07a-46b9-a9c2-a7d28a3f5cf0-maas-api-openshift-ca\") pod \"4339ad7f-a07a-46b9-a9c2-a7d28a3f5cf0\" (UID: \"4339ad7f-a07a-46b9-a9c2-a7d28a3f5cf0\") " Nov 01 21:13:06.846375 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:06.846252 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-workload-socket\") pod \"2d4f02fb-d74b-40e5-b470-f8d1e65b120c\" (UID: \"2d4f02fb-d74b-40e5-b470-f8d1e65b120c\") " Nov 01 21:13:06.846375 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:06.846287 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-istio-podinfo\") pod \"2d4f02fb-d74b-40e5-b470-f8d1e65b120c\" (UID: \"2d4f02fb-d74b-40e5-b470-f8d1e65b120c\") " Nov 01 21:13:06.846375 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:06.846320 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-istio-data\") pod \"2d4f02fb-d74b-40e5-b470-f8d1e65b120c\" (UID: \"2d4f02fb-d74b-40e5-b470-f8d1e65b120c\") " Nov 01 21:13:06.846375 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:06.846347 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4339ad7f-a07a-46b9-a9c2-a7d28a3f5cf0-kube-api-access-25w92\" (UniqueName: \"kubernetes.io/projected/4339ad7f-a07a-46b9-a9c2-a7d28a3f5cf0-kube-api-access-25w92\") pod \"4339ad7f-a07a-46b9-a9c2-a7d28a3f5cf0\" (UID: \"4339ad7f-a07a-46b9-a9c2-a7d28a3f5cf0\") " Nov 01 21:13:06.846628 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:06.846388 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-istio-token\" (UniqueName: \"kubernetes.io/projected/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-istio-token\") pod \"2d4f02fb-d74b-40e5-b470-f8d1e65b120c\" (UID: \"2d4f02fb-d74b-40e5-b470-f8d1e65b120c\") " Nov 01 21:13:06.846628 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:06.846436 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-istio-envoy\") pod \"2d4f02fb-d74b-40e5-b470-f8d1e65b120c\" (UID: \"2d4f02fb-d74b-40e5-b470-f8d1e65b120c\") " Nov 01 21:13:06.846628 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:06.846465 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-kube-api-access-klhsm\" (UniqueName: \"kubernetes.io/projected/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-kube-api-access-klhsm\") pod \"2d4f02fb-d74b-40e5-b470-f8d1e65b120c\" (UID: \"2d4f02fb-d74b-40e5-b470-f8d1e65b120c\") " Nov 01 21:13:06.846628 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:06.846498 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-workload-certs\") pod \"2d4f02fb-d74b-40e5-b470-f8d1e65b120c\" (UID: \"2d4f02fb-d74b-40e5-b470-f8d1e65b120c\") " Nov 01 21:13:06.846628 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:06.846552 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4339ad7f-a07a-46b9-a9c2-a7d28a3f5cf0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4339ad7f-a07a-46b9-a9c2-a7d28a3f5cf0-maas-api-xks-ca\") pod \"4339ad7f-a07a-46b9-a9c2-a7d28a3f5cf0\" (UID: \"4339ad7f-a07a-46b9-a9c2-a7d28a3f5cf0\") " Nov 01 21:13:06.846628 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:06.846595 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-istiod-ca-cert\") pod \"2d4f02fb-d74b-40e5-b470-f8d1e65b120c\" (UID: \"2d4f02fb-d74b-40e5-b470-f8d1e65b120c\") " Nov 01 21:13:06.846628 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:06.846592 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-workload-socket" pod "2d4f02fb-d74b-40e5-b470-f8d1e65b120c" (UID: "2d4f02fb-d74b-40e5-b470-f8d1e65b120c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:13:06.846981 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:06.846642 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4339ad7f-a07a-46b9-a9c2-a7d28a3f5cf0-maas-api-openshift-ca" pod "4339ad7f-a07a-46b9-a9c2-a7d28a3f5cf0" (UID: "4339ad7f-a07a-46b9-a9c2-a7d28a3f5cf0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:13:06.846981 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:06.846863 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-credential-socket\") pod \"2d4f02fb-d74b-40e5-b470-f8d1e65b120c\" (UID: \"2d4f02fb-d74b-40e5-b470-f8d1e65b120c\") " Nov 01 21:13:06.846981 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:06.846881 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-workload-certs" pod "2d4f02fb-d74b-40e5-b470-f8d1e65b120c" (UID: "2d4f02fb-d74b-40e5-b470-f8d1e65b120c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:13:06.846981 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:06.846645 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-istio-data" pod "2d4f02fb-d74b-40e5-b470-f8d1e65b120c" (UID: "2d4f02fb-d74b-40e5-b470-f8d1e65b120c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:13:06.846981 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:06.846937 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-istiod-ca-cert" pod "2d4f02fb-d74b-40e5-b470-f8d1e65b120c" (UID: "2d4f02fb-d74b-40e5-b470-f8d1e65b120c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:13:06.847236 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:06.847181 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-credential-socket" pod "2d4f02fb-d74b-40e5-b470-f8d1e65b120c" (UID: "2d4f02fb-d74b-40e5-b470-f8d1e65b120c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:13:06.847642 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:06.847603 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:06.847642 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:06.847630 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4339ad7f-a07a-46b9-a9c2-a7d28a3f5cf0-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:06.847902 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:06.847648 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:06.847902 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:06.847662 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:06.847902 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:06.847677 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:06.847902 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:06.847693 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:06.848882 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:06.848855 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4339ad7f-a07a-46b9-a9c2-a7d28a3f5cf0-maas-api-xks-ca" pod "4339ad7f-a07a-46b9-a9c2-a7d28a3f5cf0" (UID: "4339ad7f-a07a-46b9-a9c2-a7d28a3f5cf0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:13:06.849193 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:06.849175 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-kube-api-access-klhsm" pod "2d4f02fb-d74b-40e5-b470-f8d1e65b120c" (UID: "2d4f02fb-d74b-40e5-b470-f8d1e65b120c"). InnerVolumeSpecName "kube-api-access-klhsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:13:06.849268 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:06.849243 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-istio-token" pod "2d4f02fb-d74b-40e5-b470-f8d1e65b120c" (UID: "2d4f02fb-d74b-40e5-b470-f8d1e65b120c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:13:06.849314 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:06.849294 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-istio-podinfo" pod "2d4f02fb-d74b-40e5-b470-f8d1e65b120c" (UID: "2d4f02fb-d74b-40e5-b470-f8d1e65b120c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:13:06.849365 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:06.849347 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-istio-envoy" pod "2d4f02fb-d74b-40e5-b470-f8d1e65b120c" (UID: "2d4f02fb-d74b-40e5-b470-f8d1e65b120c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:13:06.849404 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:06.849356 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4339ad7f-a07a-46b9-a9c2-a7d28a3f5cf0-kube-api-access-25w92" pod "4339ad7f-a07a-46b9-a9c2-a7d28a3f5cf0" (UID: "4339ad7f-a07a-46b9-a9c2-a7d28a3f5cf0"). InnerVolumeSpecName "kube-api-access-25w92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:13:06.949567 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:06.949480 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25w92\" (UniqueName: \"kubernetes.io/projected/4339ad7f-a07a-46b9-a9c2-a7d28a3f5cf0-kube-api-access-25w92\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:06.949567 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:06.949514 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:06.949567 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:06.949526 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:06.949567 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:06.949535 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klhsm\" (UniqueName: \"kubernetes.io/projected/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-kube-api-access-klhsm\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:06.949567 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:06.949544 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4339ad7f-a07a-46b9-a9c2-a7d28a3f5cf0-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:06.949567 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:06.949552 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d4f02fb-d74b-40e5-b470-f8d1e65b120c-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:07.601324 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:07.601295 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk" Nov 01 21:13:07.601807 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:07.601320 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk" event={"ID":"2d4f02fb-d74b-40e5-b470-f8d1e65b120c","Type":"ContainerDied","Data":"da06263bdf5430453760dbe738495cea3896b8e9abe99d832ef89c9e1cf118c4"} Nov 01 21:13:07.601807 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:07.601377 4484 scope.go:122] "RemoveContainer" containerID="bad8b92d50d5251531869bf9c2ff31eb420058dd763ac270d443dde84fbaec94" Nov 01 21:13:07.602841 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:07.602821 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-cd248f-ffrxv" Nov 01 21:13:07.602925 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:07.602855 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-cd248f-ffrxv" event={"ID":"4339ad7f-a07a-46b9-a9c2-a7d28a3f5cf0","Type":"ContainerDied","Data":"8fa488f5e605ba1438236257fb04966c3b2948e1678e32889bf563050854408d"} Nov 01 21:13:07.602925 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:07.602901 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8fa488f5e605ba1438236257fb04966c3b2948e1678e32889bf563050854408d" Nov 01 21:13:07.735227 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:07.735146 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:07.803742 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:07.803699 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk"] Nov 01 21:13:07.805579 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:07.805551 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-3ab520-openshift-default-6765b98db9-jrzgk"] Nov 01 21:13:07.910464 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:07.910432 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:08.070880 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:08.070845 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:08.150019 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:08.141629 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4"] Nov 01 21:13:08.232236 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:08.232199 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:08.822522 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:08.822483 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7091fd-24flr"] Nov 01 21:13:08.823201 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:08.823184 4484 cpu_manager.go:414] "containerMap: removing container" podUID="4339ad7f-a07a-46b9-a9c2-a7d28a3f5cf0" containerName="revoke-keys" Nov 01 21:13:08.823259 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:08.823201 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4339ad7f-a07a-46b9-a9c2-a7d28a3f5cf0" containerName="revoke-keys" Nov 01 21:13:08.823259 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:08.823233 4484 cpu_manager.go:414] "containerMap: removing container" podUID="2d4f02fb-d74b-40e5-b470-f8d1e65b120c" containerName="istio-proxy" Nov 01 21:13:08.823259 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:08.823238 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d4f02fb-d74b-40e5-b470-f8d1e65b120c" containerName="istio-proxy" Nov 01 21:13:08.823259 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:08.823251 4484 cpu_manager.go:414] "containerMap: removing container" podUID="2d4f02fb-d74b-40e5-b470-f8d1e65b120c" containerName="istio-proxy" Nov 01 21:13:08.823259 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:08.823256 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d4f02fb-d74b-40e5-b470-f8d1e65b120c" containerName="istio-proxy" Nov 01 21:13:08.823776 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:08.823353 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="4339ad7f-a07a-46b9-a9c2-a7d28a3f5cf0" containerName="revoke-keys" Nov 01 21:13:08.823776 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:08.823367 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d4f02fb-d74b-40e5-b470-f8d1e65b120c" containerName="istio-proxy" Nov 01 21:13:08.823776 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:08.823374 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d4f02fb-d74b-40e5-b470-f8d1e65b120c" containerName="istio-proxy" Nov 01 21:13:08.828727 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:08.828708 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7091fd-24flr" Nov 01 21:13:08.882958 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:08.882923 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7091fd-24flr"] Nov 01 21:13:08.883826 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:08.883296 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:13:08.883826 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:08.883321 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:13:08.883826 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:08.883393 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:13:08.888493 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:08.888458 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7lgk\" (UniqueName: \"kubernetes.io/projected/2f2370cd-fd08-456b-a312-0b648cd4ebea-kube-api-access-n7lgk\") pod \"maas-api-revoke-keys-e2e-inf-b-7091fd-24flr\" (UID: \"2f2370cd-fd08-456b-a312-0b648cd4ebea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7091fd-24flr" Nov 01 21:13:08.888692 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:08.888669 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f2370cd-fd08-456b-a312-0b648cd4ebea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7091fd-24flr\" (UID: \"2f2370cd-fd08-456b-a312-0b648cd4ebea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7091fd-24flr" Nov 01 21:13:08.888761 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:08.888732 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f2370cd-fd08-456b-a312-0b648cd4ebea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7091fd-24flr\" (UID: \"2f2370cd-fd08-456b-a312-0b648cd4ebea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7091fd-24flr" Nov 01 21:13:08.928213 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:08.928177 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:13:08.990378 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:08.990340 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f2370cd-fd08-456b-a312-0b648cd4ebea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7091fd-24flr\" (UID: \"2f2370cd-fd08-456b-a312-0b648cd4ebea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7091fd-24flr" Nov 01 21:13:08.990576 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:08.990405 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f2370cd-fd08-456b-a312-0b648cd4ebea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7091fd-24flr\" (UID: \"2f2370cd-fd08-456b-a312-0b648cd4ebea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7091fd-24flr" Nov 01 21:13:08.990576 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:08.990532 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7lgk\" (UniqueName: \"kubernetes.io/projected/2f2370cd-fd08-456b-a312-0b648cd4ebea-kube-api-access-n7lgk\") pod \"maas-api-revoke-keys-e2e-inf-b-7091fd-24flr\" (UID: \"2f2370cd-fd08-456b-a312-0b648cd4ebea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7091fd-24flr" Nov 01 21:13:08.991075 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:08.990991 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f2370cd-fd08-456b-a312-0b648cd4ebea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7091fd-24flr\" (UID: \"2f2370cd-fd08-456b-a312-0b648cd4ebea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7091fd-24flr" Nov 01 21:13:08.992768 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:08.992749 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f2370cd-fd08-456b-a312-0b648cd4ebea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7091fd-24flr\" (UID: \"2f2370cd-fd08-456b-a312-0b648cd4ebea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7091fd-24flr" Nov 01 21:13:09.049688 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:09.049653 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7lgk\" (UniqueName: \"kubernetes.io/projected/2f2370cd-fd08-456b-a312-0b648cd4ebea-kube-api-access-n7lgk\") pod \"maas-api-revoke-keys-e2e-inf-b-7091fd-24flr\" (UID: \"2f2370cd-fd08-456b-a312-0b648cd4ebea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7091fd-24flr" Nov 01 21:13:09.142627 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:09.142528 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7091fd-24flr" Nov 01 21:13:09.382989 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:09.382953 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d4f02fb-d74b-40e5-b470-f8d1e65b120c" path="/var/lib/kubelet/pods/2d4f02fb-d74b-40e5-b470-f8d1e65b120c/volumes" Nov 01 21:13:09.521640 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:09.521616 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7091fd-24flr"] Nov 01 21:13:09.522033 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:13:09.522008 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f2370cd_fd08_456b_a312_0b648cd4ebea.slice/crio-d59adb9cd453ba6b4d40fc50e8655eb9803cb34b0ce0b5a8695185cf8f316042 WatchSource:0}: Error finding container d59adb9cd453ba6b4d40fc50e8655eb9803cb34b0ce0b5a8695185cf8f316042: Status 404 returned error can't find the container with id d59adb9cd453ba6b4d40fc50e8655eb9803cb34b0ce0b5a8695185cf8f316042 Nov 01 21:13:09.619390 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:09.619354 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7091fd-24flr" event={"ID":"2f2370cd-fd08-456b-a312-0b648cd4ebea","Type":"ContainerStarted","Data":"d59adb9cd453ba6b4d40fc50e8655eb9803cb34b0ce0b5a8695185cf8f316042"} Nov 01 21:13:10.628661 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:10.628626 4484 generic.go:356] "Generic (PLEG): container finished" podID="2f2370cd-fd08-456b-a312-0b648cd4ebea" containerID="edd06cad245f511642863abf4ca35484a6eaf6647204b9f2f26b9f9fab41c1ca" exitCode=0 Nov 01 21:13:10.629113 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:10.628711 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7091fd-24flr" event={"ID":"2f2370cd-fd08-456b-a312-0b648cd4ebea","Type":"ContainerDied","Data":"edd06cad245f511642863abf4ca35484a6eaf6647204b9f2f26b9f9fab41c1ca"} Nov 01 21:13:11.816877 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:11.816852 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7091fd-24flr" Nov 01 21:13:11.819845 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:11.819815 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:11.929193 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:11.929109 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f2370cd-fd08-456b-a312-0b648cd4ebea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f2370cd-fd08-456b-a312-0b648cd4ebea-maas-api-xks-ca\") pod \"2f2370cd-fd08-456b-a312-0b648cd4ebea\" (UID: \"2f2370cd-fd08-456b-a312-0b648cd4ebea\") " Nov 01 21:13:11.929345 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:11.929314 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f2370cd-fd08-456b-a312-0b648cd4ebea-kube-api-access-n7lgk\" (UniqueName: \"kubernetes.io/projected/2f2370cd-fd08-456b-a312-0b648cd4ebea-kube-api-access-n7lgk\") pod \"2f2370cd-fd08-456b-a312-0b648cd4ebea\" (UID: \"2f2370cd-fd08-456b-a312-0b648cd4ebea\") " Nov 01 21:13:11.929384 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:11.929368 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f2370cd-fd08-456b-a312-0b648cd4ebea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f2370cd-fd08-456b-a312-0b648cd4ebea-maas-api-openshift-ca\") pod \"2f2370cd-fd08-456b-a312-0b648cd4ebea\" (UID: \"2f2370cd-fd08-456b-a312-0b648cd4ebea\") " Nov 01 21:13:11.929764 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:11.929734 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f2370cd-fd08-456b-a312-0b648cd4ebea-maas-api-openshift-ca" pod "2f2370cd-fd08-456b-a312-0b648cd4ebea" (UID: "2f2370cd-fd08-456b-a312-0b648cd4ebea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:13:11.931269 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:11.931242 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f2370cd-fd08-456b-a312-0b648cd4ebea-maas-api-xks-ca" pod "2f2370cd-fd08-456b-a312-0b648cd4ebea" (UID: "2f2370cd-fd08-456b-a312-0b648cd4ebea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:13:11.931438 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:11.931414 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f2370cd-fd08-456b-a312-0b648cd4ebea-kube-api-access-n7lgk" pod "2f2370cd-fd08-456b-a312-0b648cd4ebea" (UID: "2f2370cd-fd08-456b-a312-0b648cd4ebea"). InnerVolumeSpecName "kube-api-access-n7lgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:13:12.030893 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:12.030846 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7lgk\" (UniqueName: \"kubernetes.io/projected/2f2370cd-fd08-456b-a312-0b648cd4ebea-kube-api-access-n7lgk\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:12.030893 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:12.030886 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f2370cd-fd08-456b-a312-0b648cd4ebea-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:12.030893 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:12.030897 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f2370cd-fd08-456b-a312-0b648cd4ebea-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:12.645430 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:12.645397 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7091fd-24flr" event={"ID":"2f2370cd-fd08-456b-a312-0b648cd4ebea","Type":"ContainerDied","Data":"d59adb9cd453ba6b4d40fc50e8655eb9803cb34b0ce0b5a8695185cf8f316042"} Nov 01 21:13:12.645430 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:12.645420 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7091fd-24flr" Nov 01 21:13:12.645656 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:12.645442 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d59adb9cd453ba6b4d40fc50e8655eb9803cb34b0ce0b5a8695185cf8f316042" Nov 01 21:13:15.102251 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:15.102211 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-cd248f-6c9889fc9c-wbd52"] Nov 01 21:13:15.499171 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:15.499120 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:15.685430 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:15.685391 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:15.768284 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:13:15.768204 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:13:17.154631 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:17.154597 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:20.103512 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:20.103461 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-cd248f-6c9889fc9c-wbd52" podUID="69c9a660-98c7-436f-8bee-b7781c596d0c" containerName="payload-processing" containerID="cri-o://79c6672bdfe0e3dca2fa9c24bfa462019ee8f8926e053b6e218e88358fb61e76" gracePeriod=25 Nov 01 21:13:20.208285 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:20.208248 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-cd248f-6c9889fc9c-wbd52" podUID="69c9a660-98c7-436f-8bee-b7781c596d0c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 21:13:21.331721 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:21.331678 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7091fd-98896dbf7-6r5f6"] Nov 01 21:13:21.443895 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:21.443857 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7091fd-6d79589dd7-mqf74"] Nov 01 21:13:21.798121 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:21.798079 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:21.954873 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:21.954836 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:22.114745 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:22.114663 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:23.205344 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:23.205283 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" podUID="311f9699-6585-4a2c-8c20-e6b696b2f857" containerName="main" containerID="cri-o://d79b4d0528c4d3b99396b7d9bbd4d9684197f2f16be9d8007a6081ae2f669555" gracePeriod=45 Nov 01 21:13:23.269335 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:23.269299 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" podUID="311f9699-6585-4a2c-8c20-e6b696b2f857" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": dial tcp 10.134.0.54:8000: connect: connection refused" Nov 01 21:13:23.497184 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:23.497156 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" Nov 01 21:13:23.550040 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:23.549981 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/311f9699-6585-4a2c-8c20-e6b696b2f857-tls-certs\" (UniqueName: \"kubernetes.io/secret/311f9699-6585-4a2c-8c20-e6b696b2f857-tls-certs\") pod \"311f9699-6585-4a2c-8c20-e6b696b2f857\" (UID: \"311f9699-6585-4a2c-8c20-e6b696b2f857\") " Nov 01 21:13:23.550203 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:23.550096 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/311f9699-6585-4a2c-8c20-e6b696b2f857-home\" (UniqueName: \"kubernetes.io/empty-dir/311f9699-6585-4a2c-8c20-e6b696b2f857-home\") pod \"311f9699-6585-4a2c-8c20-e6b696b2f857\" (UID: \"311f9699-6585-4a2c-8c20-e6b696b2f857\") " Nov 01 21:13:23.550203 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:23.550141 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/311f9699-6585-4a2c-8c20-e6b696b2f857-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/311f9699-6585-4a2c-8c20-e6b696b2f857-tmp-dir\") pod \"311f9699-6585-4a2c-8c20-e6b696b2f857\" (UID: \"311f9699-6585-4a2c-8c20-e6b696b2f857\") " Nov 01 21:13:23.550361 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:23.550215 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/311f9699-6585-4a2c-8c20-e6b696b2f857-dshm\" (UniqueName: \"kubernetes.io/empty-dir/311f9699-6585-4a2c-8c20-e6b696b2f857-dshm\") pod \"311f9699-6585-4a2c-8c20-e6b696b2f857\" (UID: \"311f9699-6585-4a2c-8c20-e6b696b2f857\") " Nov 01 21:13:23.550361 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:23.550279 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/311f9699-6585-4a2c-8c20-e6b696b2f857-kube-api-access-2qx9m\" (UniqueName: \"kubernetes.io/projected/311f9699-6585-4a2c-8c20-e6b696b2f857-kube-api-access-2qx9m\") pod \"311f9699-6585-4a2c-8c20-e6b696b2f857\" (UID: \"311f9699-6585-4a2c-8c20-e6b696b2f857\") " Nov 01 21:13:23.550361 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:23.550335 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/311f9699-6585-4a2c-8c20-e6b696b2f857-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/311f9699-6585-4a2c-8c20-e6b696b2f857-model-cache\") pod \"311f9699-6585-4a2c-8c20-e6b696b2f857\" (UID: \"311f9699-6585-4a2c-8c20-e6b696b2f857\") " Nov 01 21:13:23.550506 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:23.550393 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/311f9699-6585-4a2c-8c20-e6b696b2f857-home" pod "311f9699-6585-4a2c-8c20-e6b696b2f857" (UID: "311f9699-6585-4a2c-8c20-e6b696b2f857"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:13:23.550506 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:23.550443 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/311f9699-6585-4a2c-8c20-e6b696b2f857-tmp-dir" pod "311f9699-6585-4a2c-8c20-e6b696b2f857" (UID: "311f9699-6585-4a2c-8c20-e6b696b2f857"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:13:23.550693 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:23.550670 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/311f9699-6585-4a2c-8c20-e6b696b2f857-model-cache" pod "311f9699-6585-4a2c-8c20-e6b696b2f857" (UID: "311f9699-6585-4a2c-8c20-e6b696b2f857"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:13:23.550940 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:23.550918 4484 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/311f9699-6585-4a2c-8c20-e6b696b2f857-home\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:23.550940 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:23.550940 4484 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/311f9699-6585-4a2c-8c20-e6b696b2f857-tmp-dir\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:23.550940 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:23.550951 4484 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/311f9699-6585-4a2c-8c20-e6b696b2f857-model-cache\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:23.552431 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:23.552408 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/311f9699-6585-4a2c-8c20-e6b696b2f857-dshm" pod "311f9699-6585-4a2c-8c20-e6b696b2f857" (UID: "311f9699-6585-4a2c-8c20-e6b696b2f857"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:13:23.552542 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:23.552481 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/311f9699-6585-4a2c-8c20-e6b696b2f857-kube-api-access-2qx9m" pod "311f9699-6585-4a2c-8c20-e6b696b2f857" (UID: "311f9699-6585-4a2c-8c20-e6b696b2f857"). InnerVolumeSpecName "kube-api-access-2qx9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:13:23.552542 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:23.552496 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/311f9699-6585-4a2c-8c20-e6b696b2f857-tls-certs" pod "311f9699-6585-4a2c-8c20-e6b696b2f857" (UID: "311f9699-6585-4a2c-8c20-e6b696b2f857"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:13:23.652523 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:23.652473 4484 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/311f9699-6585-4a2c-8c20-e6b696b2f857-tls-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:23.652523 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:23.652518 4484 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/311f9699-6585-4a2c-8c20-e6b696b2f857-dshm\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:23.652774 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:23.652535 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qx9m\" (UniqueName: \"kubernetes.io/projected/311f9699-6585-4a2c-8c20-e6b696b2f857-kube-api-access-2qx9m\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:23.878396 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:23.878358 4484 generic.go:356] "Generic (PLEG): container finished" podID="311f9699-6585-4a2c-8c20-e6b696b2f857" containerID="d79b4d0528c4d3b99396b7d9bbd4d9684197f2f16be9d8007a6081ae2f669555" exitCode=0 Nov 01 21:13:23.878603 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:23.878453 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" Nov 01 21:13:23.878603 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:23.878471 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" event={"ID":"311f9699-6585-4a2c-8c20-e6b696b2f857","Type":"ContainerDied","Data":"d79b4d0528c4d3b99396b7d9bbd4d9684197f2f16be9d8007a6081ae2f669555"} Nov 01 21:13:23.878603 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:23.878537 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4" event={"ID":"311f9699-6585-4a2c-8c20-e6b696b2f857","Type":"ContainerDied","Data":"3811d2e3366588338bf66dc607ab50825f95a30b5da36c64290f6c96b9e54cfa"} Nov 01 21:13:23.878603 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:23.878562 4484 scope.go:122] "RemoveContainer" containerID="d79b4d0528c4d3b99396b7d9bbd4d9684197f2f16be9d8007a6081ae2f669555" Nov 01 21:13:23.891330 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:23.891312 4484 scope.go:122] "RemoveContainer" containerID="d79b4d0528c4d3b99396b7d9bbd4d9684197f2f16be9d8007a6081ae2f669555" Nov 01 21:13:23.891637 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:13:23.891618 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d79b4d0528c4d3b99396b7d9bbd4d9684197f2f16be9d8007a6081ae2f669555\": container with ID starting with d79b4d0528c4d3b99396b7d9bbd4d9684197f2f16be9d8007a6081ae2f669555 not found: ID does not exist" containerID="d79b4d0528c4d3b99396b7d9bbd4d9684197f2f16be9d8007a6081ae2f669555" Nov 01 21:13:23.891719 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:23.891645 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d79b4d0528c4d3b99396b7d9bbd4d9684197f2f16be9d8007a6081ae2f669555"} err="failed to get container status \"d79b4d0528c4d3b99396b7d9bbd4d9684197f2f16be9d8007a6081ae2f669555\": rpc error: code = NotFound desc = could not find container \"d79b4d0528c4d3b99396b7d9bbd4d9684197f2f16be9d8007a6081ae2f669555\": container with ID starting with d79b4d0528c4d3b99396b7d9bbd4d9684197f2f16be9d8007a6081ae2f669555 not found: ID does not exist" Nov 01 21:13:24.048036 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:24.047984 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4"] Nov 01 21:13:24.051368 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:24.051345 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-cd248f/premium-w2-cd248f-kserve-66ddfc4c45-l74x4"] Nov 01 21:13:25.372815 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:25.372785 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="311f9699-6585-4a2c-8c20-e6b696b2f857" path="/var/lib/kubelet/pods/311f9699-6585-4a2c-8c20-e6b696b2f857/volumes" Nov 01 21:13:26.142580 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:26.142545 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:26.332504 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:26.332436 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-7091fd-98896dbf7-6r5f6" podUID="ab517a39-c7e7-4567-b294-484c21d3d799" containerName="payload-processing" containerID="cri-o://deaa910fc4ac76c40cf15cd323d28e09e38e14f8df02fb6640b538bf332b1750" gracePeriod=25 Nov 01 21:13:26.447094 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:26.446129 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7091fd-6d79589dd7-mqf74" podUID="e127ff65-ebee-4f72-952b-f4464d7592f9" containerName="payload-pre-processing" containerID="cri-o://2c65b50f57384b1b87b8389593422b84d76a17447bb9470b2bdafb57539d733f" gracePeriod=25 Nov 01 21:13:26.677103 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:26.677074 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7091fd-98896dbf7-6r5f6" Nov 01 21:13:26.709693 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:26.709607 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab517a39-c7e7-4567-b294-484c21d3d799-kube-api-access-dphzd\" (UniqueName: \"kubernetes.io/projected/ab517a39-c7e7-4567-b294-484c21d3d799-kube-api-access-dphzd\") pod \"ab517a39-c7e7-4567-b294-484c21d3d799\" (UID: \"ab517a39-c7e7-4567-b294-484c21d3d799\") " Nov 01 21:13:26.709843 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:26.709691 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab517a39-c7e7-4567-b294-484c21d3d799-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab517a39-c7e7-4567-b294-484c21d3d799-plugins-config-volume\") pod \"ab517a39-c7e7-4567-b294-484c21d3d799\" (UID: \"ab517a39-c7e7-4567-b294-484c21d3d799\") " Nov 01 21:13:26.710297 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:26.710262 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab517a39-c7e7-4567-b294-484c21d3d799-plugins-config-volume" pod "ab517a39-c7e7-4567-b294-484c21d3d799" (UID: "ab517a39-c7e7-4567-b294-484c21d3d799"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:13:26.712250 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:26.712221 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab517a39-c7e7-4567-b294-484c21d3d799-kube-api-access-dphzd" pod "ab517a39-c7e7-4567-b294-484c21d3d799" (UID: "ab517a39-c7e7-4567-b294-484c21d3d799"). InnerVolumeSpecName "kube-api-access-dphzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:13:26.784927 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:26.784897 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7091fd-6d79589dd7-mqf74" Nov 01 21:13:26.810859 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:26.810824 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e127ff65-ebee-4f72-952b-f4464d7592f9-kube-api-access-lbmpq\" (UniqueName: \"kubernetes.io/projected/e127ff65-ebee-4f72-952b-f4464d7592f9-kube-api-access-lbmpq\") pod \"e127ff65-ebee-4f72-952b-f4464d7592f9\" (UID: \"e127ff65-ebee-4f72-952b-f4464d7592f9\") " Nov 01 21:13:26.811080 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:26.810870 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e127ff65-ebee-4f72-952b-f4464d7592f9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e127ff65-ebee-4f72-952b-f4464d7592f9-plugins-config-volume\") pod \"e127ff65-ebee-4f72-952b-f4464d7592f9\" (UID: \"e127ff65-ebee-4f72-952b-f4464d7592f9\") " Nov 01 21:13:26.811421 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:26.811389 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e127ff65-ebee-4f72-952b-f4464d7592f9-plugins-config-volume" pod "e127ff65-ebee-4f72-952b-f4464d7592f9" (UID: "e127ff65-ebee-4f72-952b-f4464d7592f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:13:26.811528 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:26.811446 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dphzd\" (UniqueName: \"kubernetes.io/projected/ab517a39-c7e7-4567-b294-484c21d3d799-kube-api-access-dphzd\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:26.811528 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:26.811462 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab517a39-c7e7-4567-b294-484c21d3d799-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:26.813208 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:26.813176 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e127ff65-ebee-4f72-952b-f4464d7592f9-kube-api-access-lbmpq" pod "e127ff65-ebee-4f72-952b-f4464d7592f9" (UID: "e127ff65-ebee-4f72-952b-f4464d7592f9"). InnerVolumeSpecName "kube-api-access-lbmpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:13:26.901606 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:26.901574 4484 generic.go:356] "Generic (PLEG): container finished" podID="ab517a39-c7e7-4567-b294-484c21d3d799" containerID="deaa910fc4ac76c40cf15cd323d28e09e38e14f8df02fb6640b538bf332b1750" exitCode=0 Nov 01 21:13:26.901801 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:26.901621 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7091fd-98896dbf7-6r5f6" Nov 01 21:13:26.901801 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:26.901655 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7091fd-98896dbf7-6r5f6" event={"ID":"ab517a39-c7e7-4567-b294-484c21d3d799","Type":"ContainerDied","Data":"deaa910fc4ac76c40cf15cd323d28e09e38e14f8df02fb6640b538bf332b1750"} Nov 01 21:13:26.901801 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:26.901712 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7091fd-98896dbf7-6r5f6" event={"ID":"ab517a39-c7e7-4567-b294-484c21d3d799","Type":"ContainerDied","Data":"ce2dcfc03f1766e06d88ec09bff80eb100c18ffe8bd57a095bbae0fb2eb42703"} Nov 01 21:13:26.901801 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:26.901737 4484 scope.go:122] "RemoveContainer" containerID="deaa910fc4ac76c40cf15cd323d28e09e38e14f8df02fb6640b538bf332b1750" Nov 01 21:13:26.903682 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:26.903656 4484 generic.go:356] "Generic (PLEG): container finished" podID="e127ff65-ebee-4f72-952b-f4464d7592f9" containerID="2c65b50f57384b1b87b8389593422b84d76a17447bb9470b2bdafb57539d733f" exitCode=0 Nov 01 21:13:26.903825 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:26.903709 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7091fd-6d79589dd7-mqf74" Nov 01 21:13:26.903825 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:26.903745 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7091fd-6d79589dd7-mqf74" event={"ID":"e127ff65-ebee-4f72-952b-f4464d7592f9","Type":"ContainerDied","Data":"2c65b50f57384b1b87b8389593422b84d76a17447bb9470b2bdafb57539d733f"} Nov 01 21:13:26.903825 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:26.903796 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7091fd-6d79589dd7-mqf74" event={"ID":"e127ff65-ebee-4f72-952b-f4464d7592f9","Type":"ContainerDied","Data":"248a8cd703cc8081e2370291eb94e4a90439a2764778e1714dcc9e8c066b5e26"} Nov 01 21:13:26.913280 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:26.913257 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbmpq\" (UniqueName: \"kubernetes.io/projected/e127ff65-ebee-4f72-952b-f4464d7592f9-kube-api-access-lbmpq\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:26.913360 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:26.913284 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e127ff65-ebee-4f72-952b-f4464d7592f9-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:26.917658 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:26.917636 4484 scope.go:122] "RemoveContainer" containerID="deaa910fc4ac76c40cf15cd323d28e09e38e14f8df02fb6640b538bf332b1750" Nov 01 21:13:26.920389 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:13:26.920364 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"deaa910fc4ac76c40cf15cd323d28e09e38e14f8df02fb6640b538bf332b1750\": container with ID starting with deaa910fc4ac76c40cf15cd323d28e09e38e14f8df02fb6640b538bf332b1750 not found: ID does not exist" containerID="deaa910fc4ac76c40cf15cd323d28e09e38e14f8df02fb6640b538bf332b1750" Nov 01 21:13:26.920524 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:26.920396 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"deaa910fc4ac76c40cf15cd323d28e09e38e14f8df02fb6640b538bf332b1750"} err="failed to get container status \"deaa910fc4ac76c40cf15cd323d28e09e38e14f8df02fb6640b538bf332b1750\": rpc error: code = NotFound desc = could not find container \"deaa910fc4ac76c40cf15cd323d28e09e38e14f8df02fb6640b538bf332b1750\": container with ID starting with deaa910fc4ac76c40cf15cd323d28e09e38e14f8df02fb6640b538bf332b1750 not found: ID does not exist" Nov 01 21:13:26.920524 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:26.920418 4484 scope.go:122] "RemoveContainer" containerID="2c65b50f57384b1b87b8389593422b84d76a17447bb9470b2bdafb57539d733f" Nov 01 21:13:27.087195 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:27.087168 4484 scope.go:122] "RemoveContainer" containerID="2c65b50f57384b1b87b8389593422b84d76a17447bb9470b2bdafb57539d733f" Nov 01 21:13:27.087669 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:13:27.087639 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c65b50f57384b1b87b8389593422b84d76a17447bb9470b2bdafb57539d733f\": container with ID starting with 2c65b50f57384b1b87b8389593422b84d76a17447bb9470b2bdafb57539d733f not found: ID does not exist" containerID="2c65b50f57384b1b87b8389593422b84d76a17447bb9470b2bdafb57539d733f" Nov 01 21:13:27.087763 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:27.087682 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c65b50f57384b1b87b8389593422b84d76a17447bb9470b2bdafb57539d733f"} err="failed to get container status \"2c65b50f57384b1b87b8389593422b84d76a17447bb9470b2bdafb57539d733f\": rpc error: code = NotFound desc = could not find container \"2c65b50f57384b1b87b8389593422b84d76a17447bb9470b2bdafb57539d733f\": container with ID starting with 2c65b50f57384b1b87b8389593422b84d76a17447bb9470b2bdafb57539d733f not found: ID does not exist" Nov 01 21:13:27.179339 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:27.179312 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7091fd-6d79589dd7-mqf74"] Nov 01 21:13:27.183344 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:27.183306 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7091fd-6d79589dd7-mqf74"] Nov 01 21:13:27.291957 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:27.291924 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7091fd-98896dbf7-6r5f6"] Nov 01 21:13:27.297901 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:27.297864 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7091fd-98896dbf7-6r5f6"] Nov 01 21:13:27.372695 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:27.372661 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab517a39-c7e7-4567-b294-484c21d3d799" path="/var/lib/kubelet/pods/ab517a39-c7e7-4567-b294-484c21d3d799/volumes" Nov 01 21:13:27.373123 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:27.373109 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e127ff65-ebee-4f72-952b-f4464d7592f9" path="/var/lib/kubelet/pods/e127ff65-ebee-4f72-952b-f4464d7592f9/volumes" Nov 01 21:13:27.607261 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:27.607176 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:27.859765 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:27.859686 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:28.008732 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:28.008694 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:28.105834 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:28.105791 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b9305-wvt2d"] Nov 01 21:13:28.151888 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:28.151795 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b9305-wvt2d"] Nov 01 21:13:28.516563 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:28.516533 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:29.373427 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:29.373396 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32959521-121f-43c1-939d-a0f22e0d1f58" path="/var/lib/kubelet/pods/32959521-121f-43c1-939d-a0f22e0d1f58/volumes" Nov 01 21:13:30.209297 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:30.209263 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-cd248f-6c9889fc9c-wbd52" podUID="69c9a660-98c7-436f-8bee-b7781c596d0c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 21:13:32.087244 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:32.087164 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b"] Nov 01 21:13:32.087679 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:32.087384 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b" podUID="c75a1500-a843-43a3-98f7-a459015090c3" containerName="istio-proxy" containerID="cri-o://950f959c5ced7f6cf412c2a0dd0f53561086d99027b5bf5eda837a2be3bf52ab" gracePeriod=30 Nov 01 21:13:32.360679 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:32.360592 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:33.876058 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:33.876012 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:34.151943 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:34.151850 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:34.304676 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:34.304640 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:34.359894 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:34.359857 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e0a6f5-4tvj9"] Nov 01 21:13:34.410918 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:34.410820 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e0a6f5-4tvj9"] Nov 01 21:13:34.524680 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:34.524644 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:34.691515 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:34.691429 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:34.819422 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:34.819384 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26"] Nov 01 21:13:34.819621 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:34.819595 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26" podUID="69124be6-394b-45bd-9d52-6de87dfc5648" containerName="istio-proxy" containerID="cri-o://5e2c52f3040aa9adf29fbbc3cb9ea39368be840f802d18240cccf290f3d38c22" gracePeriod=30 Nov 01 21:13:34.862190 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:34.862141 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:35.372359 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:35.372328 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a0adb84-65af-4381-bbea-c4ed879ac334" path="/var/lib/kubelet/pods/3a0adb84-65af-4381-bbea-c4ed879ac334/volumes" Nov 01 21:13:35.596767 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:35.596729 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-044440-cz5p8"] Nov 01 21:13:35.597442 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:35.597427 4484 cpu_manager.go:414] "containerMap: removing container" podUID="ab517a39-c7e7-4567-b294-484c21d3d799" containerName="payload-processing" Nov 01 21:13:35.597496 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:35.597444 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab517a39-c7e7-4567-b294-484c21d3d799" containerName="payload-processing" Nov 01 21:13:35.597496 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:35.597471 4484 cpu_manager.go:414] "containerMap: removing container" podUID="2f2370cd-fd08-456b-a312-0b648cd4ebea" containerName="revoke-keys" Nov 01 21:13:35.597496 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:35.597476 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f2370cd-fd08-456b-a312-0b648cd4ebea" containerName="revoke-keys" Nov 01 21:13:35.597496 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:35.597484 4484 cpu_manager.go:414] "containerMap: removing container" podUID="311f9699-6585-4a2c-8c20-e6b696b2f857" containerName="main" Nov 01 21:13:35.597496 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:35.597490 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="311f9699-6585-4a2c-8c20-e6b696b2f857" containerName="main" Nov 01 21:13:35.597657 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:35.597509 4484 cpu_manager.go:414] "containerMap: removing container" podUID="e127ff65-ebee-4f72-952b-f4464d7592f9" containerName="payload-pre-processing" Nov 01 21:13:35.597657 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:35.597516 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e127ff65-ebee-4f72-952b-f4464d7592f9" containerName="payload-pre-processing" Nov 01 21:13:35.597657 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:35.597611 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab517a39-c7e7-4567-b294-484c21d3d799" containerName="payload-processing" Nov 01 21:13:35.597657 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:35.597624 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f2370cd-fd08-456b-a312-0b648cd4ebea" containerName="revoke-keys" Nov 01 21:13:35.597657 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:35.597631 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="311f9699-6585-4a2c-8c20-e6b696b2f857" containerName="main" Nov 01 21:13:35.597657 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:35.597641 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="e127ff65-ebee-4f72-952b-f4464d7592f9" containerName="payload-pre-processing" Nov 01 21:13:35.602744 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:35.602727 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-044440-cz5p8" Nov 01 21:13:35.656140 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:35.656051 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:13:35.656140 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:35.656072 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:13:35.656373 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:35.656162 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:13:35.656373 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:35.656322 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:13:35.657954 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:35.657926 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-044440-cz5p8"] Nov 01 21:13:35.665042 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:35.663699 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvqlk\" (UniqueName: \"kubernetes.io/projected/b400199f-cff6-4ca8-8204-b551846d3443-kube-api-access-cvqlk\") pod \"maas-api-revoke-keys-e2e-inf-b-044440-cz5p8\" (UID: \"b400199f-cff6-4ca8-8204-b551846d3443\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-044440-cz5p8" Nov 01 21:13:35.665042 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:35.663763 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b400199f-cff6-4ca8-8204-b551846d3443-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-044440-cz5p8\" (UID: \"b400199f-cff6-4ca8-8204-b551846d3443\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-044440-cz5p8" Nov 01 21:13:35.665042 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:35.663919 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b400199f-cff6-4ca8-8204-b551846d3443-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-044440-cz5p8\" (UID: \"b400199f-cff6-4ca8-8204-b551846d3443\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-044440-cz5p8" Nov 01 21:13:35.765576 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:35.765537 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvqlk\" (UniqueName: \"kubernetes.io/projected/b400199f-cff6-4ca8-8204-b551846d3443-kube-api-access-cvqlk\") pod \"maas-api-revoke-keys-e2e-inf-b-044440-cz5p8\" (UID: \"b400199f-cff6-4ca8-8204-b551846d3443\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-044440-cz5p8" Nov 01 21:13:35.765760 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:35.765589 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b400199f-cff6-4ca8-8204-b551846d3443-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-044440-cz5p8\" (UID: \"b400199f-cff6-4ca8-8204-b551846d3443\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-044440-cz5p8" Nov 01 21:13:35.765760 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:35.765666 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b400199f-cff6-4ca8-8204-b551846d3443-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-044440-cz5p8\" (UID: \"b400199f-cff6-4ca8-8204-b551846d3443\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-044440-cz5p8" Nov 01 21:13:35.766357 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:35.766335 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b400199f-cff6-4ca8-8204-b551846d3443-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-044440-cz5p8\" (UID: \"b400199f-cff6-4ca8-8204-b551846d3443\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-044440-cz5p8" Nov 01 21:13:35.768003 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:35.767979 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b400199f-cff6-4ca8-8204-b551846d3443-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-044440-cz5p8\" (UID: \"b400199f-cff6-4ca8-8204-b551846d3443\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-044440-cz5p8" Nov 01 21:13:35.823626 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:35.823591 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvqlk\" (UniqueName: \"kubernetes.io/projected/b400199f-cff6-4ca8-8204-b551846d3443-kube-api-access-cvqlk\") pod \"maas-api-revoke-keys-e2e-inf-b-044440-cz5p8\" (UID: \"b400199f-cff6-4ca8-8204-b551846d3443\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-044440-cz5p8" Nov 01 21:13:35.916723 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:35.916635 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-044440-cz5p8" Nov 01 21:13:36.306851 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:36.306818 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-044440-cz5p8"] Nov 01 21:13:36.307295 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:13:36.307269 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb400199f_cff6_4ca8_8204_b551846d3443.slice/crio-4db96e025a5895d8e2359927d9603df74595ab9a120c4476929960db68857add WatchSource:0}: Error finding container 4db96e025a5895d8e2359927d9603df74595ab9a120c4476929960db68857add: Status 404 returned error can't find the container with id 4db96e025a5895d8e2359927d9603df74595ab9a120c4476929960db68857add Nov 01 21:13:37.037813 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:37.037776 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-044440-cz5p8" event={"ID":"b400199f-cff6-4ca8-8204-b551846d3443","Type":"ContainerStarted","Data":"9d3642cfeab94c1bee39a239537eb8b2e1c2d07be25c1df1ede20b92dae8dfda"} Nov 01 21:13:37.038235 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:37.037835 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-044440-cz5p8" event={"ID":"b400199f-cff6-4ca8-8204-b551846d3443","Type":"ContainerStarted","Data":"4db96e025a5895d8e2359927d9603df74595ab9a120c4476929960db68857add"} Nov 01 21:13:37.156707 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:37.156631 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-044440-cz5p8" podStartSLOduration=2.156611678 podStartE2EDuration="2.156611678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:13:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:13:37.152511477 +0000 UTC m=+2567.930151639" watchObservedRunningTime="2026-11-01 21:13:37.156611678 +0000 UTC m=+2567.934251818" Nov 01 21:13:37.386990 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:37.386960 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b" Nov 01 21:13:37.488163 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:37.488129 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c75a1500-a843-43a3-98f7-a459015090c3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c75a1500-a843-43a3-98f7-a459015090c3-istio-podinfo\") pod \"c75a1500-a843-43a3-98f7-a459015090c3\" (UID: \"c75a1500-a843-43a3-98f7-a459015090c3\") " Nov 01 21:13:37.488386 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:37.488200 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c75a1500-a843-43a3-98f7-a459015090c3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c75a1500-a843-43a3-98f7-a459015090c3-istio-data\") pod \"c75a1500-a843-43a3-98f7-a459015090c3\" (UID: \"c75a1500-a843-43a3-98f7-a459015090c3\") " Nov 01 21:13:37.488386 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:37.488295 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c75a1500-a843-43a3-98f7-a459015090c3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c75a1500-a843-43a3-98f7-a459015090c3-istiod-ca-cert\") pod \"c75a1500-a843-43a3-98f7-a459015090c3\" (UID: \"c75a1500-a843-43a3-98f7-a459015090c3\") " Nov 01 21:13:37.488386 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:37.488341 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c75a1500-a843-43a3-98f7-a459015090c3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c75a1500-a843-43a3-98f7-a459015090c3-workload-certs\") pod \"c75a1500-a843-43a3-98f7-a459015090c3\" (UID: \"c75a1500-a843-43a3-98f7-a459015090c3\") " Nov 01 21:13:37.488568 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:37.488468 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c75a1500-a843-43a3-98f7-a459015090c3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c75a1500-a843-43a3-98f7-a459015090c3-credential-socket\") pod \"c75a1500-a843-43a3-98f7-a459015090c3\" (UID: \"c75a1500-a843-43a3-98f7-a459015090c3\") " Nov 01 21:13:37.488568 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:37.488507 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c75a1500-a843-43a3-98f7-a459015090c3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c75a1500-a843-43a3-98f7-a459015090c3-workload-socket\") pod \"c75a1500-a843-43a3-98f7-a459015090c3\" (UID: \"c75a1500-a843-43a3-98f7-a459015090c3\") " Nov 01 21:13:37.488568 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:37.488517 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c75a1500-a843-43a3-98f7-a459015090c3-istio-data" pod "c75a1500-a843-43a3-98f7-a459015090c3" (UID: "c75a1500-a843-43a3-98f7-a459015090c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:13:37.488568 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:37.488545 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c75a1500-a843-43a3-98f7-a459015090c3-istio-token\" (UniqueName: \"kubernetes.io/projected/c75a1500-a843-43a3-98f7-a459015090c3-istio-token\") pod \"c75a1500-a843-43a3-98f7-a459015090c3\" (UID: \"c75a1500-a843-43a3-98f7-a459015090c3\") " Nov 01 21:13:37.488856 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:37.488572 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c75a1500-a843-43a3-98f7-a459015090c3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c75a1500-a843-43a3-98f7-a459015090c3-istio-envoy\") pod \"c75a1500-a843-43a3-98f7-a459015090c3\" (UID: \"c75a1500-a843-43a3-98f7-a459015090c3\") " Nov 01 21:13:37.488856 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:37.488615 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c75a1500-a843-43a3-98f7-a459015090c3-workload-certs" pod "c75a1500-a843-43a3-98f7-a459015090c3" (UID: "c75a1500-a843-43a3-98f7-a459015090c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:13:37.488856 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:37.488622 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c75a1500-a843-43a3-98f7-a459015090c3-kube-api-access-5fqjb\" (UniqueName: \"kubernetes.io/projected/c75a1500-a843-43a3-98f7-a459015090c3-kube-api-access-5fqjb\") pod \"c75a1500-a843-43a3-98f7-a459015090c3\" (UID: \"c75a1500-a843-43a3-98f7-a459015090c3\") " Nov 01 21:13:37.488856 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:37.488709 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c75a1500-a843-43a3-98f7-a459015090c3-credential-socket" pod "c75a1500-a843-43a3-98f7-a459015090c3" (UID: "c75a1500-a843-43a3-98f7-a459015090c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:13:37.488856 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:37.488741 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c75a1500-a843-43a3-98f7-a459015090c3-istiod-ca-cert" pod "c75a1500-a843-43a3-98f7-a459015090c3" (UID: "c75a1500-a843-43a3-98f7-a459015090c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:13:37.488856 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:37.488806 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c75a1500-a843-43a3-98f7-a459015090c3-workload-socket" pod "c75a1500-a843-43a3-98f7-a459015090c3" (UID: "c75a1500-a843-43a3-98f7-a459015090c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:13:37.489414 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:37.489379 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c75a1500-a843-43a3-98f7-a459015090c3-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:37.489414 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:37.489407 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c75a1500-a843-43a3-98f7-a459015090c3-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:37.489588 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:37.489421 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c75a1500-a843-43a3-98f7-a459015090c3-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:37.489588 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:37.489435 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c75a1500-a843-43a3-98f7-a459015090c3-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:37.489588 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:37.489448 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c75a1500-a843-43a3-98f7-a459015090c3-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:37.490910 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:37.490843 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c75a1500-a843-43a3-98f7-a459015090c3-istio-token" pod "c75a1500-a843-43a3-98f7-a459015090c3" (UID: "c75a1500-a843-43a3-98f7-a459015090c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:13:37.491093 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:37.490945 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c75a1500-a843-43a3-98f7-a459015090c3-istio-envoy" pod "c75a1500-a843-43a3-98f7-a459015090c3" (UID: "c75a1500-a843-43a3-98f7-a459015090c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:13:37.491093 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:37.490950 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c75a1500-a843-43a3-98f7-a459015090c3-istio-podinfo" pod "c75a1500-a843-43a3-98f7-a459015090c3" (UID: "c75a1500-a843-43a3-98f7-a459015090c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:13:37.491203 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:37.491103 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c75a1500-a843-43a3-98f7-a459015090c3-kube-api-access-5fqjb" pod "c75a1500-a843-43a3-98f7-a459015090c3" (UID: "c75a1500-a843-43a3-98f7-a459015090c3"). InnerVolumeSpecName "kube-api-access-5fqjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:13:37.590841 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:37.590761 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c75a1500-a843-43a3-98f7-a459015090c3-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:37.590841 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:37.590789 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c75a1500-a843-43a3-98f7-a459015090c3-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:37.590841 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:37.590802 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c75a1500-a843-43a3-98f7-a459015090c3-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:37.590841 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:37.590811 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fqjb\" (UniqueName: \"kubernetes.io/projected/c75a1500-a843-43a3-98f7-a459015090c3-kube-api-access-5fqjb\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:38.046009 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:38.045962 4484 generic.go:356] "Generic (PLEG): container finished" podID="b400199f-cff6-4ca8-8204-b551846d3443" containerID="9d3642cfeab94c1bee39a239537eb8b2e1c2d07be25c1df1ede20b92dae8dfda" exitCode=0 Nov 01 21:13:38.046425 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:38.046052 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-044440-cz5p8" event={"ID":"b400199f-cff6-4ca8-8204-b551846d3443","Type":"ContainerDied","Data":"9d3642cfeab94c1bee39a239537eb8b2e1c2d07be25c1df1ede20b92dae8dfda"} Nov 01 21:13:38.047586 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:38.047564 4484 generic.go:356] "Generic (PLEG): container finished" podID="c75a1500-a843-43a3-98f7-a459015090c3" containerID="950f959c5ced7f6cf412c2a0dd0f53561086d99027b5bf5eda837a2be3bf52ab" exitCode=0 Nov 01 21:13:38.047708 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:38.047610 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b" Nov 01 21:13:38.047708 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:38.047646 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b" event={"ID":"c75a1500-a843-43a3-98f7-a459015090c3","Type":"ContainerDied","Data":"950f959c5ced7f6cf412c2a0dd0f53561086d99027b5bf5eda837a2be3bf52ab"} Nov 01 21:13:38.047708 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:38.047688 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b" event={"ID":"c75a1500-a843-43a3-98f7-a459015090c3","Type":"ContainerDied","Data":"10b510d860c91efb40f500952145912a02dd93b287f4b1bc99c5272ddc94358d"} Nov 01 21:13:38.047819 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:38.047710 4484 scope.go:122] "RemoveContainer" containerID="950f959c5ced7f6cf412c2a0dd0f53561086d99027b5bf5eda837a2be3bf52ab" Nov 01 21:13:38.059704 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:38.059685 4484 scope.go:122] "RemoveContainer" containerID="950f959c5ced7f6cf412c2a0dd0f53561086d99027b5bf5eda837a2be3bf52ab" Nov 01 21:13:38.060051 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:13:38.060031 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"950f959c5ced7f6cf412c2a0dd0f53561086d99027b5bf5eda837a2be3bf52ab\": container with ID starting with 950f959c5ced7f6cf412c2a0dd0f53561086d99027b5bf5eda837a2be3bf52ab not found: ID does not exist" containerID="950f959c5ced7f6cf412c2a0dd0f53561086d99027b5bf5eda837a2be3bf52ab" Nov 01 21:13:38.060127 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:38.060059 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"950f959c5ced7f6cf412c2a0dd0f53561086d99027b5bf5eda837a2be3bf52ab"} err="failed to get container status \"950f959c5ced7f6cf412c2a0dd0f53561086d99027b5bf5eda837a2be3bf52ab\": rpc error: code = NotFound desc = could not find container \"950f959c5ced7f6cf412c2a0dd0f53561086d99027b5bf5eda837a2be3bf52ab\": container with ID starting with 950f959c5ced7f6cf412c2a0dd0f53561086d99027b5bf5eda837a2be3bf52ab not found: ID does not exist" Nov 01 21:13:38.277884 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:38.277854 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b"] Nov 01 21:13:38.281149 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:38.281123 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-cd248f-openshift-default-d7fc5ccf5-2qv5b"] Nov 01 21:13:39.216114 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:39.216088 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-044440-cz5p8" Nov 01 21:13:39.277046 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:39.276989 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:39.321639 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:39.321536 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b400199f-cff6-4ca8-8204-b551846d3443-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b400199f-cff6-4ca8-8204-b551846d3443-maas-api-openshift-ca\") pod \"b400199f-cff6-4ca8-8204-b551846d3443\" (UID: \"b400199f-cff6-4ca8-8204-b551846d3443\") " Nov 01 21:13:39.333027 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:39.322302 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b400199f-cff6-4ca8-8204-b551846d3443-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b400199f-cff6-4ca8-8204-b551846d3443-maas-api-xks-ca\") pod \"b400199f-cff6-4ca8-8204-b551846d3443\" (UID: \"b400199f-cff6-4ca8-8204-b551846d3443\") " Nov 01 21:13:39.333377 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:39.333347 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b400199f-cff6-4ca8-8204-b551846d3443-kube-api-access-cvqlk\" (UniqueName: \"kubernetes.io/projected/b400199f-cff6-4ca8-8204-b551846d3443-kube-api-access-cvqlk\") pod \"b400199f-cff6-4ca8-8204-b551846d3443\" (UID: \"b400199f-cff6-4ca8-8204-b551846d3443\") " Nov 01 21:13:39.335028 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:39.323489 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b400199f-cff6-4ca8-8204-b551846d3443-maas-api-openshift-ca" pod "b400199f-cff6-4ca8-8204-b551846d3443" (UID: "b400199f-cff6-4ca8-8204-b551846d3443"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:13:39.335028 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:39.327500 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b400199f-cff6-4ca8-8204-b551846d3443-maas-api-xks-ca" pod "b400199f-cff6-4ca8-8204-b551846d3443" (UID: "b400199f-cff6-4ca8-8204-b551846d3443"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:13:39.339047 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:39.339015 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b400199f-cff6-4ca8-8204-b551846d3443-kube-api-access-cvqlk" pod "b400199f-cff6-4ca8-8204-b551846d3443" (UID: "b400199f-cff6-4ca8-8204-b551846d3443"). InnerVolumeSpecName "kube-api-access-cvqlk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:13:39.373258 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:39.373229 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c75a1500-a843-43a3-98f7-a459015090c3" path="/var/lib/kubelet/pods/c75a1500-a843-43a3-98f7-a459015090c3/volumes" Nov 01 21:13:39.435736 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:39.435694 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b400199f-cff6-4ca8-8204-b551846d3443-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:39.435736 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:39.435731 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b400199f-cff6-4ca8-8204-b551846d3443-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:39.435736 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:39.435741 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvqlk\" (UniqueName: \"kubernetes.io/projected/b400199f-cff6-4ca8-8204-b551846d3443-kube-api-access-cvqlk\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:39.776335 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:39.776296 4484 patch_prober.go:29] interesting pod/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 21:13:39.776528 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:39.776355 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26" podUID="69124be6-394b-45bd-9d52-6de87dfc5648" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 21:13:40.071707 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:40.071676 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-044440-cz5p8" Nov 01 21:13:40.071884 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:40.071706 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-044440-cz5p8" event={"ID":"b400199f-cff6-4ca8-8204-b551846d3443","Type":"ContainerDied","Data":"4db96e025a5895d8e2359927d9603df74595ab9a120c4476929960db68857add"} Nov 01 21:13:40.071884 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:40.071754 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4db96e025a5895d8e2359927d9603df74595ab9a120c4476929960db68857add" Nov 01 21:13:40.073794 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:40.073768 4484 generic.go:356] "Generic (PLEG): container finished" podID="69124be6-394b-45bd-9d52-6de87dfc5648" containerID="5e2c52f3040aa9adf29fbbc3cb9ea39368be840f802d18240cccf290f3d38c22" exitCode=0 Nov 01 21:13:40.073926 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:40.073849 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26" event={"ID":"69124be6-394b-45bd-9d52-6de87dfc5648","Type":"ContainerDied","Data":"5e2c52f3040aa9adf29fbbc3cb9ea39368be840f802d18240cccf290f3d38c22"} Nov 01 21:13:40.074082 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:40.073955 4484 scope.go:122] "RemoveContainer" containerID="7d182619c2a5d6c4a794f450da9d548e759cabb34a4972bf9152cc237cc01aab" Nov 01 21:13:40.074423 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:40.074404 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26" Nov 01 21:13:40.207807 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:13:40.207781 4484 logging.go:55] [core] [Channel #12485 SubChannel #12486]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.40:50052", ServerName: "10.134.0.40:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.40:50052: connect: connection refused" Nov 01 21:13:40.245592 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:40.245556 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69124be6-394b-45bd-9d52-6de87dfc5648-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69124be6-394b-45bd-9d52-6de87dfc5648-istio-podinfo\") pod \"69124be6-394b-45bd-9d52-6de87dfc5648\" (UID: \"69124be6-394b-45bd-9d52-6de87dfc5648\") " Nov 01 21:13:40.246234 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:40.245637 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69124be6-394b-45bd-9d52-6de87dfc5648-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69124be6-394b-45bd-9d52-6de87dfc5648-workload-socket\") pod \"69124be6-394b-45bd-9d52-6de87dfc5648\" (UID: \"69124be6-394b-45bd-9d52-6de87dfc5648\") " Nov 01 21:13:40.246234 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:40.245690 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69124be6-394b-45bd-9d52-6de87dfc5648-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69124be6-394b-45bd-9d52-6de87dfc5648-istiod-ca-cert\") pod \"69124be6-394b-45bd-9d52-6de87dfc5648\" (UID: \"69124be6-394b-45bd-9d52-6de87dfc5648\") " Nov 01 21:13:40.246234 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:40.245727 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69124be6-394b-45bd-9d52-6de87dfc5648-kube-api-access-kqfh8\" (UniqueName: \"kubernetes.io/projected/69124be6-394b-45bd-9d52-6de87dfc5648-kube-api-access-kqfh8\") pod \"69124be6-394b-45bd-9d52-6de87dfc5648\" (UID: \"69124be6-394b-45bd-9d52-6de87dfc5648\") " Nov 01 21:13:40.246234 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:40.245778 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69124be6-394b-45bd-9d52-6de87dfc5648-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69124be6-394b-45bd-9d52-6de87dfc5648-credential-socket\") pod \"69124be6-394b-45bd-9d52-6de87dfc5648\" (UID: \"69124be6-394b-45bd-9d52-6de87dfc5648\") " Nov 01 21:13:40.246234 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:40.245843 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69124be6-394b-45bd-9d52-6de87dfc5648-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69124be6-394b-45bd-9d52-6de87dfc5648-istio-envoy\") pod \"69124be6-394b-45bd-9d52-6de87dfc5648\" (UID: \"69124be6-394b-45bd-9d52-6de87dfc5648\") " Nov 01 21:13:40.246234 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:40.245915 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69124be6-394b-45bd-9d52-6de87dfc5648-istio-token\" (UniqueName: \"kubernetes.io/projected/69124be6-394b-45bd-9d52-6de87dfc5648-istio-token\") pod \"69124be6-394b-45bd-9d52-6de87dfc5648\" (UID: \"69124be6-394b-45bd-9d52-6de87dfc5648\") " Nov 01 21:13:40.246234 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:40.245960 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69124be6-394b-45bd-9d52-6de87dfc5648-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69124be6-394b-45bd-9d52-6de87dfc5648-istio-data\") pod \"69124be6-394b-45bd-9d52-6de87dfc5648\" (UID: \"69124be6-394b-45bd-9d52-6de87dfc5648\") " Nov 01 21:13:40.246234 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:40.246095 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69124be6-394b-45bd-9d52-6de87dfc5648-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69124be6-394b-45bd-9d52-6de87dfc5648-workload-certs\") pod \"69124be6-394b-45bd-9d52-6de87dfc5648\" (UID: \"69124be6-394b-45bd-9d52-6de87dfc5648\") " Nov 01 21:13:40.246599 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:40.246352 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69124be6-394b-45bd-9d52-6de87dfc5648-credential-socket" pod "69124be6-394b-45bd-9d52-6de87dfc5648" (UID: "69124be6-394b-45bd-9d52-6de87dfc5648"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:13:40.246599 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:40.246563 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69124be6-394b-45bd-9d52-6de87dfc5648-workload-socket" pod "69124be6-394b-45bd-9d52-6de87dfc5648" (UID: "69124be6-394b-45bd-9d52-6de87dfc5648"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:13:40.246599 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:40.246578 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69124be6-394b-45bd-9d52-6de87dfc5648-workload-certs" pod "69124be6-394b-45bd-9d52-6de87dfc5648" (UID: "69124be6-394b-45bd-9d52-6de87dfc5648"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:13:40.246832 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:40.246810 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69124be6-394b-45bd-9d52-6de87dfc5648-istiod-ca-cert" pod "69124be6-394b-45bd-9d52-6de87dfc5648" (UID: "69124be6-394b-45bd-9d52-6de87dfc5648"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:13:40.246981 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:40.246957 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69124be6-394b-45bd-9d52-6de87dfc5648-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:40.247092 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:40.246988 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69124be6-394b-45bd-9d52-6de87dfc5648-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:40.247092 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:40.247030 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69124be6-394b-45bd-9d52-6de87dfc5648-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:40.247092 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:40.247042 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69124be6-394b-45bd-9d52-6de87dfc5648-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:40.250212 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:40.250170 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69124be6-394b-45bd-9d52-6de87dfc5648-istio-data" pod "69124be6-394b-45bd-9d52-6de87dfc5648" (UID: "69124be6-394b-45bd-9d52-6de87dfc5648"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:13:40.252349 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:40.252318 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69124be6-394b-45bd-9d52-6de87dfc5648-istio-envoy" pod "69124be6-394b-45bd-9d52-6de87dfc5648" (UID: "69124be6-394b-45bd-9d52-6de87dfc5648"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:13:40.260302 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:40.260242 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69124be6-394b-45bd-9d52-6de87dfc5648-istio-podinfo" pod "69124be6-394b-45bd-9d52-6de87dfc5648" (UID: "69124be6-394b-45bd-9d52-6de87dfc5648"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:13:40.262467 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:40.262430 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69124be6-394b-45bd-9d52-6de87dfc5648-kube-api-access-kqfh8" pod "69124be6-394b-45bd-9d52-6de87dfc5648" (UID: "69124be6-394b-45bd-9d52-6de87dfc5648"). InnerVolumeSpecName "kube-api-access-kqfh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:13:40.266190 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:40.266157 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69124be6-394b-45bd-9d52-6de87dfc5648-istio-token" pod "69124be6-394b-45bd-9d52-6de87dfc5648" (UID: "69124be6-394b-45bd-9d52-6de87dfc5648"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:13:40.348780 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:40.348698 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69124be6-394b-45bd-9d52-6de87dfc5648-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:40.348780 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:40.348727 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqfh8\" (UniqueName: \"kubernetes.io/projected/69124be6-394b-45bd-9d52-6de87dfc5648-kube-api-access-kqfh8\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:40.348780 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:40.348737 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69124be6-394b-45bd-9d52-6de87dfc5648-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:40.348780 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:40.348747 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69124be6-394b-45bd-9d52-6de87dfc5648-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:40.348780 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:40.348756 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69124be6-394b-45bd-9d52-6de87dfc5648-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:40.408371 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:40.408336 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-cd248f-6c9889fc9c-wbd52" Nov 01 21:13:40.550920 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:40.550878 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69c9a660-98c7-436f-8bee-b7781c596d0c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69c9a660-98c7-436f-8bee-b7781c596d0c-plugins-config-volume\") pod \"69c9a660-98c7-436f-8bee-b7781c596d0c\" (UID: \"69c9a660-98c7-436f-8bee-b7781c596d0c\") " Nov 01 21:13:40.551173 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:40.551014 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69c9a660-98c7-436f-8bee-b7781c596d0c-kube-api-access-jvqxb\" (UniqueName: \"kubernetes.io/projected/69c9a660-98c7-436f-8bee-b7781c596d0c-kube-api-access-jvqxb\") pod \"69c9a660-98c7-436f-8bee-b7781c596d0c\" (UID: \"69c9a660-98c7-436f-8bee-b7781c596d0c\") " Nov 01 21:13:40.551372 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:40.551346 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69c9a660-98c7-436f-8bee-b7781c596d0c-plugins-config-volume" pod "69c9a660-98c7-436f-8bee-b7781c596d0c" (UID: "69c9a660-98c7-436f-8bee-b7781c596d0c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:13:40.551628 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:40.551607 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69c9a660-98c7-436f-8bee-b7781c596d0c-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:40.553230 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:40.553207 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69c9a660-98c7-436f-8bee-b7781c596d0c-kube-api-access-jvqxb" pod "69c9a660-98c7-436f-8bee-b7781c596d0c" (UID: "69c9a660-98c7-436f-8bee-b7781c596d0c"). InnerVolumeSpecName "kube-api-access-jvqxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:13:40.576448 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:40.576411 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:40.653472 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:40.653385 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvqxb\" (UniqueName: \"kubernetes.io/projected/69c9a660-98c7-436f-8bee-b7781c596d0c-kube-api-access-jvqxb\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:40.737649 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:40.737616 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:40.870681 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:40.870645 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc"] Nov 01 21:13:40.877156 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:40.870892 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc" podUID="8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff" containerName="istio-proxy" containerID="cri-o://c040b0424a73a7fc9b8c22f4e7bfcb21e09d8f33a59865c6a6db3128c4ef431e" gracePeriod=30 Nov 01 21:13:40.922867 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:40.922777 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:41.095527 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:41.095498 4484 generic.go:356] "Generic (PLEG): container finished" podID="69c9a660-98c7-436f-8bee-b7781c596d0c" containerID="79c6672bdfe0e3dca2fa9c24bfa462019ee8f8926e053b6e218e88358fb61e76" exitCode=0 Nov 01 21:13:41.095733 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:41.095548 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-cd248f-6c9889fc9c-wbd52" Nov 01 21:13:41.095733 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:41.095578 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-cd248f-6c9889fc9c-wbd52" event={"ID":"69c9a660-98c7-436f-8bee-b7781c596d0c","Type":"ContainerDied","Data":"79c6672bdfe0e3dca2fa9c24bfa462019ee8f8926e053b6e218e88358fb61e76"} Nov 01 21:13:41.095733 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:41.095634 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-cd248f-6c9889fc9c-wbd52" event={"ID":"69c9a660-98c7-436f-8bee-b7781c596d0c","Type":"ContainerDied","Data":"5503f03feb4db749e9e6eec80a9fb5028e6bc0563107ddec22210b7b555c8c72"} Nov 01 21:13:41.095733 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:41.095664 4484 scope.go:122] "RemoveContainer" containerID="79c6672bdfe0e3dca2fa9c24bfa462019ee8f8926e053b6e218e88358fb61e76" Nov 01 21:13:41.097400 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:41.097377 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26" event={"ID":"69124be6-394b-45bd-9d52-6de87dfc5648","Type":"ContainerDied","Data":"bd6991dfef6e77e33fb61901d7da4ca76aef504ef7fc039f9b362075b840480b"} Nov 01 21:13:41.097492 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:41.097454 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26" Nov 01 21:13:41.107575 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:41.107553 4484 scope.go:122] "RemoveContainer" containerID="79c6672bdfe0e3dca2fa9c24bfa462019ee8f8926e053b6e218e88358fb61e76" Nov 01 21:13:41.107901 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:13:41.107880 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79c6672bdfe0e3dca2fa9c24bfa462019ee8f8926e053b6e218e88358fb61e76\": container with ID starting with 79c6672bdfe0e3dca2fa9c24bfa462019ee8f8926e053b6e218e88358fb61e76 not found: ID does not exist" containerID="79c6672bdfe0e3dca2fa9c24bfa462019ee8f8926e053b6e218e88358fb61e76" Nov 01 21:13:41.107989 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:41.107906 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79c6672bdfe0e3dca2fa9c24bfa462019ee8f8926e053b6e218e88358fb61e76"} err="failed to get container status \"79c6672bdfe0e3dca2fa9c24bfa462019ee8f8926e053b6e218e88358fb61e76\": rpc error: code = NotFound desc = could not find container \"79c6672bdfe0e3dca2fa9c24bfa462019ee8f8926e053b6e218e88358fb61e76\": container with ID starting with 79c6672bdfe0e3dca2fa9c24bfa462019ee8f8926e053b6e218e88358fb61e76 not found: ID does not exist" Nov 01 21:13:41.107989 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:41.107930 4484 scope.go:122] "RemoveContainer" containerID="5e2c52f3040aa9adf29fbbc3cb9ea39368be840f802d18240cccf290f3d38c22" Nov 01 21:13:41.207911 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:41.207870 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-cd248f-6c9889fc9c-wbd52" podUID="69c9a660-98c7-436f-8bee-b7781c596d0c" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.40:50052\" within 1s: context deadline exceeded" Nov 01 21:13:41.208103 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:41.207960 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-cd248f-6c9889fc9c-wbd52" Nov 01 21:13:41.263620 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:41.263581 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-cd248f-6c9889fc9c-wbd52"] Nov 01 21:13:41.269093 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:41.269063 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-cd248f-6c9889fc9c-wbd52"] Nov 01 21:13:41.382354 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:41.375491 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69c9a660-98c7-436f-8bee-b7781c596d0c" path="/var/lib/kubelet/pods/69c9a660-98c7-436f-8bee-b7781c596d0c/volumes" Nov 01 21:13:41.446097 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:41.446007 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26"] Nov 01 21:13:41.451591 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:41.451560 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6b9305-openshift-default-74c965c97c-7xs26"] Nov 01 21:13:41.694947 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:41.694911 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a41242-dgcgt"] Nov 01 21:13:41.695561 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:41.695548 4484 cpu_manager.go:414] "containerMap: removing container" podUID="b400199f-cff6-4ca8-8204-b551846d3443" containerName="revoke-keys" Nov 01 21:13:41.695609 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:41.695563 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b400199f-cff6-4ca8-8204-b551846d3443" containerName="revoke-keys" Nov 01 21:13:41.695609 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:41.695575 4484 cpu_manager.go:414] "containerMap: removing container" podUID="c75a1500-a843-43a3-98f7-a459015090c3" containerName="istio-proxy" Nov 01 21:13:41.695609 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:41.695580 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c75a1500-a843-43a3-98f7-a459015090c3" containerName="istio-proxy" Nov 01 21:13:41.695609 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:41.695600 4484 cpu_manager.go:414] "containerMap: removing container" podUID="69c9a660-98c7-436f-8bee-b7781c596d0c" containerName="payload-processing" Nov 01 21:13:41.695609 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:41.695606 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69c9a660-98c7-436f-8bee-b7781c596d0c" containerName="payload-processing" Nov 01 21:13:41.695777 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:41.695613 4484 cpu_manager.go:414] "containerMap: removing container" podUID="69124be6-394b-45bd-9d52-6de87dfc5648" containerName="istio-proxy" Nov 01 21:13:41.695777 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:41.695617 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69124be6-394b-45bd-9d52-6de87dfc5648" containerName="istio-proxy" Nov 01 21:13:41.695777 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:41.695638 4484 cpu_manager.go:414] "containerMap: removing container" podUID="69124be6-394b-45bd-9d52-6de87dfc5648" containerName="istio-proxy" Nov 01 21:13:41.695777 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:41.695643 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69124be6-394b-45bd-9d52-6de87dfc5648" containerName="istio-proxy" Nov 01 21:13:41.695777 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:41.695738 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="69124be6-394b-45bd-9d52-6de87dfc5648" containerName="istio-proxy" Nov 01 21:13:41.695777 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:41.695751 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="b400199f-cff6-4ca8-8204-b551846d3443" containerName="revoke-keys" Nov 01 21:13:41.695777 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:41.695757 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="c75a1500-a843-43a3-98f7-a459015090c3" containerName="istio-proxy" Nov 01 21:13:41.695777 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:41.695766 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="69c9a660-98c7-436f-8bee-b7781c596d0c" containerName="payload-processing" Nov 01 21:13:41.700753 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:41.700712 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a41242-dgcgt" Nov 01 21:13:41.753846 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:41.753813 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:13:41.753846 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:41.753820 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:13:41.754194 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:41.753860 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:13:41.754194 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:41.753956 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:13:41.755827 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:41.755798 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a41242-dgcgt"] Nov 01 21:13:41.806761 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:41.792562 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c259f9d2-8a0f-411d-bc9d-61644916daee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a41242-dgcgt\" (UID: \"c259f9d2-8a0f-411d-bc9d-61644916daee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a41242-dgcgt" Nov 01 21:13:41.806761 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:41.792841 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m4nn\" (UniqueName: \"kubernetes.io/projected/c259f9d2-8a0f-411d-bc9d-61644916daee-kube-api-access-9m4nn\") pod \"maas-api-revoke-keys-e2e-api-b-a41242-dgcgt\" (UID: \"c259f9d2-8a0f-411d-bc9d-61644916daee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a41242-dgcgt" Nov 01 21:13:41.806761 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:41.792908 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c259f9d2-8a0f-411d-bc9d-61644916daee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a41242-dgcgt\" (UID: \"c259f9d2-8a0f-411d-bc9d-61644916daee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a41242-dgcgt" Nov 01 21:13:41.894396 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:41.894354 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c259f9d2-8a0f-411d-bc9d-61644916daee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a41242-dgcgt\" (UID: \"c259f9d2-8a0f-411d-bc9d-61644916daee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a41242-dgcgt" Nov 01 21:13:41.894578 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:41.894419 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c259f9d2-8a0f-411d-bc9d-61644916daee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a41242-dgcgt\" (UID: \"c259f9d2-8a0f-411d-bc9d-61644916daee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a41242-dgcgt" Nov 01 21:13:41.894578 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:41.894571 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m4nn\" (UniqueName: \"kubernetes.io/projected/c259f9d2-8a0f-411d-bc9d-61644916daee-kube-api-access-9m4nn\") pod \"maas-api-revoke-keys-e2e-api-b-a41242-dgcgt\" (UID: \"c259f9d2-8a0f-411d-bc9d-61644916daee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a41242-dgcgt" Nov 01 21:13:41.895062 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:41.895040 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c259f9d2-8a0f-411d-bc9d-61644916daee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a41242-dgcgt\" (UID: \"c259f9d2-8a0f-411d-bc9d-61644916daee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a41242-dgcgt" Nov 01 21:13:41.896870 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:41.896839 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c259f9d2-8a0f-411d-bc9d-61644916daee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a41242-dgcgt\" (UID: \"c259f9d2-8a0f-411d-bc9d-61644916daee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a41242-dgcgt" Nov 01 21:13:41.952803 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:41.952728 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m4nn\" (UniqueName: \"kubernetes.io/projected/c259f9d2-8a0f-411d-bc9d-61644916daee-kube-api-access-9m4nn\") pod \"maas-api-revoke-keys-e2e-api-b-a41242-dgcgt\" (UID: \"c259f9d2-8a0f-411d-bc9d-61644916daee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a41242-dgcgt" Nov 01 21:13:42.015226 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:42.015191 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a41242-dgcgt" Nov 01 21:13:42.403602 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:42.403573 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a41242-dgcgt"] Nov 01 21:13:42.404797 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:13:42.404774 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc259f9d2_8a0f_411d_bc9d_61644916daee.slice/crio-10e5d2773d777c9db006df4fbef3ef229b0f5d97ce27c551a2dd49be82700b26 WatchSource:0}: Error finding container 10e5d2773d777c9db006df4fbef3ef229b0f5d97ce27c551a2dd49be82700b26: Status 404 returned error can't find the container with id 10e5d2773d777c9db006df4fbef3ef229b0f5d97ce27c551a2dd49be82700b26 Nov 01 21:13:42.560262 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:42.560224 4484 patch_prober.go:29] interesting pod/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 21:13:42.560424 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:42.560296 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc" podUID="8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 21:13:43.117619 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:43.117580 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a41242-dgcgt" event={"ID":"c259f9d2-8a0f-411d-bc9d-61644916daee","Type":"ContainerStarted","Data":"e848dca57f980c5b40af5b7e753a9fe3394ec47c035c7c03547c11b82431d8a0"} Nov 01 21:13:43.117816 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:43.117649 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a41242-dgcgt" event={"ID":"c259f9d2-8a0f-411d-bc9d-61644916daee","Type":"ContainerStarted","Data":"10e5d2773d777c9db006df4fbef3ef229b0f5d97ce27c551a2dd49be82700b26"} Nov 01 21:13:43.234568 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:43.234514 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a41242-dgcgt" podStartSLOduration=2.234499967 podStartE2EDuration="2.234499967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:13:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:13:43.23265947 +0000 UTC m=+2574.010299605" watchObservedRunningTime="2026-11-01 21:13:43.234499967 +0000 UTC m=+2574.012140102" Nov 01 21:13:43.372801 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:43.372725 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69124be6-394b-45bd-9d52-6de87dfc5648" path="/var/lib/kubelet/pods/69124be6-394b-45bd-9d52-6de87dfc5648/volumes" Nov 01 21:13:44.126299 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:44.126264 4484 generic.go:356] "Generic (PLEG): container finished" podID="c259f9d2-8a0f-411d-bc9d-61644916daee" containerID="e848dca57f980c5b40af5b7e753a9fe3394ec47c035c7c03547c11b82431d8a0" exitCode=0 Nov 01 21:13:44.126739 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:44.126347 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a41242-dgcgt" event={"ID":"c259f9d2-8a0f-411d-bc9d-61644916daee","Type":"ContainerDied","Data":"e848dca57f980c5b40af5b7e753a9fe3394ec47c035c7c03547c11b82431d8a0"} Nov 01 21:13:44.530044 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:44.529981 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:44.837629 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:44.837543 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:45.013978 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:45.013946 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:45.298145 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:45.298121 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a41242-dgcgt" Nov 01 21:13:45.340173 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:45.340133 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c259f9d2-8a0f-411d-bc9d-61644916daee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c259f9d2-8a0f-411d-bc9d-61644916daee-maas-api-openshift-ca\") pod \"c259f9d2-8a0f-411d-bc9d-61644916daee\" (UID: \"c259f9d2-8a0f-411d-bc9d-61644916daee\") " Nov 01 21:13:45.340406 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:45.340387 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c259f9d2-8a0f-411d-bc9d-61644916daee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c259f9d2-8a0f-411d-bc9d-61644916daee-maas-api-xks-ca\") pod \"c259f9d2-8a0f-411d-bc9d-61644916daee\" (UID: \"c259f9d2-8a0f-411d-bc9d-61644916daee\") " Nov 01 21:13:45.340452 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:45.340428 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c259f9d2-8a0f-411d-bc9d-61644916daee-kube-api-access-9m4nn\" (UniqueName: \"kubernetes.io/projected/c259f9d2-8a0f-411d-bc9d-61644916daee-kube-api-access-9m4nn\") pod \"c259f9d2-8a0f-411d-bc9d-61644916daee\" (UID: \"c259f9d2-8a0f-411d-bc9d-61644916daee\") " Nov 01 21:13:45.340560 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:45.340541 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c259f9d2-8a0f-411d-bc9d-61644916daee-maas-api-openshift-ca" pod "c259f9d2-8a0f-411d-bc9d-61644916daee" (UID: "c259f9d2-8a0f-411d-bc9d-61644916daee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:13:45.340867 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:45.340845 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c259f9d2-8a0f-411d-bc9d-61644916daee-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:45.342621 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:45.342597 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c259f9d2-8a0f-411d-bc9d-61644916daee-maas-api-xks-ca" pod "c259f9d2-8a0f-411d-bc9d-61644916daee" (UID: "c259f9d2-8a0f-411d-bc9d-61644916daee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:13:45.342621 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:45.342609 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c259f9d2-8a0f-411d-bc9d-61644916daee-kube-api-access-9m4nn" pod "c259f9d2-8a0f-411d-bc9d-61644916daee" (UID: "c259f9d2-8a0f-411d-bc9d-61644916daee"). InnerVolumeSpecName "kube-api-access-9m4nn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:13:45.442623 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:45.442517 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c259f9d2-8a0f-411d-bc9d-61644916daee-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:45.442623 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:45.442560 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9m4nn\" (UniqueName: \"kubernetes.io/projected/c259f9d2-8a0f-411d-bc9d-61644916daee-kube-api-access-9m4nn\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:45.737755 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:45.737675 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:45.767899 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:13:45.767857 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:13:46.129513 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:46.129485 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc" Nov 01 21:13:46.142403 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:46.142377 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a41242-dgcgt" Nov 01 21:13:46.142576 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:46.142413 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a41242-dgcgt" event={"ID":"c259f9d2-8a0f-411d-bc9d-61644916daee","Type":"ContainerDied","Data":"10e5d2773d777c9db006df4fbef3ef229b0f5d97ce27c551a2dd49be82700b26"} Nov 01 21:13:46.142576 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:46.142471 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10e5d2773d777c9db006df4fbef3ef229b0f5d97ce27c551a2dd49be82700b26" Nov 01 21:13:46.144367 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:46.144339 4484 generic.go:356] "Generic (PLEG): container finished" podID="8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff" containerID="c040b0424a73a7fc9b8c22f4e7bfcb21e09d8f33a59865c6a6db3128c4ef431e" exitCode=0 Nov 01 21:13:46.144512 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:46.144404 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc" Nov 01 21:13:46.144512 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:46.144420 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc" event={"ID":"8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff","Type":"ContainerDied","Data":"c040b0424a73a7fc9b8c22f4e7bfcb21e09d8f33a59865c6a6db3128c4ef431e"} Nov 01 21:13:46.144512 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:46.144481 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc" event={"ID":"8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff","Type":"ContainerDied","Data":"c4697355f311056e7b68aef7b018752058a79def0a72c449a1cbb61d8571093f"} Nov 01 21:13:46.144512 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:46.144510 4484 scope.go:122] "RemoveContainer" containerID="c040b0424a73a7fc9b8c22f4e7bfcb21e09d8f33a59865c6a6db3128c4ef431e" Nov 01 21:13:46.154978 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:46.154959 4484 scope.go:122] "RemoveContainer" containerID="093879cd3f9e7ce1ed44b391516fca5207ccb2ff263dfbb60a9a50f436e9d87c" Nov 01 21:13:46.164953 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:46.164926 4484 scope.go:122] "RemoveContainer" containerID="c040b0424a73a7fc9b8c22f4e7bfcb21e09d8f33a59865c6a6db3128c4ef431e" Nov 01 21:13:46.165422 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:13:46.165401 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c040b0424a73a7fc9b8c22f4e7bfcb21e09d8f33a59865c6a6db3128c4ef431e\": container with ID starting with c040b0424a73a7fc9b8c22f4e7bfcb21e09d8f33a59865c6a6db3128c4ef431e not found: ID does not exist" containerID="c040b0424a73a7fc9b8c22f4e7bfcb21e09d8f33a59865c6a6db3128c4ef431e" Nov 01 21:13:46.165510 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:46.165430 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c040b0424a73a7fc9b8c22f4e7bfcb21e09d8f33a59865c6a6db3128c4ef431e"} err="failed to get container status \"c040b0424a73a7fc9b8c22f4e7bfcb21e09d8f33a59865c6a6db3128c4ef431e\": rpc error: code = NotFound desc = could not find container \"c040b0424a73a7fc9b8c22f4e7bfcb21e09d8f33a59865c6a6db3128c4ef431e\": container with ID starting with c040b0424a73a7fc9b8c22f4e7bfcb21e09d8f33a59865c6a6db3128c4ef431e not found: ID does not exist" Nov 01 21:13:46.165510 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:46.165449 4484 scope.go:122] "RemoveContainer" containerID="093879cd3f9e7ce1ed44b391516fca5207ccb2ff263dfbb60a9a50f436e9d87c" Nov 01 21:13:46.165683 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:13:46.165662 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"093879cd3f9e7ce1ed44b391516fca5207ccb2ff263dfbb60a9a50f436e9d87c\": container with ID starting with 093879cd3f9e7ce1ed44b391516fca5207ccb2ff263dfbb60a9a50f436e9d87c not found: ID does not exist" containerID="093879cd3f9e7ce1ed44b391516fca5207ccb2ff263dfbb60a9a50f436e9d87c" Nov 01 21:13:46.165728 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:46.165691 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"093879cd3f9e7ce1ed44b391516fca5207ccb2ff263dfbb60a9a50f436e9d87c"} err="failed to get container status \"093879cd3f9e7ce1ed44b391516fca5207ccb2ff263dfbb60a9a50f436e9d87c\": rpc error: code = NotFound desc = could not find container \"093879cd3f9e7ce1ed44b391516fca5207ccb2ff263dfbb60a9a50f436e9d87c\": container with ID starting with 093879cd3f9e7ce1ed44b391516fca5207ccb2ff263dfbb60a9a50f436e9d87c not found: ID does not exist" Nov 01 21:13:46.264205 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:46.264090 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-istiod-ca-cert\") pod \"8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff\" (UID: \"8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff\") " Nov 01 21:13:46.264205 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:46.264165 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-istio-envoy\") pod \"8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff\" (UID: \"8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff\") " Nov 01 21:13:46.264533 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:46.264273 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-workload-certs\") pod \"8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff\" (UID: \"8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff\") " Nov 01 21:13:46.264533 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:46.264363 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-istio-data\") pod \"8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff\" (UID: \"8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff\") " Nov 01 21:13:46.264533 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:46.264442 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-kube-api-access-wlngd\" (UniqueName: \"kubernetes.io/projected/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-kube-api-access-wlngd\") pod \"8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff\" (UID: \"8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff\") " Nov 01 21:13:46.264533 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:46.264490 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-credential-socket\") pod \"8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff\" (UID: \"8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff\") " Nov 01 21:13:46.264783 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:46.264550 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-istio-podinfo\") pod \"8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff\" (UID: \"8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff\") " Nov 01 21:13:46.264783 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:46.264584 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-workload-socket\") pod \"8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff\" (UID: \"8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff\") " Nov 01 21:13:46.264783 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:46.264662 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-istio-token\" (UniqueName: \"kubernetes.io/projected/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-istio-token\") pod \"8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff\" (UID: \"8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff\") " Nov 01 21:13:46.264783 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:46.264681 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-istio-data" pod "8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff" (UID: "8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:13:46.265786 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:46.265696 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-credential-socket" pod "8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff" (UID: "8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:13:46.266122 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:46.266096 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-workload-certs" pod "8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff" (UID: "8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:13:46.266882 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:46.265112 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-istiod-ca-cert" pod "8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff" (UID: "8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:13:46.266882 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:46.266640 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-workload-socket" pod "8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff" (UID: "8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:13:46.267475 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:46.267439 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-istio-envoy" pod "8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff" (UID: "8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:13:46.267589 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:46.267464 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-kube-api-access-wlngd" pod "8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff" (UID: "8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff"). InnerVolumeSpecName "kube-api-access-wlngd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:13:46.268147 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:46.268121 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-istio-token" pod "8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff" (UID: "8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:13:46.273031 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:46.269846 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:46.273031 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:46.269875 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlngd\" (UniqueName: \"kubernetes.io/projected/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-kube-api-access-wlngd\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:46.273031 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:46.269899 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:46.273031 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:46.269916 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:46.273031 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:46.269931 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:46.273031 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:46.269944 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:46.273031 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:46.269963 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:46.277188 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:46.277143 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-istio-podinfo" pod "8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff" (UID: "8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:13:46.371114 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:46.371078 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:46.371114 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:46.371107 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:46.569984 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:46.569895 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc"] Nov 01 21:13:46.573058 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:46.573037 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-e0a6f5-openshift-default-5b74475bdc-rrrbc"] Nov 01 21:13:47.214791 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:47.214754 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-044440-79875748f5-mwv7c"] Nov 01 21:13:47.374320 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:47.374283 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff" path="/var/lib/kubelet/pods/8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff/volumes" Nov 01 21:13:47.491425 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:47.491394 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:47.675914 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:47.675826 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:49.816512 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:49.816478 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:51.451716 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:51.451685 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:51.612405 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:51.612369 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:52.215708 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:52.215639 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-044440-79875748f5-mwv7c" podUID="9027b7a9-7a43-4f47-961a-d02c04c52895" containerName="payload-pre-processing" containerID="cri-o://fdb2510dd61330ae38d7cf624d911819d939d24b37fa214a7337b77d9c364c69" gracePeriod=25 Nov 01 21:13:52.490882 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:52.490857 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-044440-79875748f5-mwv7c" Nov 01 21:13:52.683024 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:52.675577 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9027b7a9-7a43-4f47-961a-d02c04c52895-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9027b7a9-7a43-4f47-961a-d02c04c52895-plugins-config-volume\") pod \"9027b7a9-7a43-4f47-961a-d02c04c52895\" (UID: \"9027b7a9-7a43-4f47-961a-d02c04c52895\") " Nov 01 21:13:52.683024 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:52.675641 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9027b7a9-7a43-4f47-961a-d02c04c52895-kube-api-access-ltdbb\" (UniqueName: \"kubernetes.io/projected/9027b7a9-7a43-4f47-961a-d02c04c52895-kube-api-access-ltdbb\") pod \"9027b7a9-7a43-4f47-961a-d02c04c52895\" (UID: \"9027b7a9-7a43-4f47-961a-d02c04c52895\") " Nov 01 21:13:52.704265 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:52.690220 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9027b7a9-7a43-4f47-961a-d02c04c52895-kube-api-access-ltdbb" pod "9027b7a9-7a43-4f47-961a-d02c04c52895" (UID: "9027b7a9-7a43-4f47-961a-d02c04c52895"). InnerVolumeSpecName "kube-api-access-ltdbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:13:52.704265 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:52.690602 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9027b7a9-7a43-4f47-961a-d02c04c52895-plugins-config-volume" pod "9027b7a9-7a43-4f47-961a-d02c04c52895" (UID: "9027b7a9-7a43-4f47-961a-d02c04c52895"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:13:52.777148 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:52.777116 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9027b7a9-7a43-4f47-961a-d02c04c52895-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:52.777148 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:52.777144 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltdbb\" (UniqueName: \"kubernetes.io/projected/9027b7a9-7a43-4f47-961a-d02c04c52895-kube-api-access-ltdbb\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:53.198411 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:53.198325 4484 generic.go:356] "Generic (PLEG): container finished" podID="9027b7a9-7a43-4f47-961a-d02c04c52895" containerID="fdb2510dd61330ae38d7cf624d911819d939d24b37fa214a7337b77d9c364c69" exitCode=0 Nov 01 21:13:53.198411 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:53.198383 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-044440-79875748f5-mwv7c" Nov 01 21:13:53.198411 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:53.198399 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-044440-79875748f5-mwv7c" event={"ID":"9027b7a9-7a43-4f47-961a-d02c04c52895","Type":"ContainerDied","Data":"fdb2510dd61330ae38d7cf624d911819d939d24b37fa214a7337b77d9c364c69"} Nov 01 21:13:53.198686 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:53.198454 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-044440-79875748f5-mwv7c" event={"ID":"9027b7a9-7a43-4f47-961a-d02c04c52895","Type":"ContainerDied","Data":"28bfc598a4d84342a2c2a1d3bc7cfbe41d72d89bbd9c61fce9036deca2c4d4a8"} Nov 01 21:13:53.198686 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:53.198477 4484 scope.go:122] "RemoveContainer" containerID="fdb2510dd61330ae38d7cf624d911819d939d24b37fa214a7337b77d9c364c69" Nov 01 21:13:53.210685 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:53.210563 4484 scope.go:122] "RemoveContainer" containerID="fdb2510dd61330ae38d7cf624d911819d939d24b37fa214a7337b77d9c364c69" Nov 01 21:13:53.212078 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:13:53.212047 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdb2510dd61330ae38d7cf624d911819d939d24b37fa214a7337b77d9c364c69\": container with ID starting with fdb2510dd61330ae38d7cf624d911819d939d24b37fa214a7337b77d9c364c69 not found: ID does not exist" containerID="fdb2510dd61330ae38d7cf624d911819d939d24b37fa214a7337b77d9c364c69" Nov 01 21:13:53.212198 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:53.212085 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdb2510dd61330ae38d7cf624d911819d939d24b37fa214a7337b77d9c364c69"} err="failed to get container status \"fdb2510dd61330ae38d7cf624d911819d939d24b37fa214a7337b77d9c364c69\": rpc error: code = NotFound desc = could not find container \"fdb2510dd61330ae38d7cf624d911819d939d24b37fa214a7337b77d9c364c69\": container with ID starting with fdb2510dd61330ae38d7cf624d911819d939d24b37fa214a7337b77d9c364c69 not found: ID does not exist" Nov 01 21:13:53.372618 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:53.372585 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-044440-79875748f5-mwv7c"] Nov 01 21:13:53.416809 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:53.416773 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-044440-79875748f5-mwv7c"] Nov 01 21:13:54.336571 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:54.336537 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a41242-6bfc8bf88f-cs272"] Nov 01 21:13:54.449294 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:54.449262 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a41242-5956bf7979-dsqpz"] Nov 01 21:13:54.726843 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:54.726759 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:54.904325 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:54.904260 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:55.180615 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:55.180582 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:55.372283 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:55.372252 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9027b7a9-7a43-4f47-961a-d02c04c52895" path="/var/lib/kubelet/pods/9027b7a9-7a43-4f47-961a-d02c04c52895/volumes" Nov 01 21:13:59.337814 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:59.337764 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-a41242-6bfc8bf88f-cs272" podUID="2d919681-4cac-4aba-836b-88051de45b3f" containerName="payload-processing" containerID="cri-o://b68f986cf7861b744670b0766727381103890969bca916cdd51cea94e4687cc2" gracePeriod=25 Nov 01 21:13:59.449782 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:59.449734 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a41242-5956bf7979-dsqpz" podUID="b2d3a183-2efc-4840-be4d-6d583e48f3e2" containerName="payload-pre-processing" containerID="cri-o://c9fcc666b4805c194351590c916ac68dd661fd6120913d6792834661f4160bfe" gracePeriod=25 Nov 01 21:13:59.632731 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:59.632704 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a41242-6bfc8bf88f-cs272" Nov 01 21:13:59.702130 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:59.702099 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d919681-4cac-4aba-836b-88051de45b3f-kube-api-access-jqmsz\" (UniqueName: \"kubernetes.io/projected/2d919681-4cac-4aba-836b-88051de45b3f-kube-api-access-jqmsz\") pod \"2d919681-4cac-4aba-836b-88051de45b3f\" (UID: \"2d919681-4cac-4aba-836b-88051de45b3f\") " Nov 01 21:13:59.702334 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:59.702164 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d919681-4cac-4aba-836b-88051de45b3f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d919681-4cac-4aba-836b-88051de45b3f-plugins-config-volume\") pod \"2d919681-4cac-4aba-836b-88051de45b3f\" (UID: \"2d919681-4cac-4aba-836b-88051de45b3f\") " Nov 01 21:13:59.702615 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:59.702578 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d919681-4cac-4aba-836b-88051de45b3f-plugins-config-volume" pod "2d919681-4cac-4aba-836b-88051de45b3f" (UID: "2d919681-4cac-4aba-836b-88051de45b3f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:13:59.705035 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:59.704984 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d919681-4cac-4aba-836b-88051de45b3f-kube-api-access-jqmsz" pod "2d919681-4cac-4aba-836b-88051de45b3f" (UID: "2d919681-4cac-4aba-836b-88051de45b3f"). InnerVolumeSpecName "kube-api-access-jqmsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:13:59.728206 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:59.728180 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a41242-5956bf7979-dsqpz" Nov 01 21:13:59.803391 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:59.803359 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2d3a183-2efc-4840-be4d-6d583e48f3e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2d3a183-2efc-4840-be4d-6d583e48f3e2-plugins-config-volume\") pod \"b2d3a183-2efc-4840-be4d-6d583e48f3e2\" (UID: \"b2d3a183-2efc-4840-be4d-6d583e48f3e2\") " Nov 01 21:13:59.803570 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:59.803464 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2d3a183-2efc-4840-be4d-6d583e48f3e2-kube-api-access-dcqbj\" (UniqueName: \"kubernetes.io/projected/b2d3a183-2efc-4840-be4d-6d583e48f3e2-kube-api-access-dcqbj\") pod \"b2d3a183-2efc-4840-be4d-6d583e48f3e2\" (UID: \"b2d3a183-2efc-4840-be4d-6d583e48f3e2\") " Nov 01 21:13:59.803764 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:59.803739 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2d3a183-2efc-4840-be4d-6d583e48f3e2-plugins-config-volume" pod "b2d3a183-2efc-4840-be4d-6d583e48f3e2" (UID: "b2d3a183-2efc-4840-be4d-6d583e48f3e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:13:59.803918 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:59.803866 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2d3a183-2efc-4840-be4d-6d583e48f3e2-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:59.803918 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:59.803884 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqmsz\" (UniqueName: \"kubernetes.io/projected/2d919681-4cac-4aba-836b-88051de45b3f-kube-api-access-jqmsz\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:59.803918 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:59.803898 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d919681-4cac-4aba-836b-88051de45b3f-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:59.805540 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:59.805516 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2d3a183-2efc-4840-be4d-6d583e48f3e2-kube-api-access-dcqbj" pod "b2d3a183-2efc-4840-be4d-6d583e48f3e2" (UID: "b2d3a183-2efc-4840-be4d-6d583e48f3e2"). InnerVolumeSpecName "kube-api-access-dcqbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:13:59.905482 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:13:59.905399 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcqbj\" (UniqueName: \"kubernetes.io/projected/b2d3a183-2efc-4840-be4d-6d583e48f3e2-kube-api-access-dcqbj\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:00.266775 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:00.266743 4484 generic.go:356] "Generic (PLEG): container finished" podID="2d919681-4cac-4aba-836b-88051de45b3f" containerID="b68f986cf7861b744670b0766727381103890969bca916cdd51cea94e4687cc2" exitCode=0 Nov 01 21:14:00.266963 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:00.266790 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a41242-6bfc8bf88f-cs272" Nov 01 21:14:00.266963 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:00.266833 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a41242-6bfc8bf88f-cs272" event={"ID":"2d919681-4cac-4aba-836b-88051de45b3f","Type":"ContainerDied","Data":"b68f986cf7861b744670b0766727381103890969bca916cdd51cea94e4687cc2"} Nov 01 21:14:00.266963 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:00.266888 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a41242-6bfc8bf88f-cs272" event={"ID":"2d919681-4cac-4aba-836b-88051de45b3f","Type":"ContainerDied","Data":"66e1e466c983332a3b86067054d944f3a03a3409db3f92940a6f8dc314c3dcc1"} Nov 01 21:14:00.266963 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:00.266916 4484 scope.go:122] "RemoveContainer" containerID="b68f986cf7861b744670b0766727381103890969bca916cdd51cea94e4687cc2" Nov 01 21:14:00.268462 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:00.268440 4484 generic.go:356] "Generic (PLEG): container finished" podID="b2d3a183-2efc-4840-be4d-6d583e48f3e2" containerID="c9fcc666b4805c194351590c916ac68dd661fd6120913d6792834661f4160bfe" exitCode=0 Nov 01 21:14:00.268585 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:00.268482 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a41242-5956bf7979-dsqpz" Nov 01 21:14:00.268585 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:00.268536 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a41242-5956bf7979-dsqpz" event={"ID":"b2d3a183-2efc-4840-be4d-6d583e48f3e2","Type":"ContainerDied","Data":"c9fcc666b4805c194351590c916ac68dd661fd6120913d6792834661f4160bfe"} Nov 01 21:14:00.268701 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:00.268587 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a41242-5956bf7979-dsqpz" event={"ID":"b2d3a183-2efc-4840-be4d-6d583e48f3e2","Type":"ContainerDied","Data":"285944cd3e4c18903398dfa40e6871dc7b3a340cb9d845d33d0d30b6fbf1b382"} Nov 01 21:14:00.278029 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:00.278009 4484 scope.go:122] "RemoveContainer" containerID="b68f986cf7861b744670b0766727381103890969bca916cdd51cea94e4687cc2" Nov 01 21:14:00.278326 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:14:00.278303 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b68f986cf7861b744670b0766727381103890969bca916cdd51cea94e4687cc2\": container with ID starting with b68f986cf7861b744670b0766727381103890969bca916cdd51cea94e4687cc2 not found: ID does not exist" containerID="b68f986cf7861b744670b0766727381103890969bca916cdd51cea94e4687cc2" Nov 01 21:14:00.278441 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:00.278333 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b68f986cf7861b744670b0766727381103890969bca916cdd51cea94e4687cc2"} err="failed to get container status \"b68f986cf7861b744670b0766727381103890969bca916cdd51cea94e4687cc2\": rpc error: code = NotFound desc = could not find container \"b68f986cf7861b744670b0766727381103890969bca916cdd51cea94e4687cc2\": container with ID starting with b68f986cf7861b744670b0766727381103890969bca916cdd51cea94e4687cc2 not found: ID does not exist" Nov 01 21:14:00.278441 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:00.278355 4484 scope.go:122] "RemoveContainer" containerID="c9fcc666b4805c194351590c916ac68dd661fd6120913d6792834661f4160bfe" Nov 01 21:14:00.287760 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:00.287738 4484 scope.go:122] "RemoveContainer" containerID="c9fcc666b4805c194351590c916ac68dd661fd6120913d6792834661f4160bfe" Nov 01 21:14:00.288119 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:14:00.288093 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9fcc666b4805c194351590c916ac68dd661fd6120913d6792834661f4160bfe\": container with ID starting with c9fcc666b4805c194351590c916ac68dd661fd6120913d6792834661f4160bfe not found: ID does not exist" containerID="c9fcc666b4805c194351590c916ac68dd661fd6120913d6792834661f4160bfe" Nov 01 21:14:00.288205 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:00.288132 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9fcc666b4805c194351590c916ac68dd661fd6120913d6792834661f4160bfe"} err="failed to get container status \"c9fcc666b4805c194351590c916ac68dd661fd6120913d6792834661f4160bfe\": rpc error: code = NotFound desc = could not find container \"c9fcc666b4805c194351590c916ac68dd661fd6120913d6792834661f4160bfe\": container with ID starting with c9fcc666b4805c194351590c916ac68dd661fd6120913d6792834661f4160bfe not found: ID does not exist" Nov 01 21:14:00.423121 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:00.423083 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:00.435827 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:00.435790 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a41242-6bfc8bf88f-cs272"] Nov 01 21:14:00.438959 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:00.438935 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a41242-6bfc8bf88f-cs272"] Nov 01 21:14:00.546771 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:00.546665 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a41242-5956bf7979-dsqpz"] Nov 01 21:14:00.551793 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:00.551764 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a41242-5956bf7979-dsqpz"] Nov 01 21:14:01.373588 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:01.373506 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d919681-4cac-4aba-836b-88051de45b3f" path="/var/lib/kubelet/pods/2d919681-4cac-4aba-836b-88051de45b3f/volumes" Nov 01 21:14:01.373960 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:01.373945 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2d3a183-2efc-4840-be4d-6d583e48f3e2" path="/var/lib/kubelet/pods/b2d3a183-2efc-4840-be4d-6d583e48f3e2/volumes" Nov 01 21:14:05.555939 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:05.555898 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:07.047096 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:07.047057 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd0eb5ca-7hmxn"] Nov 01 21:14:07.098245 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:07.098213 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd0eb5ca-7hmxn"] Nov 01 21:14:07.373300 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:07.373210 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8347318b-2d4c-4db1-af2d-881a04c8a5c0" path="/var/lib/kubelet/pods/8347318b-2d4c-4db1-af2d-881a04c8a5c0/volumes" Nov 01 21:14:10.690715 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:10.690677 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:15.407289 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:15.407253 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3819a541-dh9c6"] Nov 01 21:14:15.407850 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:15.407835 4484 cpu_manager.go:414] "containerMap: removing container" podUID="2d919681-4cac-4aba-836b-88051de45b3f" containerName="payload-processing" Nov 01 21:14:15.407896 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:15.407851 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d919681-4cac-4aba-836b-88051de45b3f" containerName="payload-processing" Nov 01 21:14:15.407896 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:15.407865 4484 cpu_manager.go:414] "containerMap: removing container" podUID="8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff" containerName="istio-proxy" Nov 01 21:14:15.407896 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:15.407870 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff" containerName="istio-proxy" Nov 01 21:14:15.407896 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:15.407880 4484 cpu_manager.go:414] "containerMap: removing container" podUID="b2d3a183-2efc-4840-be4d-6d583e48f3e2" containerName="payload-pre-processing" Nov 01 21:14:15.407896 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:15.407885 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2d3a183-2efc-4840-be4d-6d583e48f3e2" containerName="payload-pre-processing" Nov 01 21:14:15.408096 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:15.407906 4484 cpu_manager.go:414] "containerMap: removing container" podUID="9027b7a9-7a43-4f47-961a-d02c04c52895" containerName="payload-pre-processing" Nov 01 21:14:15.408096 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:15.407912 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9027b7a9-7a43-4f47-961a-d02c04c52895" containerName="payload-pre-processing" Nov 01 21:14:15.408096 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:15.407924 4484 cpu_manager.go:414] "containerMap: removing container" podUID="8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff" containerName="istio-proxy" Nov 01 21:14:15.408096 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:15.407929 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff" containerName="istio-proxy" Nov 01 21:14:15.408096 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:15.407939 4484 cpu_manager.go:414] "containerMap: removing container" podUID="c259f9d2-8a0f-411d-bc9d-61644916daee" containerName="revoke-keys" Nov 01 21:14:15.408096 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:15.407944 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c259f9d2-8a0f-411d-bc9d-61644916daee" containerName="revoke-keys" Nov 01 21:14:15.408096 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:15.408054 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d919681-4cac-4aba-836b-88051de45b3f" containerName="payload-processing" Nov 01 21:14:15.408096 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:15.408064 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff" containerName="istio-proxy" Nov 01 21:14:15.408096 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:15.408070 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2d3a183-2efc-4840-be4d-6d583e48f3e2" containerName="payload-pre-processing" Nov 01 21:14:15.408096 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:15.408079 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="9027b7a9-7a43-4f47-961a-d02c04c52895" containerName="payload-pre-processing" Nov 01 21:14:15.408096 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:15.408086 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ef2ec69-fb02-409c-94cb-4e1cfa9d06ff" containerName="istio-proxy" Nov 01 21:14:15.408096 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:15.408093 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="c259f9d2-8a0f-411d-bc9d-61644916daee" containerName="revoke-keys" Nov 01 21:14:15.408096 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:15.408099 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="69124be6-394b-45bd-9d52-6de87dfc5648" containerName="istio-proxy" Nov 01 21:14:15.413502 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:15.413483 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3819a541-dh9c6" Nov 01 21:14:15.465192 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:15.465146 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04f5a54f-88jsj"] Nov 01 21:14:15.470075 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:15.470050 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3819a541-dh9c6"] Nov 01 21:14:15.470219 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:15.470145 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f5a54f-88jsj" Nov 01 21:14:15.511643 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:15.511601 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58935e56-56fd-4275-bd56-5a4b28f2c186-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04f5a54f-88jsj\" (UID: \"58935e56-56fd-4275-bd56-5a4b28f2c186\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f5a54f-88jsj" Nov 01 21:14:15.511824 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:15.511719 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58935e56-56fd-4275-bd56-5a4b28f2c186-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04f5a54f-88jsj\" (UID: \"58935e56-56fd-4275-bd56-5a4b28f2c186\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f5a54f-88jsj" Nov 01 21:14:15.511824 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:15.511764 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ae8b00c-b101-4a28-a618-a7f2b0ce4bed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3819a541-dh9c6\" (UID: \"5ae8b00c-b101-4a28-a618-a7f2b0ce4bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3819a541-dh9c6" Nov 01 21:14:15.511824 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:15.511819 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcrjw\" (UniqueName: \"kubernetes.io/projected/5ae8b00c-b101-4a28-a618-a7f2b0ce4bed-kube-api-access-wcrjw\") pod \"maas-api-revoke-sub-3819a541-dh9c6\" (UID: \"5ae8b00c-b101-4a28-a618-a7f2b0ce4bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3819a541-dh9c6" Nov 01 21:14:15.511959 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:15.511917 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ae8b00c-b101-4a28-a618-a7f2b0ce4bed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3819a541-dh9c6\" (UID: \"5ae8b00c-b101-4a28-a618-a7f2b0ce4bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3819a541-dh9c6" Nov 01 21:14:15.512015 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:15.511968 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkh6h\" (UniqueName: \"kubernetes.io/projected/58935e56-56fd-4275-bd56-5a4b28f2c186-kube-api-access-fkh6h\") pod \"maas-api-revoke-sub-04f5a54f-88jsj\" (UID: \"58935e56-56fd-4275-bd56-5a4b28f2c186\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f5a54f-88jsj" Nov 01 21:14:15.512425 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:15.512399 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:14:15.515180 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:15.515158 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:14:15.515335 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:15.515217 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:14:15.515335 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:15.515303 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:14:15.524469 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:15.524429 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04f5a54f-88jsj"] Nov 01 21:14:15.612934 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:15.612895 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58935e56-56fd-4275-bd56-5a4b28f2c186-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04f5a54f-88jsj\" (UID: \"58935e56-56fd-4275-bd56-5a4b28f2c186\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f5a54f-88jsj" Nov 01 21:14:15.613189 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:15.612944 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ae8b00c-b101-4a28-a618-a7f2b0ce4bed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3819a541-dh9c6\" (UID: \"5ae8b00c-b101-4a28-a618-a7f2b0ce4bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3819a541-dh9c6" Nov 01 21:14:15.613189 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:15.613030 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcrjw\" (UniqueName: \"kubernetes.io/projected/5ae8b00c-b101-4a28-a618-a7f2b0ce4bed-kube-api-access-wcrjw\") pod \"maas-api-revoke-sub-3819a541-dh9c6\" (UID: \"5ae8b00c-b101-4a28-a618-a7f2b0ce4bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3819a541-dh9c6" Nov 01 21:14:15.613189 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:15.613090 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ae8b00c-b101-4a28-a618-a7f2b0ce4bed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3819a541-dh9c6\" (UID: \"5ae8b00c-b101-4a28-a618-a7f2b0ce4bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3819a541-dh9c6" Nov 01 21:14:15.613189 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:15.613150 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkh6h\" (UniqueName: \"kubernetes.io/projected/58935e56-56fd-4275-bd56-5a4b28f2c186-kube-api-access-fkh6h\") pod \"maas-api-revoke-sub-04f5a54f-88jsj\" (UID: \"58935e56-56fd-4275-bd56-5a4b28f2c186\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f5a54f-88jsj" Nov 01 21:14:15.613424 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:15.613236 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58935e56-56fd-4275-bd56-5a4b28f2c186-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04f5a54f-88jsj\" (UID: \"58935e56-56fd-4275-bd56-5a4b28f2c186\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f5a54f-88jsj" Nov 01 21:14:15.613885 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:15.613860 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58935e56-56fd-4275-bd56-5a4b28f2c186-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04f5a54f-88jsj\" (UID: \"58935e56-56fd-4275-bd56-5a4b28f2c186\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f5a54f-88jsj" Nov 01 21:14:15.614302 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:15.614283 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ae8b00c-b101-4a28-a618-a7f2b0ce4bed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3819a541-dh9c6\" (UID: \"5ae8b00c-b101-4a28-a618-a7f2b0ce4bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3819a541-dh9c6" Nov 01 21:14:15.615364 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:15.615337 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ae8b00c-b101-4a28-a618-a7f2b0ce4bed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3819a541-dh9c6\" (UID: \"5ae8b00c-b101-4a28-a618-a7f2b0ce4bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3819a541-dh9c6" Nov 01 21:14:15.615541 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:15.615520 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58935e56-56fd-4275-bd56-5a4b28f2c186-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04f5a54f-88jsj\" (UID: \"58935e56-56fd-4275-bd56-5a4b28f2c186\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f5a54f-88jsj" Nov 01 21:14:15.671096 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:15.671021 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcrjw\" (UniqueName: \"kubernetes.io/projected/5ae8b00c-b101-4a28-a618-a7f2b0ce4bed-kube-api-access-wcrjw\") pod \"maas-api-revoke-sub-3819a541-dh9c6\" (UID: \"5ae8b00c-b101-4a28-a618-a7f2b0ce4bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3819a541-dh9c6" Nov 01 21:14:15.671096 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:15.671033 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkh6h\" (UniqueName: \"kubernetes.io/projected/58935e56-56fd-4275-bd56-5a4b28f2c186-kube-api-access-fkh6h\") pod \"maas-api-revoke-sub-04f5a54f-88jsj\" (UID: \"58935e56-56fd-4275-bd56-5a4b28f2c186\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f5a54f-88jsj" Nov 01 21:14:15.727672 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:15.727635 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3819a541-dh9c6" Nov 01 21:14:15.770507 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:14:15.770468 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:14:15.791288 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:15.791258 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f5a54f-88jsj" Nov 01 21:14:16.108700 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:16.108671 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3819a541-dh9c6"] Nov 01 21:14:16.109131 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:14:16.109103 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ae8b00c_b101_4a28_a618_a7f2b0ce4bed.slice/crio-dd1f115cb35b4f87877759297a2dd68fd6d40f9d339b99505238f16990870f21 WatchSource:0}: Error finding container dd1f115cb35b4f87877759297a2dd68fd6d40f9d339b99505238f16990870f21: Status 404 returned error can't find the container with id dd1f115cb35b4f87877759297a2dd68fd6d40f9d339b99505238f16990870f21 Nov 01 21:14:16.218041 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:16.218007 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04f5a54f-88jsj"] Nov 01 21:14:16.219282 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:14:16.219256 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58935e56_56fd_4275_bd56_5a4b28f2c186.slice/crio-805f12a3faa46e24fd944dc7ffc4bf860db10ef6a72e76a35c4c130fdb3a9f01 WatchSource:0}: Error finding container 805f12a3faa46e24fd944dc7ffc4bf860db10ef6a72e76a35c4c130fdb3a9f01: Status 404 returned error can't find the container with id 805f12a3faa46e24fd944dc7ffc4bf860db10ef6a72e76a35c4c130fdb3a9f01 Nov 01 21:14:16.252101 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:16.252073 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:16.381524 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:16.381434 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3819a541-dh9c6" event={"ID":"5ae8b00c-b101-4a28-a618-a7f2b0ce4bed","Type":"ContainerStarted","Data":"dd1f115cb35b4f87877759297a2dd68fd6d40f9d339b99505238f16990870f21"} Nov 01 21:14:16.382819 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:16.382788 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f5a54f-88jsj" event={"ID":"58935e56-56fd-4275-bd56-5a4b28f2c186","Type":"ContainerStarted","Data":"805f12a3faa46e24fd944dc7ffc4bf860db10ef6a72e76a35c4c130fdb3a9f01"} Nov 01 21:14:17.390365 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:17.390274 4484 generic.go:356] "Generic (PLEG): container finished" podID="58935e56-56fd-4275-bd56-5a4b28f2c186" containerID="470dc8d703080ade25b2e65b5f8ea500a1199dc1cc6f2736df1ebef44b6d63d1" exitCode=0 Nov 01 21:14:17.390775 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:17.390364 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f5a54f-88jsj" event={"ID":"58935e56-56fd-4275-bd56-5a4b28f2c186","Type":"ContainerDied","Data":"470dc8d703080ade25b2e65b5f8ea500a1199dc1cc6f2736df1ebef44b6d63d1"} Nov 01 21:14:17.392047 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:17.392020 4484 generic.go:356] "Generic (PLEG): container finished" podID="5ae8b00c-b101-4a28-a618-a7f2b0ce4bed" containerID="5b843468db3131a27782cc0d7a399657c2be47ac4c09c0f32b2d196425c7927b" exitCode=0 Nov 01 21:14:17.392191 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:17.392092 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3819a541-dh9c6" event={"ID":"5ae8b00c-b101-4a28-a618-a7f2b0ce4bed","Type":"ContainerDied","Data":"5b843468db3131a27782cc0d7a399657c2be47ac4c09c0f32b2d196425c7927b"} Nov 01 21:14:18.581045 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:18.581019 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f5a54f-88jsj" Nov 01 21:14:18.584542 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:18.584523 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3819a541-dh9c6" Nov 01 21:14:18.650748 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:18.650654 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ae8b00c-b101-4a28-a618-a7f2b0ce4bed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ae8b00c-b101-4a28-a618-a7f2b0ce4bed-maas-api-openshift-ca\") pod \"5ae8b00c-b101-4a28-a618-a7f2b0ce4bed\" (UID: \"5ae8b00c-b101-4a28-a618-a7f2b0ce4bed\") " Nov 01 21:14:18.650748 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:18.650705 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ae8b00c-b101-4a28-a618-a7f2b0ce4bed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ae8b00c-b101-4a28-a618-a7f2b0ce4bed-maas-api-xks-ca\") pod \"5ae8b00c-b101-4a28-a618-a7f2b0ce4bed\" (UID: \"5ae8b00c-b101-4a28-a618-a7f2b0ce4bed\") " Nov 01 21:14:18.651039 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:18.650826 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58935e56-56fd-4275-bd56-5a4b28f2c186-kube-api-access-fkh6h\" (UniqueName: \"kubernetes.io/projected/58935e56-56fd-4275-bd56-5a4b28f2c186-kube-api-access-fkh6h\") pod \"58935e56-56fd-4275-bd56-5a4b28f2c186\" (UID: \"58935e56-56fd-4275-bd56-5a4b28f2c186\") " Nov 01 21:14:18.651039 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:18.650900 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58935e56-56fd-4275-bd56-5a4b28f2c186-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58935e56-56fd-4275-bd56-5a4b28f2c186-maas-api-openshift-ca\") pod \"58935e56-56fd-4275-bd56-5a4b28f2c186\" (UID: \"58935e56-56fd-4275-bd56-5a4b28f2c186\") " Nov 01 21:14:18.651039 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:18.650961 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58935e56-56fd-4275-bd56-5a4b28f2c186-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58935e56-56fd-4275-bd56-5a4b28f2c186-maas-api-xks-ca\") pod \"58935e56-56fd-4275-bd56-5a4b28f2c186\" (UID: \"58935e56-56fd-4275-bd56-5a4b28f2c186\") " Nov 01 21:14:18.651224 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:18.651071 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ae8b00c-b101-4a28-a618-a7f2b0ce4bed-kube-api-access-wcrjw\" (UniqueName: \"kubernetes.io/projected/5ae8b00c-b101-4a28-a618-a7f2b0ce4bed-kube-api-access-wcrjw\") pod \"5ae8b00c-b101-4a28-a618-a7f2b0ce4bed\" (UID: \"5ae8b00c-b101-4a28-a618-a7f2b0ce4bed\") " Nov 01 21:14:18.651224 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:18.651119 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ae8b00c-b101-4a28-a618-a7f2b0ce4bed-maas-api-openshift-ca" pod "5ae8b00c-b101-4a28-a618-a7f2b0ce4bed" (UID: "5ae8b00c-b101-4a28-a618-a7f2b0ce4bed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:14:18.651335 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:18.651295 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58935e56-56fd-4275-bd56-5a4b28f2c186-maas-api-openshift-ca" pod "58935e56-56fd-4275-bd56-5a4b28f2c186" (UID: "58935e56-56fd-4275-bd56-5a4b28f2c186"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:14:18.651699 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:18.651675 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ae8b00c-b101-4a28-a618-a7f2b0ce4bed-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:18.651799 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:18.651704 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58935e56-56fd-4275-bd56-5a4b28f2c186-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:18.653365 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:18.653338 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ae8b00c-b101-4a28-a618-a7f2b0ce4bed-maas-api-xks-ca" pod "5ae8b00c-b101-4a28-a618-a7f2b0ce4bed" (UID: "5ae8b00c-b101-4a28-a618-a7f2b0ce4bed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:14:18.653599 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:18.653575 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58935e56-56fd-4275-bd56-5a4b28f2c186-maas-api-xks-ca" pod "58935e56-56fd-4275-bd56-5a4b28f2c186" (UID: "58935e56-56fd-4275-bd56-5a4b28f2c186"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:14:18.653682 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:18.653607 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58935e56-56fd-4275-bd56-5a4b28f2c186-kube-api-access-fkh6h" pod "58935e56-56fd-4275-bd56-5a4b28f2c186" (UID: "58935e56-56fd-4275-bd56-5a4b28f2c186"). InnerVolumeSpecName "kube-api-access-fkh6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:14:18.654403 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:18.654374 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ae8b00c-b101-4a28-a618-a7f2b0ce4bed-kube-api-access-wcrjw" pod "5ae8b00c-b101-4a28-a618-a7f2b0ce4bed" (UID: "5ae8b00c-b101-4a28-a618-a7f2b0ce4bed"). InnerVolumeSpecName "kube-api-access-wcrjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:14:18.752980 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:18.752946 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkh6h\" (UniqueName: \"kubernetes.io/projected/58935e56-56fd-4275-bd56-5a4b28f2c186-kube-api-access-fkh6h\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:18.752980 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:18.752974 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58935e56-56fd-4275-bd56-5a4b28f2c186-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:18.752980 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:18.752985 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcrjw\" (UniqueName: \"kubernetes.io/projected/5ae8b00c-b101-4a28-a618-a7f2b0ce4bed-kube-api-access-wcrjw\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:18.752980 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:18.753005 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ae8b00c-b101-4a28-a618-a7f2b0ce4bed-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:19.407075 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:19.407038 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f5a54f-88jsj" event={"ID":"58935e56-56fd-4275-bd56-5a4b28f2c186","Type":"ContainerDied","Data":"805f12a3faa46e24fd944dc7ffc4bf860db10ef6a72e76a35c4c130fdb3a9f01"} Nov 01 21:14:19.407261 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:19.407099 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="805f12a3faa46e24fd944dc7ffc4bf860db10ef6a72e76a35c4c130fdb3a9f01" Nov 01 21:14:19.407261 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:19.407056 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f5a54f-88jsj" Nov 01 21:14:19.408458 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:19.408430 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3819a541-dh9c6" event={"ID":"5ae8b00c-b101-4a28-a618-a7f2b0ce4bed","Type":"ContainerDied","Data":"dd1f115cb35b4f87877759297a2dd68fd6d40f9d339b99505238f16990870f21"} Nov 01 21:14:19.408595 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:19.408467 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3819a541-dh9c6" Nov 01 21:14:19.408595 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:19.408471 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd1f115cb35b4f87877759297a2dd68fd6d40f9d339b99505238f16990870f21" Nov 01 21:14:20.737855 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:20.737824 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:22.413240 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:22.413199 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:22.633658 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:22.633625 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:22.798968 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:22.798931 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:22.887753 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:22.887721 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7091fd-24flr"] Nov 01 21:14:22.937934 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:22.937897 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7091fd-24flr"] Nov 01 21:14:23.349564 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:23.349528 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:23.373128 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:23.373097 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f2370cd-fd08-456b-a312-0b648cd4ebea" path="/var/lib/kubelet/pods/2f2370cd-fd08-456b-a312-0b648cd4ebea/volumes" Nov 01 21:14:27.759446 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:27.759383 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:28.106273 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:28.106169 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:28.262663 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:28.262617 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:29.065321 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:29.065279 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:29.217618 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:29.217583 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:29.359988 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:29.359902 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td"] Nov 01 21:14:29.360206 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:29.360178 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td" podUID="f2ddd589-df2c-4d9e-a504-dd6029e7f99b" containerName="istio-proxy" containerID="cri-o://1367aa64dd18542dca6efa2a415391c8daf86cb47e3ffaf953b8767ac5144a63" gracePeriod=30 Nov 01 21:14:29.377065 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:29.377027 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh"] Nov 01 21:14:29.377242 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:29.377222 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh" podUID="d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe" containerName="istio-proxy" containerID="cri-o://c4bda6656f8f23bccaf4362114fd946e89fd87aed8cc7825ea9178be76944b79" gracePeriod=30 Nov 01 21:14:29.390550 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:29.390521 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:30.228638 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:30.228603 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-dce296-m5bf4"] Nov 01 21:14:30.229272 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:30.229255 4484 cpu_manager.go:414] "containerMap: removing container" podUID="5ae8b00c-b101-4a28-a618-a7f2b0ce4bed" containerName="revoke-keys" Nov 01 21:14:30.229317 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:30.229273 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ae8b00c-b101-4a28-a618-a7f2b0ce4bed" containerName="revoke-keys" Nov 01 21:14:30.229317 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:30.229306 4484 cpu_manager.go:414] "containerMap: removing container" podUID="58935e56-56fd-4275-bd56-5a4b28f2c186" containerName="revoke-keys" Nov 01 21:14:30.229317 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:30.229312 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58935e56-56fd-4275-bd56-5a4b28f2c186" containerName="revoke-keys" Nov 01 21:14:30.229416 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:30.229407 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="58935e56-56fd-4275-bd56-5a4b28f2c186" containerName="revoke-keys" Nov 01 21:14:30.229464 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:30.229423 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ae8b00c-b101-4a28-a618-a7f2b0ce4bed" containerName="revoke-keys" Nov 01 21:14:30.234299 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:30.234275 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-dce296-m5bf4" Nov 01 21:14:30.287638 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:30.287605 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:14:30.290305 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:30.290279 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-dce296-m5bf4"] Nov 01 21:14:30.310473 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:30.310419 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24b80eae-a972-46e8-bd86-f67c5452d3b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-dce296-m5bf4\" (UID: \"24b80eae-a972-46e8-bd86-f67c5452d3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-dce296-m5bf4" Nov 01 21:14:30.310686 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:30.310484 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r74b\" (UniqueName: \"kubernetes.io/projected/24b80eae-a972-46e8-bd86-f67c5452d3b1-kube-api-access-7r74b\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-dce296-m5bf4\" (UID: \"24b80eae-a972-46e8-bd86-f67c5452d3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-dce296-m5bf4" Nov 01 21:14:30.310827 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:30.310784 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24b80eae-a972-46e8-bd86-f67c5452d3b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-dce296-m5bf4\" (UID: \"24b80eae-a972-46e8-bd86-f67c5452d3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-dce296-m5bf4" Nov 01 21:14:30.325927 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:30.325899 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:14:30.335675 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:30.335647 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:14:30.335675 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:30.335647 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:14:30.412189 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:30.412151 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24b80eae-a972-46e8-bd86-f67c5452d3b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-dce296-m5bf4\" (UID: \"24b80eae-a972-46e8-bd86-f67c5452d3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-dce296-m5bf4" Nov 01 21:14:30.412417 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:30.412218 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24b80eae-a972-46e8-bd86-f67c5452d3b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-dce296-m5bf4\" (UID: \"24b80eae-a972-46e8-bd86-f67c5452d3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-dce296-m5bf4" Nov 01 21:14:30.412417 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:30.412311 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r74b\" (UniqueName: \"kubernetes.io/projected/24b80eae-a972-46e8-bd86-f67c5452d3b1-kube-api-access-7r74b\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-dce296-m5bf4\" (UID: \"24b80eae-a972-46e8-bd86-f67c5452d3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-dce296-m5bf4" Nov 01 21:14:30.412908 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:30.412883 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24b80eae-a972-46e8-bd86-f67c5452d3b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-dce296-m5bf4\" (UID: \"24b80eae-a972-46e8-bd86-f67c5452d3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-dce296-m5bf4" Nov 01 21:14:30.414714 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:30.414695 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24b80eae-a972-46e8-bd86-f67c5452d3b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-dce296-m5bf4\" (UID: \"24b80eae-a972-46e8-bd86-f67c5452d3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-dce296-m5bf4" Nov 01 21:14:30.469961 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:30.469924 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r74b\" (UniqueName: \"kubernetes.io/projected/24b80eae-a972-46e8-bd86-f67c5452d3b1-kube-api-access-7r74b\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-dce296-m5bf4\" (UID: \"24b80eae-a972-46e8-bd86-f67c5452d3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-dce296-m5bf4" Nov 01 21:14:30.547972 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:30.547939 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-dce296-m5bf4" Nov 01 21:14:30.924237 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:30.924206 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-dce296-m5bf4"] Nov 01 21:14:30.924646 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:14:30.924624 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24b80eae_a972_46e8_bd86_f67c5452d3b1.slice/crio-488269b8d5b16c09082794e1b6c5e8f336c7dc072930dd1f129909a620062aa7 WatchSource:0}: Error finding container 488269b8d5b16c09082794e1b6c5e8f336c7dc072930dd1f129909a620062aa7: Status 404 returned error can't find the container with id 488269b8d5b16c09082794e1b6c5e8f336c7dc072930dd1f129909a620062aa7 Nov 01 21:14:31.504686 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:31.504631 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-dce296-m5bf4" event={"ID":"24b80eae-a972-46e8-bd86-f67c5452d3b1","Type":"ContainerStarted","Data":"1e7fac1909090041b59086d6b03c0a3d0807a204641f4c05caa3da1d63f5fb9c"} Nov 01 21:14:31.505187 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:31.504699 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-dce296-m5bf4" event={"ID":"24b80eae-a972-46e8-bd86-f67c5452d3b1","Type":"ContainerStarted","Data":"488269b8d5b16c09082794e1b6c5e8f336c7dc072930dd1f129909a620062aa7"} Nov 01 21:14:31.623445 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:31.623391 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-dce296-m5bf4" podStartSLOduration=1.6233771049999999 podStartE2EDuration="1.623377105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:14:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:14:31.620814969 +0000 UTC m=+2622.398455103" watchObservedRunningTime="2026-11-01 21:14:31.623377105 +0000 UTC m=+2622.401017234" Nov 01 21:14:32.156407 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:32.156319 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:32.514042 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:32.513991 4484 generic.go:356] "Generic (PLEG): container finished" podID="24b80eae-a972-46e8-bd86-f67c5452d3b1" containerID="1e7fac1909090041b59086d6b03c0a3d0807a204641f4c05caa3da1d63f5fb9c" exitCode=0 Nov 01 21:14:32.514452 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:32.514044 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-dce296-m5bf4" event={"ID":"24b80eae-a972-46e8-bd86-f67c5452d3b1","Type":"ContainerDied","Data":"1e7fac1909090041b59086d6b03c0a3d0807a204641f4c05caa3da1d63f5fb9c"} Nov 01 21:14:33.669064 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:33.669037 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-dce296-m5bf4" Nov 01 21:14:33.769923 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:33.769886 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24b80eae-a972-46e8-bd86-f67c5452d3b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24b80eae-a972-46e8-bd86-f67c5452d3b1-maas-api-xks-ca\") pod \"24b80eae-a972-46e8-bd86-f67c5452d3b1\" (UID: \"24b80eae-a972-46e8-bd86-f67c5452d3b1\") " Nov 01 21:14:33.770119 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:33.769952 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24b80eae-a972-46e8-bd86-f67c5452d3b1-kube-api-access-7r74b\" (UniqueName: \"kubernetes.io/projected/24b80eae-a972-46e8-bd86-f67c5452d3b1-kube-api-access-7r74b\") pod \"24b80eae-a972-46e8-bd86-f67c5452d3b1\" (UID: \"24b80eae-a972-46e8-bd86-f67c5452d3b1\") " Nov 01 21:14:33.770119 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:33.770034 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24b80eae-a972-46e8-bd86-f67c5452d3b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24b80eae-a972-46e8-bd86-f67c5452d3b1-maas-api-openshift-ca\") pod \"24b80eae-a972-46e8-bd86-f67c5452d3b1\" (UID: \"24b80eae-a972-46e8-bd86-f67c5452d3b1\") " Nov 01 21:14:33.770404 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:33.770368 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24b80eae-a972-46e8-bd86-f67c5452d3b1-maas-api-openshift-ca" pod "24b80eae-a972-46e8-bd86-f67c5452d3b1" (UID: "24b80eae-a972-46e8-bd86-f67c5452d3b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:14:33.770556 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:33.770539 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24b80eae-a972-46e8-bd86-f67c5452d3b1-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:33.772178 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:33.772122 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24b80eae-a972-46e8-bd86-f67c5452d3b1-maas-api-xks-ca" pod "24b80eae-a972-46e8-bd86-f67c5452d3b1" (UID: "24b80eae-a972-46e8-bd86-f67c5452d3b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:14:33.772425 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:33.772410 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24b80eae-a972-46e8-bd86-f67c5452d3b1-kube-api-access-7r74b" pod "24b80eae-a972-46e8-bd86-f67c5452d3b1" (UID: "24b80eae-a972-46e8-bd86-f67c5452d3b1"). InnerVolumeSpecName "kube-api-access-7r74b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:14:33.797321 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:33.797284 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:33.871625 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:33.871587 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24b80eae-a972-46e8-bd86-f67c5452d3b1-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:33.871625 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:33.871619 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r74b\" (UniqueName: \"kubernetes.io/projected/24b80eae-a972-46e8-bd86-f67c5452d3b1-kube-api-access-7r74b\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:34.006928 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.006892 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:34.158298 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.158206 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:34.344026 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.343980 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-deff5e-cbgm9"] Nov 01 21:14:34.347021 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.346986 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-deff5e-cbgm9"] Nov 01 21:14:34.533201 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.533168 4484 generic.go:356] "Generic (PLEG): container finished" podID="f2ddd589-df2c-4d9e-a504-dd6029e7f99b" containerID="1367aa64dd18542dca6efa2a415391c8daf86cb47e3ffaf953b8767ac5144a63" exitCode=0 Nov 01 21:14:34.533409 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.533257 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td" event={"ID":"f2ddd589-df2c-4d9e-a504-dd6029e7f99b","Type":"ContainerDied","Data":"1367aa64dd18542dca6efa2a415391c8daf86cb47e3ffaf953b8767ac5144a63"} Nov 01 21:14:34.535034 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.534831 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-dce296-m5bf4" Nov 01 21:14:34.535034 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.534846 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-dce296-m5bf4" event={"ID":"24b80eae-a972-46e8-bd86-f67c5452d3b1","Type":"ContainerDied","Data":"488269b8d5b16c09082794e1b6c5e8f336c7dc072930dd1f129909a620062aa7"} Nov 01 21:14:34.535034 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.534884 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="488269b8d5b16c09082794e1b6c5e8f336c7dc072930dd1f129909a620062aa7" Nov 01 21:14:34.537358 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.537336 4484 generic.go:356] "Generic (PLEG): container finished" podID="d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe" containerID="c4bda6656f8f23bccaf4362114fd946e89fd87aed8cc7825ea9178be76944b79" exitCode=0 Nov 01 21:14:34.537461 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.537365 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh" event={"ID":"d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe","Type":"ContainerDied","Data":"c4bda6656f8f23bccaf4362114fd946e89fd87aed8cc7825ea9178be76944b79"} Nov 01 21:14:34.701422 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.701396 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td" Nov 01 21:14:34.724523 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.724501 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh" Nov 01 21:14:34.746508 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.746476 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:34.794863 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.794824 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-istio-podinfo\") pod \"f2ddd589-df2c-4d9e-a504-dd6029e7f99b\" (UID: \"f2ddd589-df2c-4d9e-a504-dd6029e7f99b\") " Nov 01 21:14:34.794863 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.794865 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-kube-api-access-q9f6j\" (UniqueName: \"kubernetes.io/projected/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-kube-api-access-q9f6j\") pod \"f2ddd589-df2c-4d9e-a504-dd6029e7f99b\" (UID: \"f2ddd589-df2c-4d9e-a504-dd6029e7f99b\") " Nov 01 21:14:34.795159 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.794939 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-workload-certs\") pod \"f2ddd589-df2c-4d9e-a504-dd6029e7f99b\" (UID: \"f2ddd589-df2c-4d9e-a504-dd6029e7f99b\") " Nov 01 21:14:34.795159 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.795032 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-istio-token\" (UniqueName: \"kubernetes.io/projected/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-istio-token\") pod \"f2ddd589-df2c-4d9e-a504-dd6029e7f99b\" (UID: \"f2ddd589-df2c-4d9e-a504-dd6029e7f99b\") " Nov 01 21:14:34.795159 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.795059 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-istiod-ca-cert\") pod \"f2ddd589-df2c-4d9e-a504-dd6029e7f99b\" (UID: \"f2ddd589-df2c-4d9e-a504-dd6029e7f99b\") " Nov 01 21:14:34.795159 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.795083 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-istio-data\") pod \"f2ddd589-df2c-4d9e-a504-dd6029e7f99b\" (UID: \"f2ddd589-df2c-4d9e-a504-dd6029e7f99b\") " Nov 01 21:14:34.795159 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.795121 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-istio-envoy\") pod \"f2ddd589-df2c-4d9e-a504-dd6029e7f99b\" (UID: \"f2ddd589-df2c-4d9e-a504-dd6029e7f99b\") " Nov 01 21:14:34.795420 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.795190 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-credential-socket\") pod \"f2ddd589-df2c-4d9e-a504-dd6029e7f99b\" (UID: \"f2ddd589-df2c-4d9e-a504-dd6029e7f99b\") " Nov 01 21:14:34.795420 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.795224 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-workload-socket\") pod \"f2ddd589-df2c-4d9e-a504-dd6029e7f99b\" (UID: \"f2ddd589-df2c-4d9e-a504-dd6029e7f99b\") " Nov 01 21:14:34.795420 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.795385 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-workload-certs" pod "f2ddd589-df2c-4d9e-a504-dd6029e7f99b" (UID: "f2ddd589-df2c-4d9e-a504-dd6029e7f99b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:14:34.795616 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.795591 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-workload-socket" pod "f2ddd589-df2c-4d9e-a504-dd6029e7f99b" (UID: "f2ddd589-df2c-4d9e-a504-dd6029e7f99b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:14:34.795616 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.795512 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-istiod-ca-cert" pod "f2ddd589-df2c-4d9e-a504-dd6029e7f99b" (UID: "f2ddd589-df2c-4d9e-a504-dd6029e7f99b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:14:34.796109 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.795800 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-credential-socket" pod "f2ddd589-df2c-4d9e-a504-dd6029e7f99b" (UID: "f2ddd589-df2c-4d9e-a504-dd6029e7f99b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:14:34.796109 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.795818 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:34.796109 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.795814 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-istio-data" pod "f2ddd589-df2c-4d9e-a504-dd6029e7f99b" (UID: "f2ddd589-df2c-4d9e-a504-dd6029e7f99b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:14:34.796109 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.795839 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:34.796109 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.795858 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:34.797286 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.797261 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-istio-podinfo" pod "f2ddd589-df2c-4d9e-a504-dd6029e7f99b" (UID: "f2ddd589-df2c-4d9e-a504-dd6029e7f99b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:14:34.797418 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.797369 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-kube-api-access-q9f6j" pod "f2ddd589-df2c-4d9e-a504-dd6029e7f99b" (UID: "f2ddd589-df2c-4d9e-a504-dd6029e7f99b"). InnerVolumeSpecName "kube-api-access-q9f6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:14:34.797418 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.797375 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-istio-envoy" pod "f2ddd589-df2c-4d9e-a504-dd6029e7f99b" (UID: "f2ddd589-df2c-4d9e-a504-dd6029e7f99b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:14:34.797576 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.797558 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-istio-token" pod "f2ddd589-df2c-4d9e-a504-dd6029e7f99b" (UID: "f2ddd589-df2c-4d9e-a504-dd6029e7f99b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:14:34.897305 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.897269 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-istio-data\") pod \"d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe\" (UID: \"d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe\") " Nov 01 21:14:34.897305 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.897308 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-kube-api-access-qs8bf\" (UniqueName: \"kubernetes.io/projected/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-kube-api-access-qs8bf\") pod \"d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe\" (UID: \"d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe\") " Nov 01 21:14:34.897560 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.897377 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-workload-certs\") pod \"d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe\" (UID: \"d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe\") " Nov 01 21:14:34.897560 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.897413 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-istio-envoy\") pod \"d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe\" (UID: \"d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe\") " Nov 01 21:14:34.897560 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.897436 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-credential-socket\") pod \"d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe\" (UID: \"d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe\") " Nov 01 21:14:34.897560 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.897489 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-istio-podinfo\") pod \"d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe\" (UID: \"d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe\") " Nov 01 21:14:34.897829 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.897574 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-workload-socket\") pod \"d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe\" (UID: \"d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe\") " Nov 01 21:14:34.897829 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.897631 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-istio-data" pod "d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe" (UID: "d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:14:34.897829 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.897711 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-istio-token\" (UniqueName: \"kubernetes.io/projected/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-istio-token\") pod \"d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe\" (UID: \"d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe\") " Nov 01 21:14:34.897829 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.897746 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-istiod-ca-cert\") pod \"d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe\" (UID: \"d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe\") " Nov 01 21:14:34.897829 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.897758 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-workload-certs" pod "d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe" (UID: "d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:14:34.897829 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.897775 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-credential-socket" pod "d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe" (UID: "d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:14:34.898254 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.897908 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-workload-socket" pod "d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe" (UID: "d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:14:34.898254 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.898174 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-istiod-ca-cert" pod "d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe" (UID: "d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:14:34.898412 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.898388 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:34.898483 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.898416 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:34.898483 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.898432 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:34.898483 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.898445 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9f6j\" (UniqueName: \"kubernetes.io/projected/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-kube-api-access-q9f6j\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:34.898483 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.898460 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:34.898483 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.898473 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:34.898775 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.898488 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:34.898775 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.898505 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:34.898775 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.898518 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:34.898775 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.898530 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:34.898775 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.898544 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2ddd589-df2c-4d9e-a504-dd6029e7f99b-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:34.899922 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.899897 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-istio-envoy" pod "d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe" (UID: "d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:14:34.900137 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.900119 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-istio-podinfo" pod "d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe" (UID: "d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:14:34.900464 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.900441 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-kube-api-access-qs8bf" pod "d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe" (UID: "d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe"). InnerVolumeSpecName "kube-api-access-qs8bf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:14:34.900565 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:34.900470 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-istio-token" pod "d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe" (UID: "d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:14:35.000571 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:35.000531 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:35.000571 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:35.000575 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:35.000784 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:35.000592 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:35.000784 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:35.000607 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs8bf\" (UniqueName: \"kubernetes.io/projected/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe-kube-api-access-qs8bf\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:35.372645 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:35.372572 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b63c3f9-f156-46dd-a0ee-854b5b9c5350" path="/var/lib/kubelet/pods/7b63c3f9-f156-46dd-a0ee-854b5b9c5350/volumes" Nov 01 21:14:35.546160 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:35.546129 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td" Nov 01 21:14:35.546370 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:35.546160 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td" event={"ID":"f2ddd589-df2c-4d9e-a504-dd6029e7f99b","Type":"ContainerDied","Data":"abf1703672cff47df88d572269847b9f2a5fcd4c2ef558d0e84266639412aede"} Nov 01 21:14:35.546370 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:35.546233 4484 scope.go:122] "RemoveContainer" containerID="1367aa64dd18542dca6efa2a415391c8daf86cb47e3ffaf953b8767ac5144a63" Nov 01 21:14:35.547789 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:35.547771 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh" Nov 01 21:14:35.547903 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:35.547786 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh" event={"ID":"d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe","Type":"ContainerDied","Data":"e8adba4108b07a1c74c51d463eb9a8d586d5ee19ba28432f036f8316e49b52b2"} Nov 01 21:14:35.556890 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:35.556830 4484 scope.go:122] "RemoveContainer" containerID="c4bda6656f8f23bccaf4362114fd946e89fd87aed8cc7825ea9178be76944b79" Nov 01 21:14:35.714159 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:35.714125 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td"] Nov 01 21:14:35.768628 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:35.768585 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-7b6td"] Nov 01 21:14:35.883793 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:35.883758 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh"] Nov 01 21:14:35.886748 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:35.886725 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-7091fd-openshift-default-56c9644bc-hnhqh"] Nov 01 21:14:37.372196 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:37.372165 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe" path="/var/lib/kubelet/pods/d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe/volumes" Nov 01 21:14:37.372680 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:37.372642 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2ddd589-df2c-4d9e-a504-dd6029e7f99b" path="/var/lib/kubelet/pods/f2ddd589-df2c-4d9e-a504-dd6029e7f99b/volumes" Nov 01 21:14:39.244344 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:39.244305 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:40.661813 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:40.661780 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:40.832553 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:40.832518 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:41.008075 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:41.008041 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:41.763733 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:41.763700 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-670df9-fplpk"] Nov 01 21:14:41.764317 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:41.764303 4484 cpu_manager.go:414] "containerMap: removing container" podUID="f2ddd589-df2c-4d9e-a504-dd6029e7f99b" containerName="istio-proxy" Nov 01 21:14:41.764366 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:41.764318 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2ddd589-df2c-4d9e-a504-dd6029e7f99b" containerName="istio-proxy" Nov 01 21:14:41.764366 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:41.764333 4484 cpu_manager.go:414] "containerMap: removing container" podUID="d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe" containerName="istio-proxy" Nov 01 21:14:41.764366 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:41.764338 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe" containerName="istio-proxy" Nov 01 21:14:41.764366 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:41.764356 4484 cpu_manager.go:414] "containerMap: removing container" podUID="24b80eae-a972-46e8-bd86-f67c5452d3b1" containerName="revoke-keys" Nov 01 21:14:41.764366 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:41.764361 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24b80eae-a972-46e8-bd86-f67c5452d3b1" containerName="revoke-keys" Nov 01 21:14:41.764537 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:41.764466 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2ddd589-df2c-4d9e-a504-dd6029e7f99b" containerName="istio-proxy" Nov 01 21:14:41.764537 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:41.764474 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1daf54e-deaf-48ab-ab8c-e16cdd7ef0fe" containerName="istio-proxy" Nov 01 21:14:41.764537 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:41.764484 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="24b80eae-a972-46e8-bd86-f67c5452d3b1" containerName="revoke-keys" Nov 01 21:14:41.769557 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:41.769537 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-670df9-fplpk" Nov 01 21:14:41.823442 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:41.823408 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-670df9-fplpk"] Nov 01 21:14:41.870470 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:41.870433 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:14:41.870470 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:41.870433 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:14:41.870710 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:41.870477 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:14:41.873347 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:41.873328 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:14:41.889178 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:41.889144 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtm8n\" (UniqueName: \"kubernetes.io/projected/8bb7e26a-1e45-4f1d-b40c-bfeb2efce36e-kube-api-access-rtm8n\") pod \"maas-api-revoke-keys-e2e-api-b-670df9-fplpk\" (UID: \"8bb7e26a-1e45-4f1d-b40c-bfeb2efce36e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-670df9-fplpk" Nov 01 21:14:41.889306 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:41.889286 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bb7e26a-1e45-4f1d-b40c-bfeb2efce36e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-670df9-fplpk\" (UID: \"8bb7e26a-1e45-4f1d-b40c-bfeb2efce36e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-670df9-fplpk" Nov 01 21:14:41.889432 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:41.889415 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bb7e26a-1e45-4f1d-b40c-bfeb2efce36e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-670df9-fplpk\" (UID: \"8bb7e26a-1e45-4f1d-b40c-bfeb2efce36e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-670df9-fplpk" Nov 01 21:14:41.990251 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:41.990214 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bb7e26a-1e45-4f1d-b40c-bfeb2efce36e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-670df9-fplpk\" (UID: \"8bb7e26a-1e45-4f1d-b40c-bfeb2efce36e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-670df9-fplpk" Nov 01 21:14:41.990427 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:41.990294 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bb7e26a-1e45-4f1d-b40c-bfeb2efce36e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-670df9-fplpk\" (UID: \"8bb7e26a-1e45-4f1d-b40c-bfeb2efce36e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-670df9-fplpk" Nov 01 21:14:41.990427 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:41.990365 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtm8n\" (UniqueName: \"kubernetes.io/projected/8bb7e26a-1e45-4f1d-b40c-bfeb2efce36e-kube-api-access-rtm8n\") pod \"maas-api-revoke-keys-e2e-api-b-670df9-fplpk\" (UID: \"8bb7e26a-1e45-4f1d-b40c-bfeb2efce36e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-670df9-fplpk" Nov 01 21:14:41.990975 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:41.990956 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bb7e26a-1e45-4f1d-b40c-bfeb2efce36e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-670df9-fplpk\" (UID: \"8bb7e26a-1e45-4f1d-b40c-bfeb2efce36e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-670df9-fplpk" Nov 01 21:14:41.992614 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:41.992593 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bb7e26a-1e45-4f1d-b40c-bfeb2efce36e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-670df9-fplpk\" (UID: \"8bb7e26a-1e45-4f1d-b40c-bfeb2efce36e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-670df9-fplpk" Nov 01 21:14:42.048234 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:42.048162 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtm8n\" (UniqueName: \"kubernetes.io/projected/8bb7e26a-1e45-4f1d-b40c-bfeb2efce36e-kube-api-access-rtm8n\") pod \"maas-api-revoke-keys-e2e-api-b-670df9-fplpk\" (UID: \"8bb7e26a-1e45-4f1d-b40c-bfeb2efce36e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-670df9-fplpk" Nov 01 21:14:42.083908 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:42.083867 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-670df9-fplpk" Nov 01 21:14:42.458282 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:42.458252 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-670df9-fplpk"] Nov 01 21:14:42.458865 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:14:42.458839 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bb7e26a_1e45_4f1d_b40c_bfeb2efce36e.slice/crio-cd3bd54a6bf32548e54093e1eb845084f3f9a62f1af63b2e9c8ebcb5a87904d4 WatchSource:0}: Error finding container cd3bd54a6bf32548e54093e1eb845084f3f9a62f1af63b2e9c8ebcb5a87904d4: Status 404 returned error can't find the container with id cd3bd54a6bf32548e54093e1eb845084f3f9a62f1af63b2e9c8ebcb5a87904d4 Nov 01 21:14:42.598010 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:42.597962 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-670df9-fplpk" event={"ID":"8bb7e26a-1e45-4f1d-b40c-bfeb2efce36e","Type":"ContainerStarted","Data":"cd3bd54a6bf32548e54093e1eb845084f3f9a62f1af63b2e9c8ebcb5a87904d4"} Nov 01 21:14:43.134074 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:43.134040 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-dce296-85d75dbb6f-rq556"] Nov 01 21:14:43.246336 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:43.246302 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dce296-65f5556975-m2d7v"] Nov 01 21:14:43.522203 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:43.522168 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:43.608103 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:43.608063 4484 generic.go:356] "Generic (PLEG): container finished" podID="8bb7e26a-1e45-4f1d-b40c-bfeb2efce36e" containerID="79b38e0622b00db51a870e62e46420f69d75b6e84d94e18dc45c03037c8c8d43" exitCode=0 Nov 01 21:14:43.608287 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:43.608147 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-670df9-fplpk" event={"ID":"8bb7e26a-1e45-4f1d-b40c-bfeb2efce36e","Type":"ContainerDied","Data":"79b38e0622b00db51a870e62e46420f69d75b6e84d94e18dc45c03037c8c8d43"} Nov 01 21:14:43.702940 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:43.702066 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:43.980203 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:43.980121 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:44.969864 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:44.966592 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-670df9-fplpk" Nov 01 21:14:45.161970 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:45.161860 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bb7e26a-1e45-4f1d-b40c-bfeb2efce36e-kube-api-access-rtm8n\" (UniqueName: \"kubernetes.io/projected/8bb7e26a-1e45-4f1d-b40c-bfeb2efce36e-kube-api-access-rtm8n\") pod \"8bb7e26a-1e45-4f1d-b40c-bfeb2efce36e\" (UID: \"8bb7e26a-1e45-4f1d-b40c-bfeb2efce36e\") " Nov 01 21:14:45.161970 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:45.161967 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bb7e26a-1e45-4f1d-b40c-bfeb2efce36e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bb7e26a-1e45-4f1d-b40c-bfeb2efce36e-maas-api-xks-ca\") pod \"8bb7e26a-1e45-4f1d-b40c-bfeb2efce36e\" (UID: \"8bb7e26a-1e45-4f1d-b40c-bfeb2efce36e\") " Nov 01 21:14:45.162270 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:45.162176 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bb7e26a-1e45-4f1d-b40c-bfeb2efce36e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bb7e26a-1e45-4f1d-b40c-bfeb2efce36e-maas-api-openshift-ca\") pod \"8bb7e26a-1e45-4f1d-b40c-bfeb2efce36e\" (UID: \"8bb7e26a-1e45-4f1d-b40c-bfeb2efce36e\") " Nov 01 21:14:45.162706 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:45.162675 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bb7e26a-1e45-4f1d-b40c-bfeb2efce36e-maas-api-openshift-ca" pod "8bb7e26a-1e45-4f1d-b40c-bfeb2efce36e" (UID: "8bb7e26a-1e45-4f1d-b40c-bfeb2efce36e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:14:45.164122 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:45.164095 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bb7e26a-1e45-4f1d-b40c-bfeb2efce36e-maas-api-xks-ca" pod "8bb7e26a-1e45-4f1d-b40c-bfeb2efce36e" (UID: "8bb7e26a-1e45-4f1d-b40c-bfeb2efce36e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:14:45.164232 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:45.164191 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bb7e26a-1e45-4f1d-b40c-bfeb2efce36e-kube-api-access-rtm8n" pod "8bb7e26a-1e45-4f1d-b40c-bfeb2efce36e" (UID: "8bb7e26a-1e45-4f1d-b40c-bfeb2efce36e"). InnerVolumeSpecName "kube-api-access-rtm8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:14:45.264237 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:45.264198 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtm8n\" (UniqueName: \"kubernetes.io/projected/8bb7e26a-1e45-4f1d-b40c-bfeb2efce36e-kube-api-access-rtm8n\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:45.264237 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:45.264231 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bb7e26a-1e45-4f1d-b40c-bfeb2efce36e-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:45.264237 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:45.264242 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bb7e26a-1e45-4f1d-b40c-bfeb2efce36e-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:45.527498 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:45.527444 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:45.636962 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:45.636928 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-670df9-fplpk" Nov 01 21:14:45.637164 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:45.636929 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-670df9-fplpk" event={"ID":"8bb7e26a-1e45-4f1d-b40c-bfeb2efce36e","Type":"ContainerDied","Data":"cd3bd54a6bf32548e54093e1eb845084f3f9a62f1af63b2e9c8ebcb5a87904d4"} Nov 01 21:14:45.637164 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:45.637055 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd3bd54a6bf32548e54093e1eb845084f3f9a62f1af63b2e9c8ebcb5a87904d4" Nov 01 21:14:45.768678 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:14:45.768638 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:14:45.813664 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:45.813554 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:45.966650 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:45.966614 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:46.159507 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:46.159407 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-044440-cz5p8"] Nov 01 21:14:46.162596 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:46.162568 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-044440-cz5p8"] Nov 01 21:14:46.626041 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:46.625963 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:47.372272 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:47.372242 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b400199f-cff6-4ca8-8204-b551846d3443" path="/var/lib/kubelet/pods/b400199f-cff6-4ca8-8204-b551846d3443/volumes" Nov 01 21:14:47.954693 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:47.954653 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:48.135323 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:48.135281 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-dce296-85d75dbb6f-rq556" podUID="310baaa5-6b66-4af9-b08b-fe96fb2343bc" containerName="payload-processing" containerID="cri-o://694caacbd8b4a69773fc61e773f0b0ba4c3e268d45f78f66a0d6fa45558b50c7" gracePeriod=25 Nov 01 21:14:48.247038 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:48.246979 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dce296-65f5556975-m2d7v" podUID="67b581c3-6009-4d4d-ba06-28df3b32142d" containerName="payload-pre-processing" containerID="cri-o://37e444356439ca69ee7f3503bfbb2a9b769e319e877fc9078b0ee7fc9c4120d0" gracePeriod=25 Nov 01 21:14:48.407828 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:48.407800 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-dce296-85d75dbb6f-rq556" Nov 01 21:14:48.495085 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:48.495054 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/310baaa5-6b66-4af9-b08b-fe96fb2343bc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/310baaa5-6b66-4af9-b08b-fe96fb2343bc-plugins-config-volume\") pod \"310baaa5-6b66-4af9-b08b-fe96fb2343bc\" (UID: \"310baaa5-6b66-4af9-b08b-fe96fb2343bc\") " Nov 01 21:14:48.495287 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:48.495105 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/310baaa5-6b66-4af9-b08b-fe96fb2343bc-kube-api-access-jmwrc\" (UniqueName: \"kubernetes.io/projected/310baaa5-6b66-4af9-b08b-fe96fb2343bc-kube-api-access-jmwrc\") pod \"310baaa5-6b66-4af9-b08b-fe96fb2343bc\" (UID: \"310baaa5-6b66-4af9-b08b-fe96fb2343bc\") " Nov 01 21:14:48.495454 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:48.495432 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/310baaa5-6b66-4af9-b08b-fe96fb2343bc-plugins-config-volume" pod "310baaa5-6b66-4af9-b08b-fe96fb2343bc" (UID: "310baaa5-6b66-4af9-b08b-fe96fb2343bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:14:48.495616 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:48.495598 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/310baaa5-6b66-4af9-b08b-fe96fb2343bc-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:48.497336 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:48.497277 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/310baaa5-6b66-4af9-b08b-fe96fb2343bc-kube-api-access-jmwrc" pod "310baaa5-6b66-4af9-b08b-fe96fb2343bc" (UID: "310baaa5-6b66-4af9-b08b-fe96fb2343bc"). InnerVolumeSpecName "kube-api-access-jmwrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:14:48.515316 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:48.515285 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dce296-65f5556975-m2d7v" Nov 01 21:14:48.596321 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:48.596286 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67b581c3-6009-4d4d-ba06-28df3b32142d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67b581c3-6009-4d4d-ba06-28df3b32142d-plugins-config-volume\") pod \"67b581c3-6009-4d4d-ba06-28df3b32142d\" (UID: \"67b581c3-6009-4d4d-ba06-28df3b32142d\") " Nov 01 21:14:48.596512 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:48.596425 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67b581c3-6009-4d4d-ba06-28df3b32142d-kube-api-access-vtzfw\" (UniqueName: \"kubernetes.io/projected/67b581c3-6009-4d4d-ba06-28df3b32142d-kube-api-access-vtzfw\") pod \"67b581c3-6009-4d4d-ba06-28df3b32142d\" (UID: \"67b581c3-6009-4d4d-ba06-28df3b32142d\") " Nov 01 21:14:48.596677 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:48.596650 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67b581c3-6009-4d4d-ba06-28df3b32142d-plugins-config-volume" pod "67b581c3-6009-4d4d-ba06-28df3b32142d" (UID: "67b581c3-6009-4d4d-ba06-28df3b32142d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:14:48.596775 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:48.596762 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67b581c3-6009-4d4d-ba06-28df3b32142d-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:48.596827 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:48.596778 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmwrc\" (UniqueName: \"kubernetes.io/projected/310baaa5-6b66-4af9-b08b-fe96fb2343bc-kube-api-access-jmwrc\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:48.598492 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:48.598470 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67b581c3-6009-4d4d-ba06-28df3b32142d-kube-api-access-vtzfw" pod "67b581c3-6009-4d4d-ba06-28df3b32142d" (UID: "67b581c3-6009-4d4d-ba06-28df3b32142d"). InnerVolumeSpecName "kube-api-access-vtzfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:14:48.662095 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:48.662058 4484 generic.go:356] "Generic (PLEG): container finished" podID="67b581c3-6009-4d4d-ba06-28df3b32142d" containerID="37e444356439ca69ee7f3503bfbb2a9b769e319e877fc9078b0ee7fc9c4120d0" exitCode=0 Nov 01 21:14:48.662295 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:48.662124 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dce296-65f5556975-m2d7v" Nov 01 21:14:48.662295 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:48.662148 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dce296-65f5556975-m2d7v" event={"ID":"67b581c3-6009-4d4d-ba06-28df3b32142d","Type":"ContainerDied","Data":"37e444356439ca69ee7f3503bfbb2a9b769e319e877fc9078b0ee7fc9c4120d0"} Nov 01 21:14:48.662295 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:48.662211 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dce296-65f5556975-m2d7v" event={"ID":"67b581c3-6009-4d4d-ba06-28df3b32142d","Type":"ContainerDied","Data":"6ec73e695ae378551b349ca8eee20c0bd2aa4db87127f98a1f37ac4fb9cfd1a9"} Nov 01 21:14:48.662295 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:48.662243 4484 scope.go:122] "RemoveContainer" containerID="37e444356439ca69ee7f3503bfbb2a9b769e319e877fc9078b0ee7fc9c4120d0" Nov 01 21:14:48.668554 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:48.668525 4484 generic.go:356] "Generic (PLEG): container finished" podID="310baaa5-6b66-4af9-b08b-fe96fb2343bc" containerID="694caacbd8b4a69773fc61e773f0b0ba4c3e268d45f78f66a0d6fa45558b50c7" exitCode=0 Nov 01 21:14:48.668712 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:48.668579 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-dce296-85d75dbb6f-rq556" Nov 01 21:14:48.668712 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:48.668599 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-dce296-85d75dbb6f-rq556" event={"ID":"310baaa5-6b66-4af9-b08b-fe96fb2343bc","Type":"ContainerDied","Data":"694caacbd8b4a69773fc61e773f0b0ba4c3e268d45f78f66a0d6fa45558b50c7"} Nov 01 21:14:48.668712 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:48.668657 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-dce296-85d75dbb6f-rq556" event={"ID":"310baaa5-6b66-4af9-b08b-fe96fb2343bc","Type":"ContainerDied","Data":"328e939a7b85691f6a88c408c59ab35732c2160183ee66fce0e97c266b18730b"} Nov 01 21:14:48.680353 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:48.680328 4484 scope.go:122] "RemoveContainer" containerID="37e444356439ca69ee7f3503bfbb2a9b769e319e877fc9078b0ee7fc9c4120d0" Nov 01 21:14:48.680684 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:14:48.680664 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37e444356439ca69ee7f3503bfbb2a9b769e319e877fc9078b0ee7fc9c4120d0\": container with ID starting with 37e444356439ca69ee7f3503bfbb2a9b769e319e877fc9078b0ee7fc9c4120d0 not found: ID does not exist" containerID="37e444356439ca69ee7f3503bfbb2a9b769e319e877fc9078b0ee7fc9c4120d0" Nov 01 21:14:48.680778 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:48.680690 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37e444356439ca69ee7f3503bfbb2a9b769e319e877fc9078b0ee7fc9c4120d0"} err="failed to get container status \"37e444356439ca69ee7f3503bfbb2a9b769e319e877fc9078b0ee7fc9c4120d0\": rpc error: code = NotFound desc = could not find container \"37e444356439ca69ee7f3503bfbb2a9b769e319e877fc9078b0ee7fc9c4120d0\": container with ID starting with 37e444356439ca69ee7f3503bfbb2a9b769e319e877fc9078b0ee7fc9c4120d0 not found: ID does not exist" Nov 01 21:14:48.680778 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:48.680706 4484 scope.go:122] "RemoveContainer" containerID="694caacbd8b4a69773fc61e773f0b0ba4c3e268d45f78f66a0d6fa45558b50c7" Nov 01 21:14:48.692738 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:48.692711 4484 scope.go:122] "RemoveContainer" containerID="694caacbd8b4a69773fc61e773f0b0ba4c3e268d45f78f66a0d6fa45558b50c7" Nov 01 21:14:48.693107 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:14:48.693088 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"694caacbd8b4a69773fc61e773f0b0ba4c3e268d45f78f66a0d6fa45558b50c7\": container with ID starting with 694caacbd8b4a69773fc61e773f0b0ba4c3e268d45f78f66a0d6fa45558b50c7 not found: ID does not exist" containerID="694caacbd8b4a69773fc61e773f0b0ba4c3e268d45f78f66a0d6fa45558b50c7" Nov 01 21:14:48.693196 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:48.693116 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"694caacbd8b4a69773fc61e773f0b0ba4c3e268d45f78f66a0d6fa45558b50c7"} err="failed to get container status \"694caacbd8b4a69773fc61e773f0b0ba4c3e268d45f78f66a0d6fa45558b50c7\": rpc error: code = NotFound desc = could not find container \"694caacbd8b4a69773fc61e773f0b0ba4c3e268d45f78f66a0d6fa45558b50c7\": container with ID starting with 694caacbd8b4a69773fc61e773f0b0ba4c3e268d45f78f66a0d6fa45558b50c7 not found: ID does not exist" Nov 01 21:14:48.697800 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:48.697781 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtzfw\" (UniqueName: \"kubernetes.io/projected/67b581c3-6009-4d4d-ba06-28df3b32142d-kube-api-access-vtzfw\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:48.831035 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:48.830921 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-dce296-85d75dbb6f-rq556"] Nov 01 21:14:48.833836 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:48.833816 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-dce296-85d75dbb6f-rq556"] Nov 01 21:14:48.942596 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:48.942557 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dce296-65f5556975-m2d7v"] Nov 01 21:14:48.945689 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:48.945649 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dce296-65f5556975-m2d7v"] Nov 01 21:14:49.380532 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:49.380501 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="310baaa5-6b66-4af9-b08b-fe96fb2343bc" path="/var/lib/kubelet/pods/310baaa5-6b66-4af9-b08b-fe96fb2343bc/volumes" Nov 01 21:14:49.380918 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:49.380904 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67b581c3-6009-4d4d-ba06-28df3b32142d" path="/var/lib/kubelet/pods/67b581c3-6009-4d4d-ba06-28df3b32142d/volumes" Nov 01 21:14:51.054979 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:51.054944 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:52.160006 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:52.159956 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:52.322758 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:52.322722 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:52.523017 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:52.516779 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:53.229729 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:53.229685 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-24f923-stmpt"] Nov 01 21:14:53.230490 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:53.230468 4484 cpu_manager.go:414] "containerMap: removing container" podUID="67b581c3-6009-4d4d-ba06-28df3b32142d" containerName="payload-pre-processing" Nov 01 21:14:53.230622 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:53.230490 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67b581c3-6009-4d4d-ba06-28df3b32142d" containerName="payload-pre-processing" Nov 01 21:14:53.230622 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:53.230530 4484 cpu_manager.go:414] "containerMap: removing container" podUID="310baaa5-6b66-4af9-b08b-fe96fb2343bc" containerName="payload-processing" Nov 01 21:14:53.230622 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:53.230538 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="310baaa5-6b66-4af9-b08b-fe96fb2343bc" containerName="payload-processing" Nov 01 21:14:53.230622 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:53.230575 4484 cpu_manager.go:414] "containerMap: removing container" podUID="8bb7e26a-1e45-4f1d-b40c-bfeb2efce36e" containerName="revoke-keys" Nov 01 21:14:53.230622 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:53.230583 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bb7e26a-1e45-4f1d-b40c-bfeb2efce36e" containerName="revoke-keys" Nov 01 21:14:53.230903 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:53.230732 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="310baaa5-6b66-4af9-b08b-fe96fb2343bc" containerName="payload-processing" Nov 01 21:14:53.230903 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:53.230755 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bb7e26a-1e45-4f1d-b40c-bfeb2efce36e" containerName="revoke-keys" Nov 01 21:14:53.230903 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:53.230770 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="67b581c3-6009-4d4d-ba06-28df3b32142d" containerName="payload-pre-processing" Nov 01 21:14:53.235493 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:53.235474 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-24f923-stmpt" Nov 01 21:14:53.260370 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:53.260339 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jb8b\" (UniqueName: \"kubernetes.io/projected/fdfb78fd-83db-44e6-adeb-903d034934ab-kube-api-access-2jb8b\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-24f923-stmpt\" (UID: \"fdfb78fd-83db-44e6-adeb-903d034934ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-24f923-stmpt" Nov 01 21:14:53.260541 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:53.260389 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdfb78fd-83db-44e6-adeb-903d034934ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-24f923-stmpt\" (UID: \"fdfb78fd-83db-44e6-adeb-903d034934ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-24f923-stmpt" Nov 01 21:14:53.260607 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:53.260554 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdfb78fd-83db-44e6-adeb-903d034934ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-24f923-stmpt\" (UID: \"fdfb78fd-83db-44e6-adeb-903d034934ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-24f923-stmpt" Nov 01 21:14:53.289126 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:53.289096 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-24f923-stmpt"] Nov 01 21:14:53.336370 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:53.336337 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:14:53.336370 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:53.336364 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:14:53.339113 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:53.339092 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:14:53.339246 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:53.339148 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:14:53.362037 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:53.361985 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdfb78fd-83db-44e6-adeb-903d034934ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-24f923-stmpt\" (UID: \"fdfb78fd-83db-44e6-adeb-903d034934ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-24f923-stmpt" Nov 01 21:14:53.362182 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:53.362130 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jb8b\" (UniqueName: \"kubernetes.io/projected/fdfb78fd-83db-44e6-adeb-903d034934ab-kube-api-access-2jb8b\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-24f923-stmpt\" (UID: \"fdfb78fd-83db-44e6-adeb-903d034934ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-24f923-stmpt" Nov 01 21:14:53.362182 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:53.362159 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdfb78fd-83db-44e6-adeb-903d034934ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-24f923-stmpt\" (UID: \"fdfb78fd-83db-44e6-adeb-903d034934ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-24f923-stmpt" Nov 01 21:14:53.362819 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:53.362792 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdfb78fd-83db-44e6-adeb-903d034934ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-24f923-stmpt\" (UID: \"fdfb78fd-83db-44e6-adeb-903d034934ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-24f923-stmpt" Nov 01 21:14:53.364381 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:53.364360 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdfb78fd-83db-44e6-adeb-903d034934ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-24f923-stmpt\" (UID: \"fdfb78fd-83db-44e6-adeb-903d034934ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-24f923-stmpt" Nov 01 21:14:53.419418 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:53.419387 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jb8b\" (UniqueName: \"kubernetes.io/projected/fdfb78fd-83db-44e6-adeb-903d034934ab-kube-api-access-2jb8b\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-24f923-stmpt\" (UID: \"fdfb78fd-83db-44e6-adeb-903d034934ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-24f923-stmpt" Nov 01 21:14:53.549267 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:53.549234 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-24f923-stmpt" Nov 01 21:14:53.924811 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:14:53.924781 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdfb78fd_83db_44e6_adeb_903d034934ab.slice/crio-9d8dab296baa7426b81ce5ddcc82449b92019e6e10dca09f91951162e704a3b2 WatchSource:0}: Error finding container 9d8dab296baa7426b81ce5ddcc82449b92019e6e10dca09f91951162e704a3b2: Status 404 returned error can't find the container with id 9d8dab296baa7426b81ce5ddcc82449b92019e6e10dca09f91951162e704a3b2 Nov 01 21:14:53.924922 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:53.924897 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-24f923-stmpt"] Nov 01 21:14:54.598234 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:54.598197 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-670df9-6b57976fb7-rgk2d"] Nov 01 21:14:54.729609 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:54.729572 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-24f923-stmpt" event={"ID":"fdfb78fd-83db-44e6-adeb-903d034934ab","Type":"ContainerStarted","Data":"10e406f0d707285b6072f28068f51a4104fe5fc16c121f1fce29070d297b21e8"} Nov 01 21:14:54.729790 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:54.729631 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-24f923-stmpt" event={"ID":"fdfb78fd-83db-44e6-adeb-903d034934ab","Type":"ContainerStarted","Data":"9d8dab296baa7426b81ce5ddcc82449b92019e6e10dca09f91951162e704a3b2"} Nov 01 21:14:54.845319 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:54.845262 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-24f923-stmpt" podStartSLOduration=1.845246602 podStartE2EDuration="1.845246602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:14:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:14:54.844889689 +0000 UTC m=+2645.622529826" watchObservedRunningTime="2026-11-01 21:14:54.845246602 +0000 UTC m=+2645.622887148" Nov 01 21:14:54.879295 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:54.879257 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:55.047085 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:55.041976 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:55.738027 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:55.737973 4484 generic.go:356] "Generic (PLEG): container finished" podID="fdfb78fd-83db-44e6-adeb-903d034934ab" containerID="10e406f0d707285b6072f28068f51a4104fe5fc16c121f1fce29070d297b21e8" exitCode=0 Nov 01 21:14:55.738421 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:55.738044 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-24f923-stmpt" event={"ID":"fdfb78fd-83db-44e6-adeb-903d034934ab","Type":"ContainerDied","Data":"10e406f0d707285b6072f28068f51a4104fe5fc16c121f1fce29070d297b21e8"} Nov 01 21:14:56.250966 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:56.250930 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:56.900863 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:56.900837 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-24f923-stmpt" Nov 01 21:14:56.927191 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:56.927155 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdfb78fd-83db-44e6-adeb-903d034934ab-kube-api-access-2jb8b\" (UniqueName: \"kubernetes.io/projected/fdfb78fd-83db-44e6-adeb-903d034934ab-kube-api-access-2jb8b\") pod \"fdfb78fd-83db-44e6-adeb-903d034934ab\" (UID: \"fdfb78fd-83db-44e6-adeb-903d034934ab\") " Nov 01 21:14:56.927376 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:56.927229 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdfb78fd-83db-44e6-adeb-903d034934ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdfb78fd-83db-44e6-adeb-903d034934ab-maas-api-xks-ca\") pod \"fdfb78fd-83db-44e6-adeb-903d034934ab\" (UID: \"fdfb78fd-83db-44e6-adeb-903d034934ab\") " Nov 01 21:14:56.927423 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:56.927374 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdfb78fd-83db-44e6-adeb-903d034934ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdfb78fd-83db-44e6-adeb-903d034934ab-maas-api-openshift-ca\") pod \"fdfb78fd-83db-44e6-adeb-903d034934ab\" (UID: \"fdfb78fd-83db-44e6-adeb-903d034934ab\") " Nov 01 21:14:56.927696 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:56.927673 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdfb78fd-83db-44e6-adeb-903d034934ab-maas-api-openshift-ca" pod "fdfb78fd-83db-44e6-adeb-903d034934ab" (UID: "fdfb78fd-83db-44e6-adeb-903d034934ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:14:56.927901 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:56.927885 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdfb78fd-83db-44e6-adeb-903d034934ab-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:56.929245 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:56.929226 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdfb78fd-83db-44e6-adeb-903d034934ab-kube-api-access-2jb8b" pod "fdfb78fd-83db-44e6-adeb-903d034934ab" (UID: "fdfb78fd-83db-44e6-adeb-903d034934ab"). InnerVolumeSpecName "kube-api-access-2jb8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:14:56.929331 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:56.929313 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdfb78fd-83db-44e6-adeb-903d034934ab-maas-api-xks-ca" pod "fdfb78fd-83db-44e6-adeb-903d034934ab" (UID: "fdfb78fd-83db-44e6-adeb-903d034934ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:14:57.029321 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:57.029241 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdfb78fd-83db-44e6-adeb-903d034934ab-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:57.029321 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:57.029271 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jb8b\" (UniqueName: \"kubernetes.io/projected/fdfb78fd-83db-44e6-adeb-903d034934ab-kube-api-access-2jb8b\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:57.752963 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:57.752929 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-24f923-stmpt" event={"ID":"fdfb78fd-83db-44e6-adeb-903d034934ab","Type":"ContainerDied","Data":"9d8dab296baa7426b81ce5ddcc82449b92019e6e10dca09f91951162e704a3b2"} Nov 01 21:14:57.752963 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:57.752953 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-24f923-stmpt" Nov 01 21:14:57.753206 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:57.752981 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d8dab296baa7426b81ce5ddcc82449b92019e6e10dca09f91951162e704a3b2" Nov 01 21:14:59.598928 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:59.598878 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-670df9-6b57976fb7-rgk2d" podUID="f34be5c3-8cbe-441e-b86d-7fcbb910db3d" containerName="payload-pre-processing" containerID="cri-o://09094612853c5ef1a71feb5503566aeba88f113bddd212a123f7a987e67f6c4e" gracePeriod=25 Nov 01 21:14:59.771278 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:59.769160 4484 generic.go:356] "Generic (PLEG): container finished" podID="f34be5c3-8cbe-441e-b86d-7fcbb910db3d" containerID="09094612853c5ef1a71feb5503566aeba88f113bddd212a123f7a987e67f6c4e" exitCode=0 Nov 01 21:14:59.771278 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:59.769239 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-670df9-6b57976fb7-rgk2d" event={"ID":"f34be5c3-8cbe-441e-b86d-7fcbb910db3d","Type":"ContainerDied","Data":"09094612853c5ef1a71feb5503566aeba88f113bddd212a123f7a987e67f6c4e"} Nov 01 21:14:59.879709 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:59.879680 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-670df9-6b57976fb7-rgk2d" Nov 01 21:14:59.965393 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:59.965356 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f34be5c3-8cbe-441e-b86d-7fcbb910db3d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f34be5c3-8cbe-441e-b86d-7fcbb910db3d-plugins-config-volume\") pod \"f34be5c3-8cbe-441e-b86d-7fcbb910db3d\" (UID: \"f34be5c3-8cbe-441e-b86d-7fcbb910db3d\") " Nov 01 21:14:59.965608 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:59.965421 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f34be5c3-8cbe-441e-b86d-7fcbb910db3d-kube-api-access-bnjsz\" (UniqueName: \"kubernetes.io/projected/f34be5c3-8cbe-441e-b86d-7fcbb910db3d-kube-api-access-bnjsz\") pod \"f34be5c3-8cbe-441e-b86d-7fcbb910db3d\" (UID: \"f34be5c3-8cbe-441e-b86d-7fcbb910db3d\") " Nov 01 21:14:59.965764 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:59.965737 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f34be5c3-8cbe-441e-b86d-7fcbb910db3d-plugins-config-volume" pod "f34be5c3-8cbe-441e-b86d-7fcbb910db3d" (UID: "f34be5c3-8cbe-441e-b86d-7fcbb910db3d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:14:59.967628 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:14:59.967594 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f34be5c3-8cbe-441e-b86d-7fcbb910db3d-kube-api-access-bnjsz" pod "f34be5c3-8cbe-441e-b86d-7fcbb910db3d" (UID: "f34be5c3-8cbe-441e-b86d-7fcbb910db3d"). InnerVolumeSpecName "kube-api-access-bnjsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:15:00.067483 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.067446 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f34be5c3-8cbe-441e-b86d-7fcbb910db3d-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:00.067483 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.067482 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnjsz\" (UniqueName: \"kubernetes.io/projected/f34be5c3-8cbe-441e-b86d-7fcbb910db3d-kube-api-access-bnjsz\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:00.158553 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.158463 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892795-x5p6l"] Nov 01 21:15:00.159097 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.159082 4484 cpu_manager.go:414] "containerMap: removing container" podUID="fdfb78fd-83db-44e6-adeb-903d034934ab" containerName="revoke-keys" Nov 01 21:15:00.159167 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.159098 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdfb78fd-83db-44e6-adeb-903d034934ab" containerName="revoke-keys" Nov 01 21:15:00.159167 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.159109 4484 cpu_manager.go:414] "containerMap: removing container" podUID="f34be5c3-8cbe-441e-b86d-7fcbb910db3d" containerName="payload-pre-processing" Nov 01 21:15:00.159167 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.159114 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f34be5c3-8cbe-441e-b86d-7fcbb910db3d" containerName="payload-pre-processing" Nov 01 21:15:00.159280 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.159222 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="f34be5c3-8cbe-441e-b86d-7fcbb910db3d" containerName="payload-pre-processing" Nov 01 21:15:00.159280 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.159242 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdfb78fd-83db-44e6-adeb-903d034934ab" containerName="revoke-keys" Nov 01 21:15:00.162812 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.162787 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892795-x5p6l" Nov 01 21:15:00.163661 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.163638 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892795-8dd2w"] Nov 01 21:15:00.167433 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.167412 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892795-njr5t"] Nov 01 21:15:00.167563 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.167522 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892795-8dd2w" Nov 01 21:15:00.171392 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.171361 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-670df9-29892795-n9d77"] Nov 01 21:15:00.171517 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.171494 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892795-njr5t" Nov 01 21:15:00.175403 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.175385 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892795-sks9n"] Nov 01 21:15:00.175535 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.175517 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-670df9-29892795-n9d77" Nov 01 21:15:00.179556 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.179523 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892795-fqldx"] Nov 01 21:15:00.179811 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.179662 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892795-sks9n" Nov 01 21:15:00.184285 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.184264 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a41242-29892795-lg5ct"] Nov 01 21:15:00.184377 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.184362 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892795-fqldx" Nov 01 21:15:00.189051 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.189026 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892795-hjxd2"] Nov 01 21:15:00.189130 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.189070 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a41242-29892795-lg5ct" Nov 01 21:15:00.195026 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.193573 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892795-hjxd2" Nov 01 21:15:00.211796 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.211769 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892795-vj4nc"] Nov 01 21:15:00.216019 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.215986 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:15:00.216156 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.216030 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:15:00.216156 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.216071 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:15:00.217447 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.217429 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892795-vj4nc" Nov 01 21:15:00.259063 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.259028 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892795-x5p6l"] Nov 01 21:15:00.271050 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.270199 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892795-8dd2w"] Nov 01 21:15:00.271050 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.270312 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x5mp\" (UniqueName: \"kubernetes.io/projected/9da77fa2-9d96-4a3d-9a1e-2d23a7d8fb0c-kube-api-access-5x5mp\") pod \"maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892795-fqldx\" (UID: \"9da77fa2-9d96-4a3d-9a1e-2d23a7d8fb0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892795-fqldx" Nov 01 21:15:00.271050 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.270377 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4hfj\" (UniqueName: \"kubernetes.io/projected/be6565c5-c4bd-4f2c-9d50-0b5359d589e4-kube-api-access-m4hfj\") pod \"maas-api-key-cleanup-e2e-worker-w2-cd248f-29892795-njr5t\" (UID: \"be6565c5-c4bd-4f2c-9d50-0b5359d589e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892795-njr5t" Nov 01 21:15:00.271050 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.270464 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zgd9\" (UniqueName: \"kubernetes.io/projected/f695676a-20ed-4db3-a4f3-2961bc48be9a-kube-api-access-8zgd9\") pod \"maas-api-key-cleanup-e2e-worker-w1-db0157-29892795-x5p6l\" (UID: \"f695676a-20ed-4db3-a4f3-2961bc48be9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892795-x5p6l" Nov 01 21:15:00.271050 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.270495 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8gjv\" (UniqueName: \"kubernetes.io/projected/cb830eed-0dbe-4080-b6a0-2c3092adcf5b-kube-api-access-l8gjv\") pod \"maas-api-key-cleanup-e2e-worker-w3-944263-29892795-sks9n\" (UID: \"cb830eed-0dbe-4080-b6a0-2c3092adcf5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892795-sks9n" Nov 01 21:15:00.271050 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.270515 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb830eed-0dbe-4080-b6a0-2c3092adcf5b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-944263-29892795-sks9n\" (UID: \"cb830eed-0dbe-4080-b6a0-2c3092adcf5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892795-sks9n" Nov 01 21:15:00.271050 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.270612 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm4mf\" (UniqueName: \"kubernetes.io/projected/59565b69-6f84-415f-b765-223943cd28d1-kube-api-access-nm4mf\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892795-vj4nc\" (UID: \"59565b69-6f84-415f-b765-223943cd28d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892795-vj4nc" Nov 01 21:15:00.271050 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.270652 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be6565c5-c4bd-4f2c-9d50-0b5359d589e4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-cd248f-29892795-njr5t\" (UID: \"be6565c5-c4bd-4f2c-9d50-0b5359d589e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892795-njr5t" Nov 01 21:15:00.271050 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.270692 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d44ebf49-b772-485f-a982-055f9dd19efc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-670df9-29892795-n9d77\" (UID: \"d44ebf49-b772-485f-a982-055f9dd19efc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-670df9-29892795-n9d77" Nov 01 21:15:00.271050 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.270714 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9da77fa2-9d96-4a3d-9a1e-2d23a7d8fb0c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892795-fqldx\" (UID: \"9da77fa2-9d96-4a3d-9a1e-2d23a7d8fb0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892795-fqldx" Nov 01 21:15:00.271050 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.270871 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8grt8\" (UniqueName: \"kubernetes.io/projected/d44ebf49-b772-485f-a982-055f9dd19efc-kube-api-access-8grt8\") pod \"maas-api-key-cleanup-e2e-api-b-670df9-29892795-n9d77\" (UID: \"d44ebf49-b772-485f-a982-055f9dd19efc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-670df9-29892795-n9d77" Nov 01 21:15:00.271050 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.270908 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pzqv\" (UniqueName: \"kubernetes.io/projected/9464b311-fde8-4875-aafa-ee5c07166a4e-kube-api-access-6pzqv\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892795-8dd2w\" (UID: \"9464b311-fde8-4875-aafa-ee5c07166a4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892795-8dd2w" Nov 01 21:15:00.271050 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.270935 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9464b311-fde8-4875-aafa-ee5c07166a4e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892795-8dd2w\" (UID: \"9464b311-fde8-4875-aafa-ee5c07166a4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892795-8dd2w" Nov 01 21:15:00.271050 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.271038 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/425105dd-4b1f-4f03-94a1-121733d0ad62-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892795-hjxd2\" (UID: \"425105dd-4b1f-4f03-94a1-121733d0ad62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892795-hjxd2" Nov 01 21:15:00.271960 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.271090 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8jzj\" (UniqueName: \"kubernetes.io/projected/425105dd-4b1f-4f03-94a1-121733d0ad62-kube-api-access-x8jzj\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892795-hjxd2\" (UID: \"425105dd-4b1f-4f03-94a1-121733d0ad62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892795-hjxd2" Nov 01 21:15:00.271960 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.271167 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f695676a-20ed-4db3-a4f3-2961bc48be9a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-db0157-29892795-x5p6l\" (UID: \"f695676a-20ed-4db3-a4f3-2961bc48be9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892795-x5p6l" Nov 01 21:15:00.271960 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.271196 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vwwx\" (UniqueName: \"kubernetes.io/projected/cff29d06-8657-4d3f-aa13-88d10e0c6532-kube-api-access-2vwwx\") pod \"maas-api-key-cleanup-e2e-api-b-a41242-29892795-lg5ct\" (UID: \"cff29d06-8657-4d3f-aa13-88d10e0c6532\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a41242-29892795-lg5ct" Nov 01 21:15:00.271960 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.271221 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cff29d06-8657-4d3f-aa13-88d10e0c6532-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-a41242-29892795-lg5ct\" (UID: \"cff29d06-8657-4d3f-aa13-88d10e0c6532\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a41242-29892795-lg5ct" Nov 01 21:15:00.271960 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.271243 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59565b69-6f84-415f-b765-223943cd28d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892795-vj4nc\" (UID: \"59565b69-6f84-415f-b765-223943cd28d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892795-vj4nc" Nov 01 21:15:00.309034 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.308814 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892795-sks9n"] Nov 01 21:15:00.316741 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.316689 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-670df9-29892795-n9d77"] Nov 01 21:15:00.324097 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.324059 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892795-njr5t"] Nov 01 21:15:00.325285 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.325257 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892795-fqldx"] Nov 01 21:15:00.328737 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.328709 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a41242-29892795-lg5ct"] Nov 01 21:15:00.331685 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.331658 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892795-hjxd2"] Nov 01 21:15:00.372642 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.372610 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9464b311-fde8-4875-aafa-ee5c07166a4e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892795-8dd2w\" (UID: \"9464b311-fde8-4875-aafa-ee5c07166a4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892795-8dd2w" Nov 01 21:15:00.372809 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.372645 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/425105dd-4b1f-4f03-94a1-121733d0ad62-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892795-hjxd2\" (UID: \"425105dd-4b1f-4f03-94a1-121733d0ad62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892795-hjxd2" Nov 01 21:15:00.372809 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.372681 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8jzj\" (UniqueName: \"kubernetes.io/projected/425105dd-4b1f-4f03-94a1-121733d0ad62-kube-api-access-x8jzj\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892795-hjxd2\" (UID: \"425105dd-4b1f-4f03-94a1-121733d0ad62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892795-hjxd2" Nov 01 21:15:00.372809 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.372709 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f695676a-20ed-4db3-a4f3-2961bc48be9a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-db0157-29892795-x5p6l\" (UID: \"f695676a-20ed-4db3-a4f3-2961bc48be9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892795-x5p6l" Nov 01 21:15:00.372809 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.372729 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vwwx\" (UniqueName: \"kubernetes.io/projected/cff29d06-8657-4d3f-aa13-88d10e0c6532-kube-api-access-2vwwx\") pod \"maas-api-key-cleanup-e2e-api-b-a41242-29892795-lg5ct\" (UID: \"cff29d06-8657-4d3f-aa13-88d10e0c6532\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a41242-29892795-lg5ct" Nov 01 21:15:00.372809 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.372755 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cff29d06-8657-4d3f-aa13-88d10e0c6532-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-a41242-29892795-lg5ct\" (UID: \"cff29d06-8657-4d3f-aa13-88d10e0c6532\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a41242-29892795-lg5ct" Nov 01 21:15:00.373133 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.372841 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59565b69-6f84-415f-b765-223943cd28d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892795-vj4nc\" (UID: \"59565b69-6f84-415f-b765-223943cd28d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892795-vj4nc" Nov 01 21:15:00.373133 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.372902 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x5mp\" (UniqueName: \"kubernetes.io/projected/9da77fa2-9d96-4a3d-9a1e-2d23a7d8fb0c-kube-api-access-5x5mp\") pod \"maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892795-fqldx\" (UID: \"9da77fa2-9d96-4a3d-9a1e-2d23a7d8fb0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892795-fqldx" Nov 01 21:15:00.373133 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.372952 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4hfj\" (UniqueName: \"kubernetes.io/projected/be6565c5-c4bd-4f2c-9d50-0b5359d589e4-kube-api-access-m4hfj\") pod \"maas-api-key-cleanup-e2e-worker-w2-cd248f-29892795-njr5t\" (UID: \"be6565c5-c4bd-4f2c-9d50-0b5359d589e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892795-njr5t" Nov 01 21:15:00.373366 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.373342 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zgd9\" (UniqueName: \"kubernetes.io/projected/f695676a-20ed-4db3-a4f3-2961bc48be9a-kube-api-access-8zgd9\") pod \"maas-api-key-cleanup-e2e-worker-w1-db0157-29892795-x5p6l\" (UID: \"f695676a-20ed-4db3-a4f3-2961bc48be9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892795-x5p6l" Nov 01 21:15:00.373470 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.373390 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8gjv\" (UniqueName: \"kubernetes.io/projected/cb830eed-0dbe-4080-b6a0-2c3092adcf5b-kube-api-access-l8gjv\") pod \"maas-api-key-cleanup-e2e-worker-w3-944263-29892795-sks9n\" (UID: \"cb830eed-0dbe-4080-b6a0-2c3092adcf5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892795-sks9n" Nov 01 21:15:00.373470 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.373442 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb830eed-0dbe-4080-b6a0-2c3092adcf5b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-944263-29892795-sks9n\" (UID: \"cb830eed-0dbe-4080-b6a0-2c3092adcf5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892795-sks9n" Nov 01 21:15:00.373591 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.373488 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9464b311-fde8-4875-aafa-ee5c07166a4e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892795-8dd2w\" (UID: \"9464b311-fde8-4875-aafa-ee5c07166a4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892795-8dd2w" Nov 01 21:15:00.373591 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.373529 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cff29d06-8657-4d3f-aa13-88d10e0c6532-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-a41242-29892795-lg5ct\" (UID: \"cff29d06-8657-4d3f-aa13-88d10e0c6532\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a41242-29892795-lg5ct" Nov 01 21:15:00.373591 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.373540 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f695676a-20ed-4db3-a4f3-2961bc48be9a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-db0157-29892795-x5p6l\" (UID: \"f695676a-20ed-4db3-a4f3-2961bc48be9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892795-x5p6l" Nov 01 21:15:00.373591 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.373553 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm4mf\" (UniqueName: \"kubernetes.io/projected/59565b69-6f84-415f-b765-223943cd28d1-kube-api-access-nm4mf\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892795-vj4nc\" (UID: \"59565b69-6f84-415f-b765-223943cd28d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892795-vj4nc" Nov 01 21:15:00.373591 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.373576 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/425105dd-4b1f-4f03-94a1-121733d0ad62-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892795-hjxd2\" (UID: \"425105dd-4b1f-4f03-94a1-121733d0ad62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892795-hjxd2" Nov 01 21:15:00.373814 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.373611 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be6565c5-c4bd-4f2c-9d50-0b5359d589e4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-cd248f-29892795-njr5t\" (UID: \"be6565c5-c4bd-4f2c-9d50-0b5359d589e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892795-njr5t" Nov 01 21:15:00.373814 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.373694 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d44ebf49-b772-485f-a982-055f9dd19efc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-670df9-29892795-n9d77\" (UID: \"d44ebf49-b772-485f-a982-055f9dd19efc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-670df9-29892795-n9d77" Nov 01 21:15:00.373814 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.373613 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59565b69-6f84-415f-b765-223943cd28d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892795-vj4nc\" (UID: \"59565b69-6f84-415f-b765-223943cd28d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892795-vj4nc" Nov 01 21:15:00.373814 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.373724 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9da77fa2-9d96-4a3d-9a1e-2d23a7d8fb0c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892795-fqldx\" (UID: \"9da77fa2-9d96-4a3d-9a1e-2d23a7d8fb0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892795-fqldx" Nov 01 21:15:00.373814 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.373772 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8grt8\" (UniqueName: \"kubernetes.io/projected/d44ebf49-b772-485f-a982-055f9dd19efc-kube-api-access-8grt8\") pod \"maas-api-key-cleanup-e2e-api-b-670df9-29892795-n9d77\" (UID: \"d44ebf49-b772-485f-a982-055f9dd19efc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-670df9-29892795-n9d77" Nov 01 21:15:00.373814 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.373811 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pzqv\" (UniqueName: \"kubernetes.io/projected/9464b311-fde8-4875-aafa-ee5c07166a4e-kube-api-access-6pzqv\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892795-8dd2w\" (UID: \"9464b311-fde8-4875-aafa-ee5c07166a4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892795-8dd2w" Nov 01 21:15:00.374135 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.374119 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb830eed-0dbe-4080-b6a0-2c3092adcf5b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-944263-29892795-sks9n\" (UID: \"cb830eed-0dbe-4080-b6a0-2c3092adcf5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892795-sks9n" Nov 01 21:15:00.374316 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.374291 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be6565c5-c4bd-4f2c-9d50-0b5359d589e4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-cd248f-29892795-njr5t\" (UID: \"be6565c5-c4bd-4f2c-9d50-0b5359d589e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892795-njr5t" Nov 01 21:15:00.374357 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.374296 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d44ebf49-b772-485f-a982-055f9dd19efc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-670df9-29892795-n9d77\" (UID: \"d44ebf49-b772-485f-a982-055f9dd19efc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-670df9-29892795-n9d77" Nov 01 21:15:00.374421 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.374408 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9da77fa2-9d96-4a3d-9a1e-2d23a7d8fb0c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892795-fqldx\" (UID: \"9da77fa2-9d96-4a3d-9a1e-2d23a7d8fb0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892795-fqldx" Nov 01 21:15:00.386333 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.386304 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892795-vj4nc"] Nov 01 21:15:00.433558 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.433468 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8grt8\" (UniqueName: \"kubernetes.io/projected/d44ebf49-b772-485f-a982-055f9dd19efc-kube-api-access-8grt8\") pod \"maas-api-key-cleanup-e2e-api-b-670df9-29892795-n9d77\" (UID: \"d44ebf49-b772-485f-a982-055f9dd19efc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-670df9-29892795-n9d77" Nov 01 21:15:00.433734 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.433635 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4hfj\" (UniqueName: \"kubernetes.io/projected/be6565c5-c4bd-4f2c-9d50-0b5359d589e4-kube-api-access-m4hfj\") pod \"maas-api-key-cleanup-e2e-worker-w2-cd248f-29892795-njr5t\" (UID: \"be6565c5-c4bd-4f2c-9d50-0b5359d589e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892795-njr5t" Nov 01 21:15:00.433734 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.433635 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm4mf\" (UniqueName: \"kubernetes.io/projected/59565b69-6f84-415f-b765-223943cd28d1-kube-api-access-nm4mf\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892795-vj4nc\" (UID: \"59565b69-6f84-415f-b765-223943cd28d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892795-vj4nc" Nov 01 21:15:00.433734 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.433687 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zgd9\" (UniqueName: \"kubernetes.io/projected/f695676a-20ed-4db3-a4f3-2961bc48be9a-kube-api-access-8zgd9\") pod \"maas-api-key-cleanup-e2e-worker-w1-db0157-29892795-x5p6l\" (UID: \"f695676a-20ed-4db3-a4f3-2961bc48be9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892795-x5p6l" Nov 01 21:15:00.434021 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.433976 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pzqv\" (UniqueName: \"kubernetes.io/projected/9464b311-fde8-4875-aafa-ee5c07166a4e-kube-api-access-6pzqv\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892795-8dd2w\" (UID: \"9464b311-fde8-4875-aafa-ee5c07166a4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892795-8dd2w" Nov 01 21:15:00.434290 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.434268 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x5mp\" (UniqueName: \"kubernetes.io/projected/9da77fa2-9d96-4a3d-9a1e-2d23a7d8fb0c-kube-api-access-5x5mp\") pod \"maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892795-fqldx\" (UID: \"9da77fa2-9d96-4a3d-9a1e-2d23a7d8fb0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892795-fqldx" Nov 01 21:15:00.434734 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.434711 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vwwx\" (UniqueName: \"kubernetes.io/projected/cff29d06-8657-4d3f-aa13-88d10e0c6532-kube-api-access-2vwwx\") pod \"maas-api-key-cleanup-e2e-api-b-a41242-29892795-lg5ct\" (UID: \"cff29d06-8657-4d3f-aa13-88d10e0c6532\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a41242-29892795-lg5ct" Nov 01 21:15:00.434969 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.434949 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8jzj\" (UniqueName: \"kubernetes.io/projected/425105dd-4b1f-4f03-94a1-121733d0ad62-kube-api-access-x8jzj\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892795-hjxd2\" (UID: \"425105dd-4b1f-4f03-94a1-121733d0ad62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892795-hjxd2" Nov 01 21:15:00.435666 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.435647 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8gjv\" (UniqueName: \"kubernetes.io/projected/cb830eed-0dbe-4080-b6a0-2c3092adcf5b-kube-api-access-l8gjv\") pod \"maas-api-key-cleanup-e2e-worker-w3-944263-29892795-sks9n\" (UID: \"cb830eed-0dbe-4080-b6a0-2c3092adcf5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892795-sks9n" Nov 01 21:15:00.480250 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.480219 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892795-x5p6l" Nov 01 21:15:00.487830 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.487807 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892795-8dd2w" Nov 01 21:15:00.496756 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.496734 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892795-njr5t" Nov 01 21:15:00.507193 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.507161 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-670df9-29892795-n9d77" Nov 01 21:15:00.517489 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.517465 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892795-sks9n" Nov 01 21:15:00.526093 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.526042 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892795-fqldx" Nov 01 21:15:00.536308 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.536277 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a41242-29892795-lg5ct" Nov 01 21:15:00.546835 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.546642 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892795-hjxd2" Nov 01 21:15:00.561043 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.560539 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892795-vj4nc" Nov 01 21:15:00.783182 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.783148 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-670df9-6b57976fb7-rgk2d" event={"ID":"f34be5c3-8cbe-441e-b86d-7fcbb910db3d","Type":"ContainerDied","Data":"afe9077f1bbd576b5ca024747c0c0a5696a62a1b629e53dc4728f312bc07142e"} Nov 01 21:15:00.783717 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.783690 4484 scope.go:122] "RemoveContainer" containerID="09094612853c5ef1a71feb5503566aeba88f113bddd212a123f7a987e67f6c4e" Nov 01 21:15:00.783717 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:00.783182 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-670df9-6b57976fb7-rgk2d" Nov 01 21:15:01.031025 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:01.030975 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a41242-29892795-lg5ct"] Nov 01 21:15:01.032352 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:15:01.032321 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcff29d06_8657_4d3f_aa13_88d10e0c6532.slice/crio-3e41ca261be8349a2ccbf3453b713143e590f5ed6de0a65dd44f929c0c4bf6d7 WatchSource:0}: Error finding container 3e41ca261be8349a2ccbf3453b713143e590f5ed6de0a65dd44f929c0c4bf6d7: Status 404 returned error can't find the container with id 3e41ca261be8349a2ccbf3453b713143e590f5ed6de0a65dd44f929c0c4bf6d7 Nov 01 21:15:01.085345 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:15:01.085313 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf695676a_20ed_4db3_a4f3_2961bc48be9a.slice/crio-556a3b3f52e9dc87f4dc2e52bef1e045b6b05b52fd02a0f6f02ed9fc27a9b341 WatchSource:0}: Error finding container 556a3b3f52e9dc87f4dc2e52bef1e045b6b05b52fd02a0f6f02ed9fc27a9b341: Status 404 returned error can't find the container with id 556a3b3f52e9dc87f4dc2e52bef1e045b6b05b52fd02a0f6f02ed9fc27a9b341 Nov 01 21:15:01.085968 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:15:01.085944 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59565b69_6f84_415f_b765_223943cd28d1.slice/crio-f77b3e4df133ad433c4c2f2f3e4021b370b08ad545b8e21e37c5fbdf62763599 WatchSource:0}: Error finding container f77b3e4df133ad433c4c2f2f3e4021b370b08ad545b8e21e37c5fbdf62763599: Status 404 returned error can't find the container with id f77b3e4df133ad433c4c2f2f3e4021b370b08ad545b8e21e37c5fbdf62763599 Nov 01 21:15:01.086865 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:15:01.086773 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod425105dd_4b1f_4f03_94a1_121733d0ad62.slice/crio-722bc4a96a12f9f5e482af2e61bbd98308e47dac56f4558b2d00a815f079ac85 WatchSource:0}: Error finding container 722bc4a96a12f9f5e482af2e61bbd98308e47dac56f4558b2d00a815f079ac85: Status 404 returned error can't find the container with id 722bc4a96a12f9f5e482af2e61bbd98308e47dac56f4558b2d00a815f079ac85 Nov 01 21:15:01.090058 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:01.089978 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892795-hjxd2"] Nov 01 21:15:01.095555 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:01.094859 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892795-x5p6l"] Nov 01 21:15:01.099085 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:01.099061 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892795-vj4nc"] Nov 01 21:15:01.117936 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:01.117910 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892795-njr5t"] Nov 01 21:15:01.118322 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:15:01.118294 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe6565c5_c4bd_4f2c_9d50_0b5359d589e4.slice/crio-f2fbc4ec74839724d6838fb36d8ccd5964d742ae6882ad862c161effae502c03 WatchSource:0}: Error finding container f2fbc4ec74839724d6838fb36d8ccd5964d742ae6882ad862c161effae502c03: Status 404 returned error can't find the container with id f2fbc4ec74839724d6838fb36d8ccd5964d742ae6882ad862c161effae502c03 Nov 01 21:15:01.123225 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:01.123195 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892795-8dd2w"] Nov 01 21:15:01.134904 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:01.134869 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-670df9-29892795-n9d77"] Nov 01 21:15:01.144553 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:15:01.144524 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd44ebf49_b772_485f_a982_055f9dd19efc.slice/crio-43a8415276a93a2d22c94eaf7dfd46d02fb6d7200f792b02df7e49607f7b764e WatchSource:0}: Error finding container 43a8415276a93a2d22c94eaf7dfd46d02fb6d7200f792b02df7e49607f7b764e: Status 404 returned error can't find the container with id 43a8415276a93a2d22c94eaf7dfd46d02fb6d7200f792b02df7e49607f7b764e Nov 01 21:15:01.190875 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:15:01.190844 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb830eed_0dbe_4080_b6a0_2c3092adcf5b.slice/crio-ef12a02e2fac94c17db0332f97520e855592a200f3bd0013ca5387d3b71f846f WatchSource:0}: Error finding container ef12a02e2fac94c17db0332f97520e855592a200f3bd0013ca5387d3b71f846f: Status 404 returned error can't find the container with id ef12a02e2fac94c17db0332f97520e855592a200f3bd0013ca5387d3b71f846f Nov 01 21:15:01.192237 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:01.192216 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892795-sks9n"] Nov 01 21:15:01.192338 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:15:01.192233 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9da77fa2_9d96_4a3d_9a1e_2d23a7d8fb0c.slice/crio-e792ab85cb6389c3953bc9912337539d15b972299f52753a0f8ef873ff20256f WatchSource:0}: Error finding container e792ab85cb6389c3953bc9912337539d15b972299f52753a0f8ef873ff20256f: Status 404 returned error can't find the container with id e792ab85cb6389c3953bc9912337539d15b972299f52753a0f8ef873ff20256f Nov 01 21:15:01.196328 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:01.196309 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892795-fqldx"] Nov 01 21:15:01.314406 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:01.313045 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-670df9-6b57976fb7-rgk2d"] Nov 01 21:15:01.318462 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:01.318434 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-670df9-6b57976fb7-rgk2d"] Nov 01 21:15:01.372974 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:01.372938 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f34be5c3-8cbe-441e-b86d-7fcbb910db3d" path="/var/lib/kubelet/pods/f34be5c3-8cbe-441e-b86d-7fcbb910db3d/volumes" Nov 01 21:15:01.494939 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:01.494902 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:01.792931 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:01.792897 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892795-hjxd2" event={"ID":"425105dd-4b1f-4f03-94a1-121733d0ad62","Type":"ContainerStarted","Data":"722bc4a96a12f9f5e482af2e61bbd98308e47dac56f4558b2d00a815f079ac85"} Nov 01 21:15:01.795173 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:01.795142 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a41242-29892795-lg5ct" event={"ID":"cff29d06-8657-4d3f-aa13-88d10e0c6532","Type":"ContainerStarted","Data":"9949f3a4c7b5120c95088a7ced55446a90a8d140417ec575ae8b7450792a3e31"} Nov 01 21:15:01.795382 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:01.795359 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a41242-29892795-lg5ct" event={"ID":"cff29d06-8657-4d3f-aa13-88d10e0c6532","Type":"ContainerStarted","Data":"3e41ca261be8349a2ccbf3453b713143e590f5ed6de0a65dd44f929c0c4bf6d7"} Nov 01 21:15:01.796836 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:01.796813 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-670df9-29892795-n9d77" event={"ID":"d44ebf49-b772-485f-a982-055f9dd19efc","Type":"ContainerStarted","Data":"7cb2a894936db4612ffdbbf6c150edabfa12753974a8830aeb0d427fc2f0079f"} Nov 01 21:15:01.796925 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:01.796855 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-670df9-29892795-n9d77" event={"ID":"d44ebf49-b772-485f-a982-055f9dd19efc","Type":"ContainerStarted","Data":"43a8415276a93a2d22c94eaf7dfd46d02fb6d7200f792b02df7e49607f7b764e"} Nov 01 21:15:01.798122 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:01.798096 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892795-fqldx" event={"ID":"9da77fa2-9d96-4a3d-9a1e-2d23a7d8fb0c","Type":"ContainerStarted","Data":"e792ab85cb6389c3953bc9912337539d15b972299f52753a0f8ef873ff20256f"} Nov 01 21:15:01.799564 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:01.799544 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892795-x5p6l" event={"ID":"f695676a-20ed-4db3-a4f3-2961bc48be9a","Type":"ContainerStarted","Data":"ce82ac73b6491e3ff8d8683bbe316074eb4f9d1fd6ffa788ed5e166424d0bd4a"} Nov 01 21:15:01.799654 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:01.799585 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892795-x5p6l" event={"ID":"f695676a-20ed-4db3-a4f3-2961bc48be9a","Type":"ContainerStarted","Data":"556a3b3f52e9dc87f4dc2e52bef1e045b6b05b52fd02a0f6f02ed9fc27a9b341"} Nov 01 21:15:01.800933 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:01.800908 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892795-vj4nc" event={"ID":"59565b69-6f84-415f-b765-223943cd28d1","Type":"ContainerStarted","Data":"f77b3e4df133ad433c4c2f2f3e4021b370b08ad545b8e21e37c5fbdf62763599"} Nov 01 21:15:01.802227 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:01.802205 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892795-njr5t" event={"ID":"be6565c5-c4bd-4f2c-9d50-0b5359d589e4","Type":"ContainerStarted","Data":"f2fbc4ec74839724d6838fb36d8ccd5964d742ae6882ad862c161effae502c03"} Nov 01 21:15:01.803607 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:01.803584 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892795-sks9n" event={"ID":"cb830eed-0dbe-4080-b6a0-2c3092adcf5b","Type":"ContainerStarted","Data":"ef12a02e2fac94c17db0332f97520e855592a200f3bd0013ca5387d3b71f846f"} Nov 01 21:15:01.810162 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:01.810132 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892795-8dd2w" event={"ID":"9464b311-fde8-4875-aafa-ee5c07166a4e","Type":"ContainerStarted","Data":"ac54f18a07109c18ced6a82ae626d8b360a90165adba1d5579015ce6887380d1"} Nov 01 21:15:01.810401 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:01.810203 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892795-8dd2w" event={"ID":"9464b311-fde8-4875-aafa-ee5c07166a4e","Type":"ContainerStarted","Data":"ced5156f930cf354feced59cadeaa27ecd7486bfa4c2ecc40707bef49031d026"} Nov 01 21:15:01.910939 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:01.910890 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a41242-29892795-lg5ct" podStartSLOduration=1.910871911 podStartE2EDuration="1.910871911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:15:01.909249074 +0000 UTC m=+2652.686889235" watchObservedRunningTime="2026-11-01 21:15:01.910871911 +0000 UTC m=+2652.688512046" Nov 01 21:15:02.023816 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:02.023762 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892795-x5p6l" podStartSLOduration=2.023748609 podStartE2EDuration="2.023748609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:15:02.02195924 +0000 UTC m=+2652.799599376" watchObservedRunningTime="2026-11-01 21:15:02.023748609 +0000 UTC m=+2652.801388744" Nov 01 21:15:02.136463 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:02.136408 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892795-8dd2w" podStartSLOduration=2.1363953860000002 podStartE2EDuration="2.136395386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:15:02.135886775 +0000 UTC m=+2652.913526916" watchObservedRunningTime="2026-11-01 21:15:02.136395386 +0000 UTC m=+2652.914035520" Nov 01 21:15:02.181480 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:02.180393 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7b8v8"] Nov 01 21:15:02.181480 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:02.180757 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7b8v8" podUID="e23f5f45-a08b-4a20-b61e-d33119765592" containerName="istio-proxy" containerID="cri-o://a76d9d911356c21a33bd508e2d2212fbd9d35990e35e1a3bc496ef66dbef17ed" gracePeriod=30 Nov 01 21:15:02.189630 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:02.189589 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-dggk5"] Nov 01 21:15:02.189885 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:02.189836 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-dggk5" podUID="8da24464-e245-4061-a89b-9e9e9f61036d" containerName="istio-proxy" containerID="cri-o://eaa64059018ae201593fde21c20af3245ded97dabd4da9ff05a906d47dde4fb9" gracePeriod=30 Nov 01 21:15:02.195386 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:02.195353 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wwpvf"] Nov 01 21:15:02.195686 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:02.195645 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wwpvf" podUID="f411ea1f-0db8-4748-87e9-4ae94a7a12c4" containerName="istio-proxy" containerID="cri-o://af34fd9250775f247bfdcd4f9169ccc1e6bd82e6ca2cc00fcb321eb322581d1e" gracePeriod=30 Nov 01 21:15:02.200338 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:02.200298 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qswqs"] Nov 01 21:15:02.200560 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:02.200532 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qswqs" podUID="6a48915d-f25c-4aae-bd0d-2714992416d1" containerName="istio-proxy" containerID="cri-o://139285c996226cba5cc0a951c9b400fa866c0978fb3a53da6718469884696333" gracePeriod=30 Nov 01 21:15:02.258008 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:02.257928 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-670df9-29892795-n9d77" podStartSLOduration=2.257909333 podStartE2EDuration="2.257909333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:15:02.255506498 +0000 UTC m=+2653.033146632" watchObservedRunningTime="2026-11-01 21:15:02.257909333 +0000 UTC m=+2653.035549469" Nov 01 21:15:02.820819 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:02.820780 4484 generic.go:356] "Generic (PLEG): container finished" podID="425105dd-4b1f-4f03-94a1-121733d0ad62" containerID="034ea516eb0118aa8e0e1f6a3987516c01fe722a1e8fd8e4915e8c7cd693ddb4" exitCode=60 Nov 01 21:15:02.821266 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:02.820861 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892795-hjxd2" event={"ID":"425105dd-4b1f-4f03-94a1-121733d0ad62","Type":"ContainerDied","Data":"034ea516eb0118aa8e0e1f6a3987516c01fe722a1e8fd8e4915e8c7cd693ddb4"} Nov 01 21:15:02.821266 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:02.821175 4484 scope.go:122] "RemoveContainer" containerID="034ea516eb0118aa8e0e1f6a3987516c01fe722a1e8fd8e4915e8c7cd693ddb4" Nov 01 21:15:02.822533 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:02.822499 4484 generic.go:356] "Generic (PLEG): container finished" podID="cff29d06-8657-4d3f-aa13-88d10e0c6532" containerID="9949f3a4c7b5120c95088a7ced55446a90a8d140417ec575ae8b7450792a3e31" exitCode=60 Nov 01 21:15:02.822672 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:02.822529 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a41242-29892795-lg5ct" event={"ID":"cff29d06-8657-4d3f-aa13-88d10e0c6532","Type":"ContainerDied","Data":"9949f3a4c7b5120c95088a7ced55446a90a8d140417ec575ae8b7450792a3e31"} Nov 01 21:15:02.823062 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:02.823044 4484 scope.go:122] "RemoveContainer" containerID="9949f3a4c7b5120c95088a7ced55446a90a8d140417ec575ae8b7450792a3e31" Nov 01 21:15:02.824739 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:02.824675 4484 generic.go:356] "Generic (PLEG): container finished" podID="d44ebf49-b772-485f-a982-055f9dd19efc" containerID="7cb2a894936db4612ffdbbf6c150edabfa12753974a8830aeb0d427fc2f0079f" exitCode=60 Nov 01 21:15:02.824807 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:02.824757 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-670df9-29892795-n9d77" event={"ID":"d44ebf49-b772-485f-a982-055f9dd19efc","Type":"ContainerDied","Data":"7cb2a894936db4612ffdbbf6c150edabfa12753974a8830aeb0d427fc2f0079f"} Nov 01 21:15:02.824959 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:02.824944 4484 scope.go:122] "RemoveContainer" containerID="7cb2a894936db4612ffdbbf6c150edabfa12753974a8830aeb0d427fc2f0079f" Nov 01 21:15:02.826502 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:02.826478 4484 generic.go:356] "Generic (PLEG): container finished" podID="9da77fa2-9d96-4a3d-9a1e-2d23a7d8fb0c" containerID="7f6a1f104f5fd6980a7dd0f055787d5ae0a07e495d8ad7f06e518d012ad32177" exitCode=60 Nov 01 21:15:02.826623 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:02.826594 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892795-fqldx" event={"ID":"9da77fa2-9d96-4a3d-9a1e-2d23a7d8fb0c","Type":"ContainerDied","Data":"7f6a1f104f5fd6980a7dd0f055787d5ae0a07e495d8ad7f06e518d012ad32177"} Nov 01 21:15:02.826730 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:02.826715 4484 scope.go:122] "RemoveContainer" containerID="7f6a1f104f5fd6980a7dd0f055787d5ae0a07e495d8ad7f06e518d012ad32177" Nov 01 21:15:02.828080 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:02.828059 4484 generic.go:356] "Generic (PLEG): container finished" podID="f695676a-20ed-4db3-a4f3-2961bc48be9a" containerID="ce82ac73b6491e3ff8d8683bbe316074eb4f9d1fd6ffa788ed5e166424d0bd4a" exitCode=60 Nov 01 21:15:02.828181 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:02.828079 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892795-x5p6l" event={"ID":"f695676a-20ed-4db3-a4f3-2961bc48be9a","Type":"ContainerDied","Data":"ce82ac73b6491e3ff8d8683bbe316074eb4f9d1fd6ffa788ed5e166424d0bd4a"} Nov 01 21:15:02.828318 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:02.828304 4484 scope.go:122] "RemoveContainer" containerID="ce82ac73b6491e3ff8d8683bbe316074eb4f9d1fd6ffa788ed5e166424d0bd4a" Nov 01 21:15:02.829723 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:02.829707 4484 generic.go:356] "Generic (PLEG): container finished" podID="59565b69-6f84-415f-b765-223943cd28d1" containerID="cb31dc556e174643e94c668900ea24782aa693e60a2ab7142b29ea4a6444db71" exitCode=60 Nov 01 21:15:02.829824 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:02.829781 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892795-vj4nc" event={"ID":"59565b69-6f84-415f-b765-223943cd28d1","Type":"ContainerDied","Data":"cb31dc556e174643e94c668900ea24782aa693e60a2ab7142b29ea4a6444db71"} Nov 01 21:15:02.830063 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:02.830045 4484 scope.go:122] "RemoveContainer" containerID="cb31dc556e174643e94c668900ea24782aa693e60a2ab7142b29ea4a6444db71" Nov 01 21:15:02.831395 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:02.831374 4484 generic.go:356] "Generic (PLEG): container finished" podID="be6565c5-c4bd-4f2c-9d50-0b5359d589e4" containerID="9e908072db6f5ad5f09b0c45177ac6f82c44486526d155f22e5144f1810dd3c1" exitCode=60 Nov 01 21:15:02.831474 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:02.831445 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892795-njr5t" event={"ID":"be6565c5-c4bd-4f2c-9d50-0b5359d589e4","Type":"ContainerDied","Data":"9e908072db6f5ad5f09b0c45177ac6f82c44486526d155f22e5144f1810dd3c1"} Nov 01 21:15:02.831653 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:02.831637 4484 scope.go:122] "RemoveContainer" containerID="9e908072db6f5ad5f09b0c45177ac6f82c44486526d155f22e5144f1810dd3c1" Nov 01 21:15:02.833497 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:02.833474 4484 generic.go:356] "Generic (PLEG): container finished" podID="cb830eed-0dbe-4080-b6a0-2c3092adcf5b" containerID="af162da17fc4af105c608d80cf9066aab98a8ee9366f1ee391decff87bb4bdc0" exitCode=60 Nov 01 21:15:02.833596 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:02.833550 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892795-sks9n" event={"ID":"cb830eed-0dbe-4080-b6a0-2c3092adcf5b","Type":"ContainerDied","Data":"af162da17fc4af105c608d80cf9066aab98a8ee9366f1ee391decff87bb4bdc0"} Nov 01 21:15:02.833735 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:02.833719 4484 scope.go:122] "RemoveContainer" containerID="af162da17fc4af105c608d80cf9066aab98a8ee9366f1ee391decff87bb4bdc0" Nov 01 21:15:02.835235 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:02.835215 4484 generic.go:356] "Generic (PLEG): container finished" podID="9464b311-fde8-4875-aafa-ee5c07166a4e" containerID="ac54f18a07109c18ced6a82ae626d8b360a90165adba1d5579015ce6887380d1" exitCode=60 Nov 01 21:15:02.835315 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:02.835294 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892795-8dd2w" event={"ID":"9464b311-fde8-4875-aafa-ee5c07166a4e","Type":"ContainerDied","Data":"ac54f18a07109c18ced6a82ae626d8b360a90165adba1d5579015ce6887380d1"} Nov 01 21:15:02.835434 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:02.835420 4484 scope.go:122] "RemoveContainer" containerID="ac54f18a07109c18ced6a82ae626d8b360a90165adba1d5579015ce6887380d1" Nov 01 21:15:03.843701 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:03.843665 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892795-hjxd2" event={"ID":"425105dd-4b1f-4f03-94a1-121733d0ad62","Type":"ContainerStarted","Data":"a6f9319782ab2fddc1aed1f6b2fd70e47c27cf5b12ef34e1fc9dd7d6d210c801"} Nov 01 21:15:03.845700 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:03.845676 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a41242-29892795-lg5ct" event={"ID":"cff29d06-8657-4d3f-aa13-88d10e0c6532","Type":"ContainerStarted","Data":"e117337ce07be5e0e2b9478d243726b10222b54104c19d4fb908316319f7dcb3"} Nov 01 21:15:03.847769 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:03.847739 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-670df9-29892795-n9d77" event={"ID":"d44ebf49-b772-485f-a982-055f9dd19efc","Type":"ContainerStarted","Data":"112bd17004f86a22cb6b1f58048e11867d2b4fc2f5fa2f593a62c03c42f51fb1"} Nov 01 21:15:03.849603 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:03.849576 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892795-fqldx" event={"ID":"9da77fa2-9d96-4a3d-9a1e-2d23a7d8fb0c","Type":"ContainerStarted","Data":"a587a4d80aeb0ec663f463da45ae60c7c6d41df84ad66d0d736b1772379d90fb"} Nov 01 21:15:03.851340 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:03.851318 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892795-x5p6l" event={"ID":"f695676a-20ed-4db3-a4f3-2961bc48be9a","Type":"ContainerStarted","Data":"b957c0404c2b523dcfe8093de22ceab13da92a44f36b0cf6e483c24add765576"} Nov 01 21:15:03.853370 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:03.853341 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892795-vj4nc" event={"ID":"59565b69-6f84-415f-b765-223943cd28d1","Type":"ContainerStarted","Data":"ce0fe23b48571d9f09d0ead0771db9b723e7d14eb3eec117a32f36c9b5637d2a"} Nov 01 21:15:03.855488 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:03.855465 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892795-njr5t" event={"ID":"be6565c5-c4bd-4f2c-9d50-0b5359d589e4","Type":"ContainerStarted","Data":"34954b986549f7962e7bc82975e3b96755d37bf3a5715279d7a4999bba6eeba9"} Nov 01 21:15:03.858830 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:03.858787 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892795-sks9n" event={"ID":"cb830eed-0dbe-4080-b6a0-2c3092adcf5b","Type":"ContainerStarted","Data":"adab1801fc45a9c3b7f2ed514c94a8fa4f278ac1d9c2a34327f8db8a0d278b23"} Nov 01 21:15:03.860908 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:03.860885 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892795-8dd2w" event={"ID":"9464b311-fde8-4875-aafa-ee5c07166a4e","Type":"ContainerStarted","Data":"dedc905385dfc2826cd6292fa51cf6619c01581198fbb0b9deffb9365bfb6266"} Nov 01 21:15:04.206070 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:04.205984 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892795-fqldx" podStartSLOduration=4.205966489 podStartE2EDuration="4.205966489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:15:04.202712995 +0000 UTC m=+2654.980353130" watchObservedRunningTime="2026-11-01 21:15:04.205966489 +0000 UTC m=+2654.983606624" Nov 01 21:15:04.319433 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:04.319364 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892795-vj4nc" podStartSLOduration=4.319349066 podStartE2EDuration="4.319349066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:15:04.315865861 +0000 UTC m=+2655.093505996" watchObservedRunningTime="2026-11-01 21:15:04.319349066 +0000 UTC m=+2655.096989242" Nov 01 21:15:04.548757 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:04.548693 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892795-hjxd2" podStartSLOduration=4.54867976 podStartE2EDuration="4.54867976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:15:04.545919375 +0000 UTC m=+2655.323559510" watchObservedRunningTime="2026-11-01 21:15:04.54867976 +0000 UTC m=+2655.326319895" Nov 01 21:15:04.660729 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:04.660624 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892795-njr5t" podStartSLOduration=4.660609335 podStartE2EDuration="4.660609335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:15:04.659293848 +0000 UTC m=+2655.436933983" watchObservedRunningTime="2026-11-01 21:15:04.660609335 +0000 UTC m=+2655.438249514" Nov 01 21:15:04.725552 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:04.725514 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-24f923-8474d84f5-kfvdm"] Nov 01 21:15:04.837984 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:04.837936 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-24f923-869bcbd967-w25fl"] Nov 01 21:15:04.871186 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:04.871149 4484 generic.go:356] "Generic (PLEG): container finished" podID="59565b69-6f84-415f-b765-223943cd28d1" containerID="ce0fe23b48571d9f09d0ead0771db9b723e7d14eb3eec117a32f36c9b5637d2a" exitCode=60 Nov 01 21:15:04.871625 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:04.871213 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892795-vj4nc" event={"ID":"59565b69-6f84-415f-b765-223943cd28d1","Type":"ContainerDied","Data":"ce0fe23b48571d9f09d0ead0771db9b723e7d14eb3eec117a32f36c9b5637d2a"} Nov 01 21:15:04.871625 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:04.871279 4484 scope.go:122] "RemoveContainer" containerID="cb31dc556e174643e94c668900ea24782aa693e60a2ab7142b29ea4a6444db71" Nov 01 21:15:04.871625 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:04.871587 4484 scope.go:122] "RemoveContainer" containerID="ce0fe23b48571d9f09d0ead0771db9b723e7d14eb3eec117a32f36c9b5637d2a" Nov 01 21:15:04.871946 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:15:04.871919 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892795-vj4nc_odh-ai-gateway-infra(59565b69-6f84-415f-b765-223943cd28d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892795-vj4nc" podUID="59565b69-6f84-415f-b765-223943cd28d1" Nov 01 21:15:04.873902 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:04.873883 4484 generic.go:356] "Generic (PLEG): container finished" podID="be6565c5-c4bd-4f2c-9d50-0b5359d589e4" containerID="34954b986549f7962e7bc82975e3b96755d37bf3a5715279d7a4999bba6eeba9" exitCode=60 Nov 01 21:15:04.874048 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:04.873948 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892795-njr5t" event={"ID":"be6565c5-c4bd-4f2c-9d50-0b5359d589e4","Type":"ContainerDied","Data":"34954b986549f7962e7bc82975e3b96755d37bf3a5715279d7a4999bba6eeba9"} Nov 01 21:15:04.874357 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:04.874339 4484 scope.go:122] "RemoveContainer" containerID="34954b986549f7962e7bc82975e3b96755d37bf3a5715279d7a4999bba6eeba9" Nov 01 21:15:04.874717 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:15:04.874695 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-cd248f-29892795-njr5t_odh-ai-gateway-infra(be6565c5-c4bd-4f2c-9d50-0b5359d589e4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892795-njr5t" podUID="be6565c5-c4bd-4f2c-9d50-0b5359d589e4" Nov 01 21:15:04.875961 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:04.875939 4484 generic.go:356] "Generic (PLEG): container finished" podID="cb830eed-0dbe-4080-b6a0-2c3092adcf5b" containerID="adab1801fc45a9c3b7f2ed514c94a8fa4f278ac1d9c2a34327f8db8a0d278b23" exitCode=60 Nov 01 21:15:04.876144 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:04.876024 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892795-sks9n" event={"ID":"cb830eed-0dbe-4080-b6a0-2c3092adcf5b","Type":"ContainerDied","Data":"adab1801fc45a9c3b7f2ed514c94a8fa4f278ac1d9c2a34327f8db8a0d278b23"} Nov 01 21:15:04.876292 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:04.876274 4484 scope.go:122] "RemoveContainer" containerID="adab1801fc45a9c3b7f2ed514c94a8fa4f278ac1d9c2a34327f8db8a0d278b23" Nov 01 21:15:04.876586 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:15:04.876565 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-944263-29892795-sks9n_odh-ai-gateway-infra(cb830eed-0dbe-4080-b6a0-2c3092adcf5b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892795-sks9n" podUID="cb830eed-0dbe-4080-b6a0-2c3092adcf5b" Nov 01 21:15:04.877986 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:04.877963 4484 generic.go:356] "Generic (PLEG): container finished" podID="9464b311-fde8-4875-aafa-ee5c07166a4e" containerID="dedc905385dfc2826cd6292fa51cf6619c01581198fbb0b9deffb9365bfb6266" exitCode=60 Nov 01 21:15:04.878106 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:04.878040 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892795-8dd2w" event={"ID":"9464b311-fde8-4875-aafa-ee5c07166a4e","Type":"ContainerDied","Data":"dedc905385dfc2826cd6292fa51cf6619c01581198fbb0b9deffb9365bfb6266"} Nov 01 21:15:04.878319 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:04.878304 4484 scope.go:122] "RemoveContainer" containerID="dedc905385dfc2826cd6292fa51cf6619c01581198fbb0b9deffb9365bfb6266" Nov 01 21:15:04.878561 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:15:04.878536 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892795-8dd2w_odh-ai-gateway-infra(9464b311-fde8-4875-aafa-ee5c07166a4e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892795-8dd2w" podUID="9464b311-fde8-4875-aafa-ee5c07166a4e" Nov 01 21:15:04.880136 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:04.880116 4484 generic.go:356] "Generic (PLEG): container finished" podID="425105dd-4b1f-4f03-94a1-121733d0ad62" containerID="a6f9319782ab2fddc1aed1f6b2fd70e47c27cf5b12ef34e1fc9dd7d6d210c801" exitCode=60 Nov 01 21:15:04.880248 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:04.880199 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892795-hjxd2" event={"ID":"425105dd-4b1f-4f03-94a1-121733d0ad62","Type":"ContainerDied","Data":"a6f9319782ab2fddc1aed1f6b2fd70e47c27cf5b12ef34e1fc9dd7d6d210c801"} Nov 01 21:15:04.880383 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:04.880368 4484 scope.go:122] "RemoveContainer" containerID="a6f9319782ab2fddc1aed1f6b2fd70e47c27cf5b12ef34e1fc9dd7d6d210c801" Nov 01 21:15:04.880586 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:15:04.880570 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892795-hjxd2_odh-ai-gateway-infra(425105dd-4b1f-4f03-94a1-121733d0ad62)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892795-hjxd2" podUID="425105dd-4b1f-4f03-94a1-121733d0ad62" Nov 01 21:15:04.882258 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:04.882234 4484 generic.go:356] "Generic (PLEG): container finished" podID="cff29d06-8657-4d3f-aa13-88d10e0c6532" containerID="e117337ce07be5e0e2b9478d243726b10222b54104c19d4fb908316319f7dcb3" exitCode=60 Nov 01 21:15:04.882359 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:04.882317 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a41242-29892795-lg5ct" event={"ID":"cff29d06-8657-4d3f-aa13-88d10e0c6532","Type":"ContainerDied","Data":"e117337ce07be5e0e2b9478d243726b10222b54104c19d4fb908316319f7dcb3"} Nov 01 21:15:04.882497 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:04.882481 4484 scope.go:122] "RemoveContainer" containerID="e117337ce07be5e0e2b9478d243726b10222b54104c19d4fb908316319f7dcb3" Nov 01 21:15:04.882729 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:15:04.882709 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-a41242-29892795-lg5ct_odh-ai-gateway-infra(cff29d06-8657-4d3f-aa13-88d10e0c6532)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a41242-29892795-lg5ct" podUID="cff29d06-8657-4d3f-aa13-88d10e0c6532" Nov 01 21:15:04.884147 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:04.884126 4484 generic.go:356] "Generic (PLEG): container finished" podID="d44ebf49-b772-485f-a982-055f9dd19efc" containerID="112bd17004f86a22cb6b1f58048e11867d2b4fc2f5fa2f593a62c03c42f51fb1" exitCode=60 Nov 01 21:15:04.884248 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:04.884211 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-670df9-29892795-n9d77" event={"ID":"d44ebf49-b772-485f-a982-055f9dd19efc","Type":"ContainerDied","Data":"112bd17004f86a22cb6b1f58048e11867d2b4fc2f5fa2f593a62c03c42f51fb1"} Nov 01 21:15:04.884365 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:04.884350 4484 scope.go:122] "RemoveContainer" containerID="112bd17004f86a22cb6b1f58048e11867d2b4fc2f5fa2f593a62c03c42f51fb1" Nov 01 21:15:04.884596 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:15:04.884578 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-670df9-29892795-n9d77_odh-ai-gateway-infra(d44ebf49-b772-485f-a982-055f9dd19efc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-670df9-29892795-n9d77" podUID="d44ebf49-b772-485f-a982-055f9dd19efc" Nov 01 21:15:04.886054 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:04.886036 4484 generic.go:356] "Generic (PLEG): container finished" podID="9da77fa2-9d96-4a3d-9a1e-2d23a7d8fb0c" containerID="a587a4d80aeb0ec663f463da45ae60c7c6d41df84ad66d0d736b1772379d90fb" exitCode=60 Nov 01 21:15:04.886129 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:04.886063 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892795-fqldx" event={"ID":"9da77fa2-9d96-4a3d-9a1e-2d23a7d8fb0c","Type":"ContainerDied","Data":"a587a4d80aeb0ec663f463da45ae60c7c6d41df84ad66d0d736b1772379d90fb"} Nov 01 21:15:04.886337 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:04.886317 4484 scope.go:122] "RemoveContainer" containerID="a587a4d80aeb0ec663f463da45ae60c7c6d41df84ad66d0d736b1772379d90fb" Nov 01 21:15:04.886635 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:15:04.886612 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892795-fqldx_odh-ai-gateway-infra(9da77fa2-9d96-4a3d-9a1e-2d23a7d8fb0c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892795-fqldx" podUID="9da77fa2-9d96-4a3d-9a1e-2d23a7d8fb0c" Nov 01 21:15:04.886900 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:04.886887 4484 scope.go:122] "RemoveContainer" containerID="9e908072db6f5ad5f09b0c45177ac6f82c44486526d155f22e5144f1810dd3c1" Nov 01 21:15:04.888058 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:04.887966 4484 generic.go:356] "Generic (PLEG): container finished" podID="f695676a-20ed-4db3-a4f3-2961bc48be9a" containerID="b957c0404c2b523dcfe8093de22ceab13da92a44f36b0cf6e483c24add765576" exitCode=60 Nov 01 21:15:04.888525 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:04.888494 4484 scope.go:122] "RemoveContainer" containerID="b957c0404c2b523dcfe8093de22ceab13da92a44f36b0cf6e483c24add765576" Nov 01 21:15:04.888735 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:15:04.888712 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-db0157-29892795-x5p6l_odh-ai-gateway-infra(f695676a-20ed-4db3-a4f3-2961bc48be9a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892795-x5p6l" podUID="f695676a-20ed-4db3-a4f3-2961bc48be9a" Nov 01 21:15:04.889075 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:04.889052 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892795-x5p6l" event={"ID":"f695676a-20ed-4db3-a4f3-2961bc48be9a","Type":"ContainerDied","Data":"b957c0404c2b523dcfe8093de22ceab13da92a44f36b0cf6e483c24add765576"} Nov 01 21:15:04.889447 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:04.889414 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892795-sks9n" podStartSLOduration=4.889403688 podStartE2EDuration="4.889403688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:15:04.888892963 +0000 UTC m=+2655.666533099" watchObservedRunningTime="2026-11-01 21:15:04.889403688 +0000 UTC m=+2655.667043892" Nov 01 21:15:04.901085 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:04.901057 4484 scope.go:122] "RemoveContainer" containerID="af162da17fc4af105c608d80cf9066aab98a8ee9366f1ee391decff87bb4bdc0" Nov 01 21:15:04.914399 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:04.914372 4484 scope.go:122] "RemoveContainer" containerID="ac54f18a07109c18ced6a82ae626d8b360a90165adba1d5579015ce6887380d1" Nov 01 21:15:04.928194 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:04.928173 4484 scope.go:122] "RemoveContainer" containerID="034ea516eb0118aa8e0e1f6a3987516c01fe722a1e8fd8e4915e8c7cd693ddb4" Nov 01 21:15:04.946512 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:04.946489 4484 scope.go:122] "RemoveContainer" containerID="9949f3a4c7b5120c95088a7ced55446a90a8d140417ec575ae8b7450792a3e31" Nov 01 21:15:04.960358 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:04.960336 4484 scope.go:122] "RemoveContainer" containerID="7cb2a894936db4612ffdbbf6c150edabfa12753974a8830aeb0d427fc2f0079f" Nov 01 21:15:04.973722 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:04.973700 4484 scope.go:122] "RemoveContainer" containerID="7f6a1f104f5fd6980a7dd0f055787d5ae0a07e495d8ad7f06e518d012ad32177" Nov 01 21:15:04.986980 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:04.986950 4484 scope.go:122] "RemoveContainer" containerID="ce82ac73b6491e3ff8d8683bbe316074eb4f9d1fd6ffa788ed5e166424d0bd4a" Nov 01 21:15:05.121570 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:05.121534 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:05.298041 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:05.291939 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:05.897656 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:05.897622 4484 scope.go:122] "RemoveContainer" containerID="a587a4d80aeb0ec663f463da45ae60c7c6d41df84ad66d0d736b1772379d90fb" Nov 01 21:15:05.898153 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:15:05.897971 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892795-fqldx_odh-ai-gateway-infra(9da77fa2-9d96-4a3d-9a1e-2d23a7d8fb0c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892795-fqldx" podUID="9da77fa2-9d96-4a3d-9a1e-2d23a7d8fb0c" Nov 01 21:15:05.899416 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:05.899399 4484 scope.go:122] "RemoveContainer" containerID="b957c0404c2b523dcfe8093de22ceab13da92a44f36b0cf6e483c24add765576" Nov 01 21:15:05.899702 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:15:05.899682 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-db0157-29892795-x5p6l_odh-ai-gateway-infra(f695676a-20ed-4db3-a4f3-2961bc48be9a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892795-x5p6l" podUID="f695676a-20ed-4db3-a4f3-2961bc48be9a" Nov 01 21:15:05.901082 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:05.901061 4484 scope.go:122] "RemoveContainer" containerID="ce0fe23b48571d9f09d0ead0771db9b723e7d14eb3eec117a32f36c9b5637d2a" Nov 01 21:15:05.901378 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:15:05.901356 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892795-vj4nc_odh-ai-gateway-infra(59565b69-6f84-415f-b765-223943cd28d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892795-vj4nc" podUID="59565b69-6f84-415f-b765-223943cd28d1" Nov 01 21:15:05.902592 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:05.902567 4484 scope.go:122] "RemoveContainer" containerID="34954b986549f7962e7bc82975e3b96755d37bf3a5715279d7a4999bba6eeba9" Nov 01 21:15:05.902830 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:15:05.902807 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-cd248f-29892795-njr5t_odh-ai-gateway-infra(be6565c5-c4bd-4f2c-9d50-0b5359d589e4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892795-njr5t" podUID="be6565c5-c4bd-4f2c-9d50-0b5359d589e4" Nov 01 21:15:05.904181 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:05.904158 4484 scope.go:122] "RemoveContainer" containerID="adab1801fc45a9c3b7f2ed514c94a8fa4f278ac1d9c2a34327f8db8a0d278b23" Nov 01 21:15:05.904433 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:15:05.904410 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-944263-29892795-sks9n_odh-ai-gateway-infra(cb830eed-0dbe-4080-b6a0-2c3092adcf5b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892795-sks9n" podUID="cb830eed-0dbe-4080-b6a0-2c3092adcf5b" Nov 01 21:15:05.905547 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:05.905527 4484 scope.go:122] "RemoveContainer" containerID="dedc905385dfc2826cd6292fa51cf6619c01581198fbb0b9deffb9365bfb6266" Nov 01 21:15:05.905768 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:15:05.905749 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892795-8dd2w_odh-ai-gateway-infra(9464b311-fde8-4875-aafa-ee5c07166a4e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892795-8dd2w" podUID="9464b311-fde8-4875-aafa-ee5c07166a4e" Nov 01 21:15:05.907178 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:05.907159 4484 scope.go:122] "RemoveContainer" containerID="a6f9319782ab2fddc1aed1f6b2fd70e47c27cf5b12ef34e1fc9dd7d6d210c801" Nov 01 21:15:05.907389 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:15:05.907367 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892795-hjxd2_odh-ai-gateway-infra(425105dd-4b1f-4f03-94a1-121733d0ad62)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892795-hjxd2" podUID="425105dd-4b1f-4f03-94a1-121733d0ad62" Nov 01 21:15:05.908640 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:05.908624 4484 scope.go:122] "RemoveContainer" containerID="e117337ce07be5e0e2b9478d243726b10222b54104c19d4fb908316319f7dcb3" Nov 01 21:15:05.908903 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:15:05.908832 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-a41242-29892795-lg5ct_odh-ai-gateway-infra(cff29d06-8657-4d3f-aa13-88d10e0c6532)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a41242-29892795-lg5ct" podUID="cff29d06-8657-4d3f-aa13-88d10e0c6532" Nov 01 21:15:05.910338 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:05.910323 4484 scope.go:122] "RemoveContainer" containerID="112bd17004f86a22cb6b1f58048e11867d2b4fc2f5fa2f593a62c03c42f51fb1" Nov 01 21:15:05.910536 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:15:05.910519 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-670df9-29892795-n9d77_odh-ai-gateway-infra(d44ebf49-b772-485f-a982-055f9dd19efc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-670df9-29892795-n9d77" podUID="d44ebf49-b772-485f-a982-055f9dd19efc" Nov 01 21:15:06.336460 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:06.336429 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:07.111171 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.111136 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-7b8v8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 21:15:07.111589 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.111190 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7b8v8" podUID="e23f5f45-a08b-4a20-b61e-d33119765592" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 21:15:07.522058 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.522011 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wwpvf" Nov 01 21:15:07.559966 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.559941 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-dggk5" Nov 01 21:15:07.563147 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.563119 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7b8v8" Nov 01 21:15:07.568024 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.567986 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qswqs" Nov 01 21:15:07.571492 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.571471 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-istio-data\") pod \"f411ea1f-0db8-4748-87e9-4ae94a7a12c4\" (UID: \"f411ea1f-0db8-4748-87e9-4ae94a7a12c4\") " Nov 01 21:15:07.571561 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.571541 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-istiod-ca-cert\") pod \"f411ea1f-0db8-4748-87e9-4ae94a7a12c4\" (UID: \"f411ea1f-0db8-4748-87e9-4ae94a7a12c4\") " Nov 01 21:15:07.571596 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.571584 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-kube-api-access-p2klq\" (UniqueName: \"kubernetes.io/projected/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-kube-api-access-p2klq\") pod \"f411ea1f-0db8-4748-87e9-4ae94a7a12c4\" (UID: \"f411ea1f-0db8-4748-87e9-4ae94a7a12c4\") " Nov 01 21:15:07.571694 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.571663 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-workload-socket\") pod \"f411ea1f-0db8-4748-87e9-4ae94a7a12c4\" (UID: \"f411ea1f-0db8-4748-87e9-4ae94a7a12c4\") " Nov 01 21:15:07.571822 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.571699 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-istio-podinfo\") pod \"f411ea1f-0db8-4748-87e9-4ae94a7a12c4\" (UID: \"f411ea1f-0db8-4748-87e9-4ae94a7a12c4\") " Nov 01 21:15:07.571822 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.571751 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-istio-data" pod "f411ea1f-0db8-4748-87e9-4ae94a7a12c4" (UID: "f411ea1f-0db8-4748-87e9-4ae94a7a12c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:15:07.571822 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.571778 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-istio-envoy\") pod \"f411ea1f-0db8-4748-87e9-4ae94a7a12c4\" (UID: \"f411ea1f-0db8-4748-87e9-4ae94a7a12c4\") " Nov 01 21:15:07.571822 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.571806 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-credential-socket\") pod \"f411ea1f-0db8-4748-87e9-4ae94a7a12c4\" (UID: \"f411ea1f-0db8-4748-87e9-4ae94a7a12c4\") " Nov 01 21:15:07.572154 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.571884 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-istio-token\" (UniqueName: \"kubernetes.io/projected/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-istio-token\") pod \"f411ea1f-0db8-4748-87e9-4ae94a7a12c4\" (UID: \"f411ea1f-0db8-4748-87e9-4ae94a7a12c4\") " Nov 01 21:15:07.572154 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.571950 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-workload-socket" pod "f411ea1f-0db8-4748-87e9-4ae94a7a12c4" (UID: "f411ea1f-0db8-4748-87e9-4ae94a7a12c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:15:07.572154 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.571963 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-istiod-ca-cert" pod "f411ea1f-0db8-4748-87e9-4ae94a7a12c4" (UID: "f411ea1f-0db8-4748-87e9-4ae94a7a12c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:15:07.572154 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.571981 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-workload-certs\") pod \"f411ea1f-0db8-4748-87e9-4ae94a7a12c4\" (UID: \"f411ea1f-0db8-4748-87e9-4ae94a7a12c4\") " Nov 01 21:15:07.572379 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.572208 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-credential-socket" pod "f411ea1f-0db8-4748-87e9-4ae94a7a12c4" (UID: "f411ea1f-0db8-4748-87e9-4ae94a7a12c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:15:07.572379 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.572285 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-workload-certs" pod "f411ea1f-0db8-4748-87e9-4ae94a7a12c4" (UID: "f411ea1f-0db8-4748-87e9-4ae94a7a12c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:15:07.572642 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.572617 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:07.572730 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.572646 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:07.572730 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.572662 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:07.572730 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.572676 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:07.572730 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.572690 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:07.574012 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.573978 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-istio-podinfo" pod "f411ea1f-0db8-4748-87e9-4ae94a7a12c4" (UID: "f411ea1f-0db8-4748-87e9-4ae94a7a12c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:15:07.574114 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.574005 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-istio-envoy" pod "f411ea1f-0db8-4748-87e9-4ae94a7a12c4" (UID: "f411ea1f-0db8-4748-87e9-4ae94a7a12c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:15:07.574114 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.574065 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-kube-api-access-p2klq" pod "f411ea1f-0db8-4748-87e9-4ae94a7a12c4" (UID: "f411ea1f-0db8-4748-87e9-4ae94a7a12c4"). InnerVolumeSpecName "kube-api-access-p2klq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:15:07.574114 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.574094 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-istio-token" pod "f411ea1f-0db8-4748-87e9-4ae94a7a12c4" (UID: "f411ea1f-0db8-4748-87e9-4ae94a7a12c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:15:07.673957 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.673846 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a48915d-f25c-4aae-bd0d-2714992416d1-kube-api-access-h4m97\" (UniqueName: \"kubernetes.io/projected/6a48915d-f25c-4aae-bd0d-2714992416d1-kube-api-access-h4m97\") pod \"6a48915d-f25c-4aae-bd0d-2714992416d1\" (UID: \"6a48915d-f25c-4aae-bd0d-2714992416d1\") " Nov 01 21:15:07.673957 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.673912 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6a48915d-f25c-4aae-bd0d-2714992416d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a48915d-f25c-4aae-bd0d-2714992416d1-istio-podinfo\") pod \"6a48915d-f25c-4aae-bd0d-2714992416d1\" (UID: \"6a48915d-f25c-4aae-bd0d-2714992416d1\") " Nov 01 21:15:07.674223 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.673960 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a48915d-f25c-4aae-bd0d-2714992416d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a48915d-f25c-4aae-bd0d-2714992416d1-istio-envoy\") pod \"6a48915d-f25c-4aae-bd0d-2714992416d1\" (UID: \"6a48915d-f25c-4aae-bd0d-2714992416d1\") " Nov 01 21:15:07.674223 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.674056 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8da24464-e245-4061-a89b-9e9e9f61036d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8da24464-e245-4061-a89b-9e9e9f61036d-istio-data\") pod \"8da24464-e245-4061-a89b-9e9e9f61036d\" (UID: \"8da24464-e245-4061-a89b-9e9e9f61036d\") " Nov 01 21:15:07.674223 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.674087 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a48915d-f25c-4aae-bd0d-2714992416d1-istio-token\" (UniqueName: \"kubernetes.io/projected/6a48915d-f25c-4aae-bd0d-2714992416d1-istio-token\") pod \"6a48915d-f25c-4aae-bd0d-2714992416d1\" (UID: \"6a48915d-f25c-4aae-bd0d-2714992416d1\") " Nov 01 21:15:07.674223 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.674112 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e23f5f45-a08b-4a20-b61e-d33119765592-istio-token\" (UniqueName: \"kubernetes.io/projected/e23f5f45-a08b-4a20-b61e-d33119765592-istio-token\") pod \"e23f5f45-a08b-4a20-b61e-d33119765592\" (UID: \"e23f5f45-a08b-4a20-b61e-d33119765592\") " Nov 01 21:15:07.674223 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.674158 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a48915d-f25c-4aae-bd0d-2714992416d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a48915d-f25c-4aae-bd0d-2714992416d1-credential-socket\") pod \"6a48915d-f25c-4aae-bd0d-2714992416d1\" (UID: \"6a48915d-f25c-4aae-bd0d-2714992416d1\") " Nov 01 21:15:07.674223 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.674209 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8da24464-e245-4061-a89b-9e9e9f61036d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8da24464-e245-4061-a89b-9e9e9f61036d-istio-podinfo\") pod \"8da24464-e245-4061-a89b-9e9e9f61036d\" (UID: \"8da24464-e245-4061-a89b-9e9e9f61036d\") " Nov 01 21:15:07.674545 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.674238 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e23f5f45-a08b-4a20-b61e-d33119765592-kube-api-access-c9kqj\" (UniqueName: \"kubernetes.io/projected/e23f5f45-a08b-4a20-b61e-d33119765592-kube-api-access-c9kqj\") pod \"e23f5f45-a08b-4a20-b61e-d33119765592\" (UID: \"e23f5f45-a08b-4a20-b61e-d33119765592\") " Nov 01 21:15:07.674545 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.674269 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8da24464-e245-4061-a89b-9e9e9f61036d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8da24464-e245-4061-a89b-9e9e9f61036d-istio-envoy\") pod \"8da24464-e245-4061-a89b-9e9e9f61036d\" (UID: \"8da24464-e245-4061-a89b-9e9e9f61036d\") " Nov 01 21:15:07.674545 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.674296 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8da24464-e245-4061-a89b-9e9e9f61036d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da24464-e245-4061-a89b-9e9e9f61036d-credential-socket\") pod \"8da24464-e245-4061-a89b-9e9e9f61036d\" (UID: \"8da24464-e245-4061-a89b-9e9e9f61036d\") " Nov 01 21:15:07.674545 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.674326 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8da24464-e245-4061-a89b-9e9e9f61036d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8da24464-e245-4061-a89b-9e9e9f61036d-istiod-ca-cert\") pod \"8da24464-e245-4061-a89b-9e9e9f61036d\" (UID: \"8da24464-e245-4061-a89b-9e9e9f61036d\") " Nov 01 21:15:07.674545 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.674352 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a48915d-f25c-4aae-bd0d-2714992416d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a48915d-f25c-4aae-bd0d-2714992416d1-istio-data\") pod \"6a48915d-f25c-4aae-bd0d-2714992416d1\" (UID: \"6a48915d-f25c-4aae-bd0d-2714992416d1\") " Nov 01 21:15:07.674545 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.674432 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8da24464-e245-4061-a89b-9e9e9f61036d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da24464-e245-4061-a89b-9e9e9f61036d-workload-socket\") pod \"8da24464-e245-4061-a89b-9e9e9f61036d\" (UID: \"8da24464-e245-4061-a89b-9e9e9f61036d\") " Nov 01 21:15:07.674545 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.674478 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e23f5f45-a08b-4a20-b61e-d33119765592-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e23f5f45-a08b-4a20-b61e-d33119765592-workload-socket\") pod \"e23f5f45-a08b-4a20-b61e-d33119765592\" (UID: \"e23f5f45-a08b-4a20-b61e-d33119765592\") " Nov 01 21:15:07.674545 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.674492 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8da24464-e245-4061-a89b-9e9e9f61036d-istio-data" pod "8da24464-e245-4061-a89b-9e9e9f61036d" (UID: "8da24464-e245-4061-a89b-9e9e9f61036d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:15:07.674545 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.674510 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8da24464-e245-4061-a89b-9e9e9f61036d-istio-token\" (UniqueName: \"kubernetes.io/projected/8da24464-e245-4061-a89b-9e9e9f61036d-istio-token\") pod \"8da24464-e245-4061-a89b-9e9e9f61036d\" (UID: \"8da24464-e245-4061-a89b-9e9e9f61036d\") " Nov 01 21:15:07.675331 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.674595 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e23f5f45-a08b-4a20-b61e-d33119765592-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e23f5f45-a08b-4a20-b61e-d33119765592-workload-certs\") pod \"e23f5f45-a08b-4a20-b61e-d33119765592\" (UID: \"e23f5f45-a08b-4a20-b61e-d33119765592\") " Nov 01 21:15:07.675331 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.674640 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8da24464-e245-4061-a89b-9e9e9f61036d-kube-api-access-vh782\" (UniqueName: \"kubernetes.io/projected/8da24464-e245-4061-a89b-9e9e9f61036d-kube-api-access-vh782\") pod \"8da24464-e245-4061-a89b-9e9e9f61036d\" (UID: \"8da24464-e245-4061-a89b-9e9e9f61036d\") " Nov 01 21:15:07.675331 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.674674 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e23f5f45-a08b-4a20-b61e-d33119765592-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e23f5f45-a08b-4a20-b61e-d33119765592-istio-podinfo\") pod \"e23f5f45-a08b-4a20-b61e-d33119765592\" (UID: \"e23f5f45-a08b-4a20-b61e-d33119765592\") " Nov 01 21:15:07.675331 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.674710 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a48915d-f25c-4aae-bd0d-2714992416d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a48915d-f25c-4aae-bd0d-2714992416d1-workload-socket\") pod \"6a48915d-f25c-4aae-bd0d-2714992416d1\" (UID: \"6a48915d-f25c-4aae-bd0d-2714992416d1\") " Nov 01 21:15:07.675331 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.674748 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a48915d-f25c-4aae-bd0d-2714992416d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a48915d-f25c-4aae-bd0d-2714992416d1-istiod-ca-cert\") pod \"6a48915d-f25c-4aae-bd0d-2714992416d1\" (UID: \"6a48915d-f25c-4aae-bd0d-2714992416d1\") " Nov 01 21:15:07.675331 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.674798 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e23f5f45-a08b-4a20-b61e-d33119765592-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e23f5f45-a08b-4a20-b61e-d33119765592-credential-socket\") pod \"e23f5f45-a08b-4a20-b61e-d33119765592\" (UID: \"e23f5f45-a08b-4a20-b61e-d33119765592\") " Nov 01 21:15:07.675331 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.674858 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e23f5f45-a08b-4a20-b61e-d33119765592-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e23f5f45-a08b-4a20-b61e-d33119765592-istio-envoy\") pod \"e23f5f45-a08b-4a20-b61e-d33119765592\" (UID: \"e23f5f45-a08b-4a20-b61e-d33119765592\") " Nov 01 21:15:07.675331 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.674901 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a48915d-f25c-4aae-bd0d-2714992416d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a48915d-f25c-4aae-bd0d-2714992416d1-workload-certs\") pod \"6a48915d-f25c-4aae-bd0d-2714992416d1\" (UID: \"6a48915d-f25c-4aae-bd0d-2714992416d1\") " Nov 01 21:15:07.675331 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.674944 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8da24464-e245-4061-a89b-9e9e9f61036d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8da24464-e245-4061-a89b-9e9e9f61036d-workload-certs\") pod \"8da24464-e245-4061-a89b-9e9e9f61036d\" (UID: \"8da24464-e245-4061-a89b-9e9e9f61036d\") " Nov 01 21:15:07.675331 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.674977 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e23f5f45-a08b-4a20-b61e-d33119765592-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e23f5f45-a08b-4a20-b61e-d33119765592-istio-data\") pod \"e23f5f45-a08b-4a20-b61e-d33119765592\" (UID: \"e23f5f45-a08b-4a20-b61e-d33119765592\") " Nov 01 21:15:07.675331 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.675053 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e23f5f45-a08b-4a20-b61e-d33119765592-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e23f5f45-a08b-4a20-b61e-d33119765592-istiod-ca-cert\") pod \"e23f5f45-a08b-4a20-b61e-d33119765592\" (UID: \"e23f5f45-a08b-4a20-b61e-d33119765592\") " Nov 01 21:15:07.675912 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.675381 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a48915d-f25c-4aae-bd0d-2714992416d1-istio-data" pod "6a48915d-f25c-4aae-bd0d-2714992416d1" (UID: "6a48915d-f25c-4aae-bd0d-2714992416d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:15:07.675912 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.675773 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:07.675912 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.675800 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:07.675912 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.675817 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:07.675912 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.675832 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8da24464-e245-4061-a89b-9e9e9f61036d-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:07.675912 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.675848 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2klq\" (UniqueName: \"kubernetes.io/projected/f411ea1f-0db8-4748-87e9-4ae94a7a12c4-kube-api-access-p2klq\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:07.675912 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.675865 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a48915d-f25c-4aae-bd0d-2714992416d1-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:07.675912 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.675826 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8da24464-e245-4061-a89b-9e9e9f61036d-workload-socket" pod "8da24464-e245-4061-a89b-9e9e9f61036d" (UID: "8da24464-e245-4061-a89b-9e9e9f61036d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:15:07.675912 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.675750 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e23f5f45-a08b-4a20-b61e-d33119765592-istiod-ca-cert" pod "e23f5f45-a08b-4a20-b61e-d33119765592" (UID: "e23f5f45-a08b-4a20-b61e-d33119765592"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:15:07.676434 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.676080 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e23f5f45-a08b-4a20-b61e-d33119765592-workload-socket" pod "e23f5f45-a08b-4a20-b61e-d33119765592" (UID: "e23f5f45-a08b-4a20-b61e-d33119765592"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:15:07.676434 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.676098 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a48915d-f25c-4aae-bd0d-2714992416d1-credential-socket" pod "6a48915d-f25c-4aae-bd0d-2714992416d1" (UID: "6a48915d-f25c-4aae-bd0d-2714992416d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:15:07.676434 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.676121 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8da24464-e245-4061-a89b-9e9e9f61036d-credential-socket" pod "8da24464-e245-4061-a89b-9e9e9f61036d" (UID: "8da24464-e245-4061-a89b-9e9e9f61036d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:15:07.676434 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.676397 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8da24464-e245-4061-a89b-9e9e9f61036d-istiod-ca-cert" pod "8da24464-e245-4061-a89b-9e9e9f61036d" (UID: "8da24464-e245-4061-a89b-9e9e9f61036d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:15:07.676658 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.676604 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e23f5f45-a08b-4a20-b61e-d33119765592-credential-socket" pod "e23f5f45-a08b-4a20-b61e-d33119765592" (UID: "e23f5f45-a08b-4a20-b61e-d33119765592"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:15:07.677007 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.676967 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a48915d-f25c-4aae-bd0d-2714992416d1-istiod-ca-cert" pod "6a48915d-f25c-4aae-bd0d-2714992416d1" (UID: "6a48915d-f25c-4aae-bd0d-2714992416d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:15:07.677140 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.677115 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e23f5f45-a08b-4a20-b61e-d33119765592-workload-certs" pod "e23f5f45-a08b-4a20-b61e-d33119765592" (UID: "e23f5f45-a08b-4a20-b61e-d33119765592"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:15:07.677343 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.677320 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8da24464-e245-4061-a89b-9e9e9f61036d-workload-certs" pod "8da24464-e245-4061-a89b-9e9e9f61036d" (UID: "8da24464-e245-4061-a89b-9e9e9f61036d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:15:07.677595 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.677573 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a48915d-f25c-4aae-bd0d-2714992416d1-workload-certs" pod "6a48915d-f25c-4aae-bd0d-2714992416d1" (UID: "6a48915d-f25c-4aae-bd0d-2714992416d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:15:07.677794 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.677772 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e23f5f45-a08b-4a20-b61e-d33119765592-istio-data" pod "e23f5f45-a08b-4a20-b61e-d33119765592" (UID: "e23f5f45-a08b-4a20-b61e-d33119765592"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:15:07.678187 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.678160 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a48915d-f25c-4aae-bd0d-2714992416d1-workload-socket" pod "6a48915d-f25c-4aae-bd0d-2714992416d1" (UID: "6a48915d-f25c-4aae-bd0d-2714992416d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:15:07.678864 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.678836 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a48915d-f25c-4aae-bd0d-2714992416d1-kube-api-access-h4m97" pod "6a48915d-f25c-4aae-bd0d-2714992416d1" (UID: "6a48915d-f25c-4aae-bd0d-2714992416d1"). InnerVolumeSpecName "kube-api-access-h4m97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:15:07.679042 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.678985 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a48915d-f25c-4aae-bd0d-2714992416d1-istio-podinfo" pod "6a48915d-f25c-4aae-bd0d-2714992416d1" (UID: "6a48915d-f25c-4aae-bd0d-2714992416d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:15:07.679213 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.679192 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8da24464-e245-4061-a89b-9e9e9f61036d-istio-token" pod "8da24464-e245-4061-a89b-9e9e9f61036d" (UID: "8da24464-e245-4061-a89b-9e9e9f61036d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:15:07.679430 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.679398 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a48915d-f25c-4aae-bd0d-2714992416d1-istio-envoy" pod "6a48915d-f25c-4aae-bd0d-2714992416d1" (UID: "6a48915d-f25c-4aae-bd0d-2714992416d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:15:07.680446 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.680349 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e23f5f45-a08b-4a20-b61e-d33119765592-istio-envoy" pod "e23f5f45-a08b-4a20-b61e-d33119765592" (UID: "e23f5f45-a08b-4a20-b61e-d33119765592"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:15:07.680446 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.680414 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8da24464-e245-4061-a89b-9e9e9f61036d-kube-api-access-vh782" pod "8da24464-e245-4061-a89b-9e9e9f61036d" (UID: "8da24464-e245-4061-a89b-9e9e9f61036d"). InnerVolumeSpecName "kube-api-access-vh782". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:15:07.680734 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.680708 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e23f5f45-a08b-4a20-b61e-d33119765592-istio-token" pod "e23f5f45-a08b-4a20-b61e-d33119765592" (UID: "e23f5f45-a08b-4a20-b61e-d33119765592"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:15:07.680734 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.680724 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a48915d-f25c-4aae-bd0d-2714992416d1-istio-token" pod "6a48915d-f25c-4aae-bd0d-2714992416d1" (UID: "6a48915d-f25c-4aae-bd0d-2714992416d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:15:07.680909 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.680827 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8da24464-e245-4061-a89b-9e9e9f61036d-istio-envoy" pod "8da24464-e245-4061-a89b-9e9e9f61036d" (UID: "8da24464-e245-4061-a89b-9e9e9f61036d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:15:07.681131 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.681109 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8da24464-e245-4061-a89b-9e9e9f61036d-istio-podinfo" pod "8da24464-e245-4061-a89b-9e9e9f61036d" (UID: "8da24464-e245-4061-a89b-9e9e9f61036d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:15:07.681669 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.681652 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e23f5f45-a08b-4a20-b61e-d33119765592-kube-api-access-c9kqj" pod "e23f5f45-a08b-4a20-b61e-d33119765592" (UID: "e23f5f45-a08b-4a20-b61e-d33119765592"). InnerVolumeSpecName "kube-api-access-c9kqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:15:07.682639 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.682617 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e23f5f45-a08b-4a20-b61e-d33119765592-istio-podinfo" pod "e23f5f45-a08b-4a20-b61e-d33119765592" (UID: "e23f5f45-a08b-4a20-b61e-d33119765592"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:15:07.777257 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.777219 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh782\" (UniqueName: \"kubernetes.io/projected/8da24464-e245-4061-a89b-9e9e9f61036d-kube-api-access-vh782\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:07.777257 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.777257 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e23f5f45-a08b-4a20-b61e-d33119765592-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:07.777535 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.777271 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a48915d-f25c-4aae-bd0d-2714992416d1-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:07.777535 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.777290 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a48915d-f25c-4aae-bd0d-2714992416d1-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:07.777535 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.777302 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e23f5f45-a08b-4a20-b61e-d33119765592-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:07.777535 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.777315 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e23f5f45-a08b-4a20-b61e-d33119765592-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:07.777535 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.777326 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a48915d-f25c-4aae-bd0d-2714992416d1-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:07.777535 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.777338 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8da24464-e245-4061-a89b-9e9e9f61036d-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:07.777535 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.777350 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e23f5f45-a08b-4a20-b61e-d33119765592-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:07.777535 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.777362 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e23f5f45-a08b-4a20-b61e-d33119765592-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:07.777535 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.777375 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4m97\" (UniqueName: \"kubernetes.io/projected/6a48915d-f25c-4aae-bd0d-2714992416d1-kube-api-access-h4m97\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:07.777535 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.777387 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a48915d-f25c-4aae-bd0d-2714992416d1-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:07.777535 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.777399 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a48915d-f25c-4aae-bd0d-2714992416d1-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:07.777535 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.777410 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a48915d-f25c-4aae-bd0d-2714992416d1-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:07.777535 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.777422 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e23f5f45-a08b-4a20-b61e-d33119765592-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:07.777535 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.777434 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a48915d-f25c-4aae-bd0d-2714992416d1-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:07.777535 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.777446 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8da24464-e245-4061-a89b-9e9e9f61036d-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:07.777535 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.777460 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9kqj\" (UniqueName: \"kubernetes.io/projected/e23f5f45-a08b-4a20-b61e-d33119765592-kube-api-access-c9kqj\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:07.777535 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.777474 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8da24464-e245-4061-a89b-9e9e9f61036d-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:07.777535 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.777486 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da24464-e245-4061-a89b-9e9e9f61036d-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:07.777535 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.777500 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8da24464-e245-4061-a89b-9e9e9f61036d-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:07.777535 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.777515 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da24464-e245-4061-a89b-9e9e9f61036d-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:07.777535 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.777540 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e23f5f45-a08b-4a20-b61e-d33119765592-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:07.778303 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.777554 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8da24464-e245-4061-a89b-9e9e9f61036d-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:07.778303 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.777570 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e23f5f45-a08b-4a20-b61e-d33119765592-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:07.926284 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.926188 4484 generic.go:356] "Generic (PLEG): container finished" podID="6a48915d-f25c-4aae-bd0d-2714992416d1" containerID="139285c996226cba5cc0a951c9b400fa866c0978fb3a53da6718469884696333" exitCode=0 Nov 01 21:15:07.926284 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.926243 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qswqs" Nov 01 21:15:07.926530 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.926274 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qswqs" event={"ID":"6a48915d-f25c-4aae-bd0d-2714992416d1","Type":"ContainerDied","Data":"139285c996226cba5cc0a951c9b400fa866c0978fb3a53da6718469884696333"} Nov 01 21:15:07.926530 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.926338 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qswqs" event={"ID":"6a48915d-f25c-4aae-bd0d-2714992416d1","Type":"ContainerDied","Data":"53117ec2c9a2a668e16e46b7392bff298d85e20f205345361bc27b7a4348e0b6"} Nov 01 21:15:07.926530 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.926362 4484 scope.go:122] "RemoveContainer" containerID="139285c996226cba5cc0a951c9b400fa866c0978fb3a53da6718469884696333" Nov 01 21:15:07.927907 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.927878 4484 generic.go:356] "Generic (PLEG): container finished" podID="f411ea1f-0db8-4748-87e9-4ae94a7a12c4" containerID="af34fd9250775f247bfdcd4f9169ccc1e6bd82e6ca2cc00fcb321eb322581d1e" exitCode=0 Nov 01 21:15:07.928057 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.927934 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wwpvf" Nov 01 21:15:07.928125 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.928082 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wwpvf" event={"ID":"f411ea1f-0db8-4748-87e9-4ae94a7a12c4","Type":"ContainerDied","Data":"af34fd9250775f247bfdcd4f9169ccc1e6bd82e6ca2cc00fcb321eb322581d1e"} Nov 01 21:15:07.928183 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.928167 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wwpvf" event={"ID":"f411ea1f-0db8-4748-87e9-4ae94a7a12c4","Type":"ContainerDied","Data":"a0eaa2f0b7e72c67a1192bd7b78916ed7aaeb1c188736a9573d87d28cc92185a"} Nov 01 21:15:07.929838 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.929803 4484 generic.go:356] "Generic (PLEG): container finished" podID="e23f5f45-a08b-4a20-b61e-d33119765592" containerID="a76d9d911356c21a33bd508e2d2212fbd9d35990e35e1a3bc496ef66dbef17ed" exitCode=0 Nov 01 21:15:07.929948 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.929852 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7b8v8" Nov 01 21:15:07.929948 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.929857 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7b8v8" event={"ID":"e23f5f45-a08b-4a20-b61e-d33119765592","Type":"ContainerDied","Data":"a76d9d911356c21a33bd508e2d2212fbd9d35990e35e1a3bc496ef66dbef17ed"} Nov 01 21:15:07.929948 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.929906 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7b8v8" event={"ID":"e23f5f45-a08b-4a20-b61e-d33119765592","Type":"ContainerDied","Data":"cbac34cee4a4a8f9275f2f0398e3d862bca4ffde59ecd683e865237f0e287582"} Nov 01 21:15:07.931460 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.931435 4484 generic.go:356] "Generic (PLEG): container finished" podID="8da24464-e245-4061-a89b-9e9e9f61036d" containerID="eaa64059018ae201593fde21c20af3245ded97dabd4da9ff05a906d47dde4fb9" exitCode=0 Nov 01 21:15:07.931562 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.931459 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-dggk5" event={"ID":"8da24464-e245-4061-a89b-9e9e9f61036d","Type":"ContainerDied","Data":"eaa64059018ae201593fde21c20af3245ded97dabd4da9ff05a906d47dde4fb9"} Nov 01 21:15:07.931562 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.931485 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-dggk5" Nov 01 21:15:07.931562 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.931525 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-dggk5" event={"ID":"8da24464-e245-4061-a89b-9e9e9f61036d","Type":"ContainerDied","Data":"230d184dc80cbdfa017a89ce76352edc7cf13ab354e82386bddf7e0b75e6f96a"} Nov 01 21:15:07.940583 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.940561 4484 scope.go:122] "RemoveContainer" containerID="139285c996226cba5cc0a951c9b400fa866c0978fb3a53da6718469884696333" Nov 01 21:15:07.940940 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:15:07.940912 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"139285c996226cba5cc0a951c9b400fa866c0978fb3a53da6718469884696333\": container with ID starting with 139285c996226cba5cc0a951c9b400fa866c0978fb3a53da6718469884696333 not found: ID does not exist" containerID="139285c996226cba5cc0a951c9b400fa866c0978fb3a53da6718469884696333" Nov 01 21:15:07.941065 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.940951 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"139285c996226cba5cc0a951c9b400fa866c0978fb3a53da6718469884696333"} err="failed to get container status \"139285c996226cba5cc0a951c9b400fa866c0978fb3a53da6718469884696333\": rpc error: code = NotFound desc = could not find container \"139285c996226cba5cc0a951c9b400fa866c0978fb3a53da6718469884696333\": container with ID starting with 139285c996226cba5cc0a951c9b400fa866c0978fb3a53da6718469884696333 not found: ID does not exist" Nov 01 21:15:07.941065 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.940974 4484 scope.go:122] "RemoveContainer" containerID="af34fd9250775f247bfdcd4f9169ccc1e6bd82e6ca2cc00fcb321eb322581d1e" Nov 01 21:15:07.951928 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.951907 4484 scope.go:122] "RemoveContainer" containerID="af34fd9250775f247bfdcd4f9169ccc1e6bd82e6ca2cc00fcb321eb322581d1e" Nov 01 21:15:07.952260 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:15:07.952241 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af34fd9250775f247bfdcd4f9169ccc1e6bd82e6ca2cc00fcb321eb322581d1e\": container with ID starting with af34fd9250775f247bfdcd4f9169ccc1e6bd82e6ca2cc00fcb321eb322581d1e not found: ID does not exist" containerID="af34fd9250775f247bfdcd4f9169ccc1e6bd82e6ca2cc00fcb321eb322581d1e" Nov 01 21:15:07.952339 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.952272 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af34fd9250775f247bfdcd4f9169ccc1e6bd82e6ca2cc00fcb321eb322581d1e"} err="failed to get container status \"af34fd9250775f247bfdcd4f9169ccc1e6bd82e6ca2cc00fcb321eb322581d1e\": rpc error: code = NotFound desc = could not find container \"af34fd9250775f247bfdcd4f9169ccc1e6bd82e6ca2cc00fcb321eb322581d1e\": container with ID starting with af34fd9250775f247bfdcd4f9169ccc1e6bd82e6ca2cc00fcb321eb322581d1e not found: ID does not exist" Nov 01 21:15:07.952339 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.952296 4484 scope.go:122] "RemoveContainer" containerID="a76d9d911356c21a33bd508e2d2212fbd9d35990e35e1a3bc496ef66dbef17ed" Nov 01 21:15:07.962654 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.962628 4484 scope.go:122] "RemoveContainer" containerID="a76d9d911356c21a33bd508e2d2212fbd9d35990e35e1a3bc496ef66dbef17ed" Nov 01 21:15:07.962966 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:15:07.962947 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a76d9d911356c21a33bd508e2d2212fbd9d35990e35e1a3bc496ef66dbef17ed\": container with ID starting with a76d9d911356c21a33bd508e2d2212fbd9d35990e35e1a3bc496ef66dbef17ed not found: ID does not exist" containerID="a76d9d911356c21a33bd508e2d2212fbd9d35990e35e1a3bc496ef66dbef17ed" Nov 01 21:15:07.963053 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.962978 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a76d9d911356c21a33bd508e2d2212fbd9d35990e35e1a3bc496ef66dbef17ed"} err="failed to get container status \"a76d9d911356c21a33bd508e2d2212fbd9d35990e35e1a3bc496ef66dbef17ed\": rpc error: code = NotFound desc = could not find container \"a76d9d911356c21a33bd508e2d2212fbd9d35990e35e1a3bc496ef66dbef17ed\": container with ID starting with a76d9d911356c21a33bd508e2d2212fbd9d35990e35e1a3bc496ef66dbef17ed not found: ID does not exist" Nov 01 21:15:07.963053 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.963024 4484 scope.go:122] "RemoveContainer" containerID="eaa64059018ae201593fde21c20af3245ded97dabd4da9ff05a906d47dde4fb9" Nov 01 21:15:07.974179 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.974157 4484 scope.go:122] "RemoveContainer" containerID="eaa64059018ae201593fde21c20af3245ded97dabd4da9ff05a906d47dde4fb9" Nov 01 21:15:07.974510 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:15:07.974489 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaa64059018ae201593fde21c20af3245ded97dabd4da9ff05a906d47dde4fb9\": container with ID starting with eaa64059018ae201593fde21c20af3245ded97dabd4da9ff05a906d47dde4fb9 not found: ID does not exist" containerID="eaa64059018ae201593fde21c20af3245ded97dabd4da9ff05a906d47dde4fb9" Nov 01 21:15:07.974590 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:07.974524 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaa64059018ae201593fde21c20af3245ded97dabd4da9ff05a906d47dde4fb9"} err="failed to get container status \"eaa64059018ae201593fde21c20af3245ded97dabd4da9ff05a906d47dde4fb9\": rpc error: code = NotFound desc = could not find container \"eaa64059018ae201593fde21c20af3245ded97dabd4da9ff05a906d47dde4fb9\": container with ID starting with eaa64059018ae201593fde21c20af3245ded97dabd4da9ff05a906d47dde4fb9 not found: ID does not exist" Nov 01 21:15:08.109294 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:08.109261 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wwpvf"] Nov 01 21:15:08.112631 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:08.112601 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wwpvf"] Nov 01 21:15:08.221818 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:08.221788 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-dggk5"] Nov 01 21:15:08.225871 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:08.225834 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-dggk5"] Nov 01 21:15:08.336084 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:08.336051 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qswqs"] Nov 01 21:15:08.341226 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:08.341198 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qswqs"] Nov 01 21:15:08.452071 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:08.452041 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7b8v8"] Nov 01 21:15:08.456314 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:08.456284 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7b8v8"] Nov 01 21:15:09.373064 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:09.373033 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a48915d-f25c-4aae-bd0d-2714992416d1" path="/var/lib/kubelet/pods/6a48915d-f25c-4aae-bd0d-2714992416d1/volumes" Nov 01 21:15:09.373528 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:09.373514 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8da24464-e245-4061-a89b-9e9e9f61036d" path="/var/lib/kubelet/pods/8da24464-e245-4061-a89b-9e9e9f61036d/volumes" Nov 01 21:15:09.373940 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:09.373927 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e23f5f45-a08b-4a20-b61e-d33119765592" path="/var/lib/kubelet/pods/e23f5f45-a08b-4a20-b61e-d33119765592/volumes" Nov 01 21:15:09.374742 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:09.374727 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f411ea1f-0db8-4748-87e9-4ae94a7a12c4" path="/var/lib/kubelet/pods/f411ea1f-0db8-4748-87e9-4ae94a7a12c4/volumes" Nov 01 21:15:09.726891 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:09.726769 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-24f923-8474d84f5-kfvdm" podUID="f9a691db-7b44-40aa-8924-1f2b716da33a" containerName="payload-processing" containerID="cri-o://00b3d54099fc3fe310e330d47243bf6cac95017600cf9dc8f2c19695cd428dd4" gracePeriod=25 Nov 01 21:15:09.839329 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:09.839290 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-24f923-869bcbd967-w25fl" podUID="009125b3-fc6d-4cbe-aa45-5cf27cc39a2e" containerName="payload-pre-processing" containerID="cri-o://2d28d03c26fd16a454b42acc07c5e6a8621b22a7cc1389d8f64e20483b732afd" gracePeriod=25 Nov 01 21:15:09.953960 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:09.953902 4484 generic.go:356] "Generic (PLEG): container finished" podID="f9a691db-7b44-40aa-8924-1f2b716da33a" containerID="00b3d54099fc3fe310e330d47243bf6cac95017600cf9dc8f2c19695cd428dd4" exitCode=0 Nov 01 21:15:09.954051 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:09.953973 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-24f923-8474d84f5-kfvdm" event={"ID":"f9a691db-7b44-40aa-8924-1f2b716da33a","Type":"ContainerDied","Data":"00b3d54099fc3fe310e330d47243bf6cac95017600cf9dc8f2c19695cd428dd4"} Nov 01 21:15:09.998245 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:09.998218 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-24f923-8474d84f5-kfvdm" Nov 01 21:15:10.103559 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:10.103532 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9a691db-7b44-40aa-8924-1f2b716da33a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9a691db-7b44-40aa-8924-1f2b716da33a-plugins-config-volume\") pod \"f9a691db-7b44-40aa-8924-1f2b716da33a\" (UID: \"f9a691db-7b44-40aa-8924-1f2b716da33a\") " Nov 01 21:15:10.103715 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:10.103695 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9a691db-7b44-40aa-8924-1f2b716da33a-kube-api-access-7g779\" (UniqueName: \"kubernetes.io/projected/f9a691db-7b44-40aa-8924-1f2b716da33a-kube-api-access-7g779\") pod \"f9a691db-7b44-40aa-8924-1f2b716da33a\" (UID: \"f9a691db-7b44-40aa-8924-1f2b716da33a\") " Nov 01 21:15:10.103910 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:10.103887 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9a691db-7b44-40aa-8924-1f2b716da33a-plugins-config-volume" pod "f9a691db-7b44-40aa-8924-1f2b716da33a" (UID: "f9a691db-7b44-40aa-8924-1f2b716da33a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:15:10.104103 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:10.104077 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9a691db-7b44-40aa-8924-1f2b716da33a-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:10.105733 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:10.105709 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9a691db-7b44-40aa-8924-1f2b716da33a-kube-api-access-7g779" pod "f9a691db-7b44-40aa-8924-1f2b716da33a" (UID: "f9a691db-7b44-40aa-8924-1f2b716da33a"). InnerVolumeSpecName "kube-api-access-7g779". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:15:10.109541 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:10.109515 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-24f923-869bcbd967-w25fl" Nov 01 21:15:10.205063 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:10.205030 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/009125b3-fc6d-4cbe-aa45-5cf27cc39a2e-kube-api-access-vdl8q\" (UniqueName: \"kubernetes.io/projected/009125b3-fc6d-4cbe-aa45-5cf27cc39a2e-kube-api-access-vdl8q\") pod \"009125b3-fc6d-4cbe-aa45-5cf27cc39a2e\" (UID: \"009125b3-fc6d-4cbe-aa45-5cf27cc39a2e\") " Nov 01 21:15:10.205308 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:10.205176 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/009125b3-fc6d-4cbe-aa45-5cf27cc39a2e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/009125b3-fc6d-4cbe-aa45-5cf27cc39a2e-plugins-config-volume\") pod \"009125b3-fc6d-4cbe-aa45-5cf27cc39a2e\" (UID: \"009125b3-fc6d-4cbe-aa45-5cf27cc39a2e\") " Nov 01 21:15:10.205491 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:10.205472 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g779\" (UniqueName: \"kubernetes.io/projected/f9a691db-7b44-40aa-8924-1f2b716da33a-kube-api-access-7g779\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:10.205612 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:10.205589 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/009125b3-fc6d-4cbe-aa45-5cf27cc39a2e-plugins-config-volume" pod "009125b3-fc6d-4cbe-aa45-5cf27cc39a2e" (UID: "009125b3-fc6d-4cbe-aa45-5cf27cc39a2e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:15:10.207305 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:10.207285 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/009125b3-fc6d-4cbe-aa45-5cf27cc39a2e-kube-api-access-vdl8q" pod "009125b3-fc6d-4cbe-aa45-5cf27cc39a2e" (UID: "009125b3-fc6d-4cbe-aa45-5cf27cc39a2e"). InnerVolumeSpecName "kube-api-access-vdl8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:15:10.306289 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:10.306197 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/009125b3-fc6d-4cbe-aa45-5cf27cc39a2e-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:10.306289 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:10.306236 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdl8q\" (UniqueName: \"kubernetes.io/projected/009125b3-fc6d-4cbe-aa45-5cf27cc39a2e-kube-api-access-vdl8q\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:10.498555 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:10.498522 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:10.544877 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:10.544842 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892795-njr5t"] Nov 01 21:15:10.698949 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:10.698922 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892795-njr5t" Nov 01 21:15:10.812734 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:10.812697 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be6565c5-c4bd-4f2c-9d50-0b5359d589e4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be6565c5-c4bd-4f2c-9d50-0b5359d589e4-maas-api-service-ca\") pod \"be6565c5-c4bd-4f2c-9d50-0b5359d589e4\" (UID: \"be6565c5-c4bd-4f2c-9d50-0b5359d589e4\") " Nov 01 21:15:10.812914 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:10.812794 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be6565c5-c4bd-4f2c-9d50-0b5359d589e4-kube-api-access-m4hfj\" (UniqueName: \"kubernetes.io/projected/be6565c5-c4bd-4f2c-9d50-0b5359d589e4-kube-api-access-m4hfj\") pod \"be6565c5-c4bd-4f2c-9d50-0b5359d589e4\" (UID: \"be6565c5-c4bd-4f2c-9d50-0b5359d589e4\") " Nov 01 21:15:10.813107 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:10.813083 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be6565c5-c4bd-4f2c-9d50-0b5359d589e4-maas-api-service-ca" pod "be6565c5-c4bd-4f2c-9d50-0b5359d589e4" (UID: "be6565c5-c4bd-4f2c-9d50-0b5359d589e4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:15:10.814906 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:10.814877 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be6565c5-c4bd-4f2c-9d50-0b5359d589e4-kube-api-access-m4hfj" pod "be6565c5-c4bd-4f2c-9d50-0b5359d589e4" (UID: "be6565c5-c4bd-4f2c-9d50-0b5359d589e4"). InnerVolumeSpecName "kube-api-access-m4hfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:15:10.914646 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:10.914553 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be6565c5-c4bd-4f2c-9d50-0b5359d589e4-maas-api-service-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:10.914646 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:10.914587 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4hfj\" (UniqueName: \"kubernetes.io/projected/be6565c5-c4bd-4f2c-9d50-0b5359d589e4-kube-api-access-m4hfj\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:10.962968 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:10.962937 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-24f923-8474d84f5-kfvdm" Nov 01 21:15:10.963193 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:10.962969 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-24f923-8474d84f5-kfvdm" event={"ID":"f9a691db-7b44-40aa-8924-1f2b716da33a","Type":"ContainerDied","Data":"f6d06be7602d1f86dea743402725550b7c07375cc430f1abaf828c859de67a4e"} Nov 01 21:15:10.963193 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:10.963071 4484 scope.go:122] "RemoveContainer" containerID="00b3d54099fc3fe310e330d47243bf6cac95017600cf9dc8f2c19695cd428dd4" Nov 01 21:15:10.964456 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:10.964353 4484 generic.go:356] "Generic (PLEG): container finished" podID="009125b3-fc6d-4cbe-aa45-5cf27cc39a2e" containerID="2d28d03c26fd16a454b42acc07c5e6a8621b22a7cc1389d8f64e20483b732afd" exitCode=0 Nov 01 21:15:10.964456 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:10.964401 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-24f923-869bcbd967-w25fl" event={"ID":"009125b3-fc6d-4cbe-aa45-5cf27cc39a2e","Type":"ContainerDied","Data":"2d28d03c26fd16a454b42acc07c5e6a8621b22a7cc1389d8f64e20483b732afd"} Nov 01 21:15:10.964562 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:10.964457 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-24f923-869bcbd967-w25fl" event={"ID":"009125b3-fc6d-4cbe-aa45-5cf27cc39a2e","Type":"ContainerDied","Data":"72ef89d70a98b50e49304debe5b476e6227c669559fd81d52f2488c28ed4431c"} Nov 01 21:15:10.964562 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:10.964408 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-24f923-869bcbd967-w25fl" Nov 01 21:15:10.965938 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:10.965912 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892795-njr5t" event={"ID":"be6565c5-c4bd-4f2c-9d50-0b5359d589e4","Type":"ContainerDied","Data":"f2fbc4ec74839724d6838fb36d8ccd5964d742ae6882ad862c161effae502c03"} Nov 01 21:15:10.966088 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:10.965921 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892795-njr5t" Nov 01 21:15:10.975519 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:10.975504 4484 scope.go:122] "RemoveContainer" containerID="2d28d03c26fd16a454b42acc07c5e6a8621b22a7cc1389d8f64e20483b732afd" Nov 01 21:15:10.984989 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:10.984966 4484 scope.go:122] "RemoveContainer" containerID="2d28d03c26fd16a454b42acc07c5e6a8621b22a7cc1389d8f64e20483b732afd" Nov 01 21:15:10.985318 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:15:10.985296 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d28d03c26fd16a454b42acc07c5e6a8621b22a7cc1389d8f64e20483b732afd\": container with ID starting with 2d28d03c26fd16a454b42acc07c5e6a8621b22a7cc1389d8f64e20483b732afd not found: ID does not exist" containerID="2d28d03c26fd16a454b42acc07c5e6a8621b22a7cc1389d8f64e20483b732afd" Nov 01 21:15:10.985363 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:10.985329 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d28d03c26fd16a454b42acc07c5e6a8621b22a7cc1389d8f64e20483b732afd"} err="failed to get container status \"2d28d03c26fd16a454b42acc07c5e6a8621b22a7cc1389d8f64e20483b732afd\": rpc error: code = NotFound desc = could not find container \"2d28d03c26fd16a454b42acc07c5e6a8621b22a7cc1389d8f64e20483b732afd\": container with ID starting with 2d28d03c26fd16a454b42acc07c5e6a8621b22a7cc1389d8f64e20483b732afd not found: ID does not exist" Nov 01 21:15:10.985363 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:10.985348 4484 scope.go:122] "RemoveContainer" containerID="34954b986549f7962e7bc82975e3b96755d37bf3a5715279d7a4999bba6eeba9" Nov 01 21:15:11.028133 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:11.028085 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-24f923-869bcbd967-w25fl" podUID="009125b3-fc6d-4cbe-aa45-5cf27cc39a2e" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.163:50052\" within 1s: context deadline exceeded" Nov 01 21:15:11.028314 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:15:11.028192 4484 logging.go:55] [core] [Channel #12943 SubChannel #12944]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.163:50052", ServerName: "10.134.0.163:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.163:50052: operation was canceled" Nov 01 21:15:11.134444 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:11.134410 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-24f923-8474d84f5-kfvdm"] Nov 01 21:15:11.138955 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:11.138928 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-24f923-8474d84f5-kfvdm"] Nov 01 21:15:11.247930 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:11.247895 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892795-njr5t"] Nov 01 21:15:11.252057 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:11.252026 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cd248f-29892795-njr5t"] Nov 01 21:15:11.376228 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:11.376194 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be6565c5-c4bd-4f2c-9d50-0b5359d589e4" path="/var/lib/kubelet/pods/be6565c5-c4bd-4f2c-9d50-0b5359d589e4/volumes" Nov 01 21:15:11.376694 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:11.376679 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9a691db-7b44-40aa-8924-1f2b716da33a" path="/var/lib/kubelet/pods/f9a691db-7b44-40aa-8924-1f2b716da33a/volumes" Nov 01 21:15:11.377060 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:11.377043 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-24f923-869bcbd967-w25fl"] Nov 01 21:15:11.377145 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:11.377079 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-24f923-869bcbd967-w25fl"] Nov 01 21:15:12.016611 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:12.016558 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:12.230191 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:12.230155 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:12.389313 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:12.389236 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:12.493310 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:12.493278 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-cd248f-ffrxv"] Nov 01 21:15:12.544072 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:12.544030 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-cd248f-ffrxv"] Nov 01 21:15:12.970143 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:12.970103 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:13.377019 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:13.376969 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="009125b3-fc6d-4cbe-aa45-5cf27cc39a2e" path="/var/lib/kubelet/pods/009125b3-fc6d-4cbe-aa45-5cf27cc39a2e/volumes" Nov 01 21:15:13.377429 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:13.377413 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4339ad7f-a07a-46b9-a9c2-a7d28a3f5cf0" path="/var/lib/kubelet/pods/4339ad7f-a07a-46b9-a9c2-a7d28a3f5cf0/volumes" Nov 01 21:15:14.365077 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:14.365039 4484 scope.go:122] "RemoveContainer" containerID="a6f9319782ab2fddc1aed1f6b2fd70e47c27cf5b12ef34e1fc9dd7d6d210c801" Nov 01 21:15:14.998309 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:14.998272 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892795-hjxd2" event={"ID":"425105dd-4b1f-4f03-94a1-121733d0ad62","Type":"ContainerStarted","Data":"593fa24172ef2347ca89ad0941d639a7b81753d5925459b663802cad199da97f"} Nov 01 21:15:15.365422 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:15.365327 4484 scope.go:122] "RemoveContainer" containerID="adab1801fc45a9c3b7f2ed514c94a8fa4f278ac1d9c2a34327f8db8a0d278b23" Nov 01 21:15:15.365422 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:15.365390 4484 scope.go:122] "RemoveContainer" containerID="e117337ce07be5e0e2b9478d243726b10222b54104c19d4fb908316319f7dcb3" Nov 01 21:15:15.766090 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:15:15.766055 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:15:16.008137 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:16.008104 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892795-sks9n" event={"ID":"cb830eed-0dbe-4080-b6a0-2c3092adcf5b","Type":"ContainerStarted","Data":"803c39d671cbb9d4c67356acb70ddb16f25c7e2a4d36b14724337751c5f92610"} Nov 01 21:15:16.010142 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:16.010115 4484 generic.go:356] "Generic (PLEG): container finished" podID="425105dd-4b1f-4f03-94a1-121733d0ad62" containerID="593fa24172ef2347ca89ad0941d639a7b81753d5925459b663802cad199da97f" exitCode=60 Nov 01 21:15:16.010305 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:16.010186 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892795-hjxd2" event={"ID":"425105dd-4b1f-4f03-94a1-121733d0ad62","Type":"ContainerDied","Data":"593fa24172ef2347ca89ad0941d639a7b81753d5925459b663802cad199da97f"} Nov 01 21:15:16.010305 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:16.010250 4484 scope.go:122] "RemoveContainer" containerID="a6f9319782ab2fddc1aed1f6b2fd70e47c27cf5b12ef34e1fc9dd7d6d210c801" Nov 01 21:15:16.010636 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:16.010612 4484 scope.go:122] "RemoveContainer" containerID="593fa24172ef2347ca89ad0941d639a7b81753d5925459b663802cad199da97f" Nov 01 21:15:16.010949 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:15:16.010918 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892795-hjxd2_odh-ai-gateway-infra(425105dd-4b1f-4f03-94a1-121733d0ad62)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892795-hjxd2" podUID="425105dd-4b1f-4f03-94a1-121733d0ad62" Nov 01 21:15:16.012296 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:16.012278 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a41242-29892795-lg5ct" event={"ID":"cff29d06-8657-4d3f-aa13-88d10e0c6532","Type":"ContainerStarted","Data":"53d57403f7c56e0ed53ff322366cf72b24ddab7817b2e9699390ba3cac1923ff"} Nov 01 21:15:16.124391 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:16.124353 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892795-hjxd2"] Nov 01 21:15:16.364928 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:16.364898 4484 scope.go:122] "RemoveContainer" containerID="b957c0404c2b523dcfe8093de22ceab13da92a44f36b0cf6e483c24add765576" Nov 01 21:15:16.365123 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:16.365011 4484 scope.go:122] "RemoveContainer" containerID="112bd17004f86a22cb6b1f58048e11867d2b4fc2f5fa2f593a62c03c42f51fb1" Nov 01 21:15:17.022600 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:17.022560 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892795-x5p6l" event={"ID":"f695676a-20ed-4db3-a4f3-2961bc48be9a","Type":"ContainerStarted","Data":"b3bee6c76122aeed0a46c29b2dbcc706f1cf194d1c9d4bedef1cd610f52b701e"} Nov 01 21:15:17.024655 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:17.024633 4484 generic.go:356] "Generic (PLEG): container finished" podID="cb830eed-0dbe-4080-b6a0-2c3092adcf5b" containerID="803c39d671cbb9d4c67356acb70ddb16f25c7e2a4d36b14724337751c5f92610" exitCode=60 Nov 01 21:15:17.024801 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:17.024708 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892795-sks9n" event={"ID":"cb830eed-0dbe-4080-b6a0-2c3092adcf5b","Type":"ContainerDied","Data":"803c39d671cbb9d4c67356acb70ddb16f25c7e2a4d36b14724337751c5f92610"} Nov 01 21:15:17.024801 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:17.024757 4484 scope.go:122] "RemoveContainer" containerID="adab1801fc45a9c3b7f2ed514c94a8fa4f278ac1d9c2a34327f8db8a0d278b23" Nov 01 21:15:17.025087 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:17.025071 4484 scope.go:122] "RemoveContainer" containerID="803c39d671cbb9d4c67356acb70ddb16f25c7e2a4d36b14724337751c5f92610" Nov 01 21:15:17.025324 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:15:17.025300 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-944263-29892795-sks9n_odh-ai-gateway-infra(cb830eed-0dbe-4080-b6a0-2c3092adcf5b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892795-sks9n" podUID="cb830eed-0dbe-4080-b6a0-2c3092adcf5b" Nov 01 21:15:17.028483 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:17.028463 4484 generic.go:356] "Generic (PLEG): container finished" podID="cff29d06-8657-4d3f-aa13-88d10e0c6532" containerID="53d57403f7c56e0ed53ff322366cf72b24ddab7817b2e9699390ba3cac1923ff" exitCode=60 Nov 01 21:15:17.028623 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:17.028520 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a41242-29892795-lg5ct" event={"ID":"cff29d06-8657-4d3f-aa13-88d10e0c6532","Type":"ContainerDied","Data":"53d57403f7c56e0ed53ff322366cf72b24ddab7817b2e9699390ba3cac1923ff"} Nov 01 21:15:17.028884 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:17.028824 4484 scope.go:122] "RemoveContainer" containerID="53d57403f7c56e0ed53ff322366cf72b24ddab7817b2e9699390ba3cac1923ff" Nov 01 21:15:17.029089 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:15:17.029069 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-a41242-29892795-lg5ct_odh-ai-gateway-infra(cff29d06-8657-4d3f-aa13-88d10e0c6532)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a41242-29892795-lg5ct" podUID="cff29d06-8657-4d3f-aa13-88d10e0c6532" Nov 01 21:15:17.030597 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:17.030577 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-670df9-29892795-n9d77" event={"ID":"d44ebf49-b772-485f-a982-055f9dd19efc","Type":"ContainerStarted","Data":"746d53800bccaf849d346c16bcde0f9a0a4b35fef182129ed6073af1b3796fed"} Nov 01 21:15:17.038422 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:17.038397 4484 scope.go:122] "RemoveContainer" containerID="e117337ce07be5e0e2b9478d243726b10222b54104c19d4fb908316319f7dcb3" Nov 01 21:15:17.138265 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:17.138226 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892795-sks9n"] Nov 01 21:15:17.183944 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:17.183914 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892795-hjxd2" Nov 01 21:15:17.249432 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:17.249396 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a41242-29892795-lg5ct"] Nov 01 21:15:17.296868 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:17.296782 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/425105dd-4b1f-4f03-94a1-121733d0ad62-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/425105dd-4b1f-4f03-94a1-121733d0ad62-maas-api-service-ca\") pod \"425105dd-4b1f-4f03-94a1-121733d0ad62\" (UID: \"425105dd-4b1f-4f03-94a1-121733d0ad62\") " Nov 01 21:15:17.297054 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:17.296875 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/425105dd-4b1f-4f03-94a1-121733d0ad62-kube-api-access-x8jzj\" (UniqueName: \"kubernetes.io/projected/425105dd-4b1f-4f03-94a1-121733d0ad62-kube-api-access-x8jzj\") pod \"425105dd-4b1f-4f03-94a1-121733d0ad62\" (UID: \"425105dd-4b1f-4f03-94a1-121733d0ad62\") " Nov 01 21:15:17.297169 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:17.297146 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/425105dd-4b1f-4f03-94a1-121733d0ad62-maas-api-service-ca" pod "425105dd-4b1f-4f03-94a1-121733d0ad62" (UID: "425105dd-4b1f-4f03-94a1-121733d0ad62"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:15:17.297313 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:17.297294 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/425105dd-4b1f-4f03-94a1-121733d0ad62-maas-api-service-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:17.298898 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:17.298879 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/425105dd-4b1f-4f03-94a1-121733d0ad62-kube-api-access-x8jzj" pod "425105dd-4b1f-4f03-94a1-121733d0ad62" (UID: "425105dd-4b1f-4f03-94a1-121733d0ad62"). InnerVolumeSpecName "kube-api-access-x8jzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:15:17.364750 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:17.364722 4484 scope.go:122] "RemoveContainer" containerID="ce0fe23b48571d9f09d0ead0771db9b723e7d14eb3eec117a32f36c9b5637d2a" Nov 01 21:15:17.364944 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:17.364870 4484 scope.go:122] "RemoveContainer" containerID="a587a4d80aeb0ec663f463da45ae60c7c6d41df84ad66d0d736b1772379d90fb" Nov 01 21:15:17.364944 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:17.364934 4484 scope.go:122] "RemoveContainer" containerID="dedc905385dfc2826cd6292fa51cf6619c01581198fbb0b9deffb9365bfb6266" Nov 01 21:15:17.399220 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:17.399188 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8jzj\" (UniqueName: \"kubernetes.io/projected/425105dd-4b1f-4f03-94a1-121733d0ad62-kube-api-access-x8jzj\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:17.419587 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:17.419552 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:18.038915 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:18.038876 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892795-fqldx" event={"ID":"9da77fa2-9d96-4a3d-9a1e-2d23a7d8fb0c","Type":"ContainerStarted","Data":"b9f80c055a86d58a1553a5b6bd191876fd43ed799b0f07aba7672f9c8c46e893"} Nov 01 21:15:18.040863 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:18.040840 4484 generic.go:356] "Generic (PLEG): container finished" podID="f695676a-20ed-4db3-a4f3-2961bc48be9a" containerID="b3bee6c76122aeed0a46c29b2dbcc706f1cf194d1c9d4bedef1cd610f52b701e" exitCode=60 Nov 01 21:15:18.041028 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:18.040912 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892795-x5p6l" event={"ID":"f695676a-20ed-4db3-a4f3-2961bc48be9a","Type":"ContainerDied","Data":"b3bee6c76122aeed0a46c29b2dbcc706f1cf194d1c9d4bedef1cd610f52b701e"} Nov 01 21:15:18.041028 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:18.040960 4484 scope.go:122] "RemoveContainer" containerID="b957c0404c2b523dcfe8093de22ceab13da92a44f36b0cf6e483c24add765576" Nov 01 21:15:18.041339 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:18.041312 4484 scope.go:122] "RemoveContainer" containerID="b3bee6c76122aeed0a46c29b2dbcc706f1cf194d1c9d4bedef1cd610f52b701e" Nov 01 21:15:18.041655 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:15:18.041628 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-db0157-29892795-x5p6l_odh-ai-gateway-infra(f695676a-20ed-4db3-a4f3-2961bc48be9a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892795-x5p6l" podUID="f695676a-20ed-4db3-a4f3-2961bc48be9a" Nov 01 21:15:18.043389 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:18.043366 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892795-vj4nc" event={"ID":"59565b69-6f84-415f-b765-223943cd28d1","Type":"ContainerStarted","Data":"2f1c3fd81e32857327a68174b3e4cd0ed9e00d54c6207e669257cb9a71562f68"} Nov 01 21:15:18.049090 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:18.048538 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892795-8dd2w" event={"ID":"9464b311-fde8-4875-aafa-ee5c07166a4e","Type":"ContainerStarted","Data":"7fd9992e66f1fb3dad09129b46c8da8b663a8a7786fccb29a9075894754e09fe"} Nov 01 21:15:18.052269 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:18.052236 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892795-hjxd2" Nov 01 21:15:18.052269 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:18.052254 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892795-hjxd2" event={"ID":"425105dd-4b1f-4f03-94a1-121733d0ad62","Type":"ContainerDied","Data":"722bc4a96a12f9f5e482af2e61bbd98308e47dac56f4558b2d00a815f079ac85"} Nov 01 21:15:18.059093 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:18.059053 4484 generic.go:356] "Generic (PLEG): container finished" podID="d44ebf49-b772-485f-a982-055f9dd19efc" containerID="746d53800bccaf849d346c16bcde0f9a0a4b35fef182129ed6073af1b3796fed" exitCode=60 Nov 01 21:15:18.059232 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:18.059148 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-670df9-29892795-n9d77" event={"ID":"d44ebf49-b772-485f-a982-055f9dd19efc","Type":"ContainerDied","Data":"746d53800bccaf849d346c16bcde0f9a0a4b35fef182129ed6073af1b3796fed"} Nov 01 21:15:18.059554 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:18.059531 4484 scope.go:122] "RemoveContainer" containerID="746d53800bccaf849d346c16bcde0f9a0a4b35fef182129ed6073af1b3796fed" Nov 01 21:15:18.059870 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:15:18.059844 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-670df9-29892795-n9d77_odh-ai-gateway-infra(d44ebf49-b772-485f-a982-055f9dd19efc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-670df9-29892795-n9d77" podUID="d44ebf49-b772-485f-a982-055f9dd19efc" Nov 01 21:15:18.060094 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:18.060076 4484 scope.go:122] "RemoveContainer" containerID="593fa24172ef2347ca89ad0941d639a7b81753d5925459b663802cad199da97f" Nov 01 21:15:18.073579 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:18.073548 4484 scope.go:122] "RemoveContainer" containerID="112bd17004f86a22cb6b1f58048e11867d2b4fc2f5fa2f593a62c03c42f51fb1" Nov 01 21:15:18.147383 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:18.147348 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892795-x5p6l"] Nov 01 21:15:18.230263 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:18.230225 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892795-sks9n" Nov 01 21:15:18.243927 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:18.243902 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a41242-29892795-lg5ct" Nov 01 21:15:18.310490 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:18.310372 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb830eed-0dbe-4080-b6a0-2c3092adcf5b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb830eed-0dbe-4080-b6a0-2c3092adcf5b-maas-api-service-ca\") pod \"cb830eed-0dbe-4080-b6a0-2c3092adcf5b\" (UID: \"cb830eed-0dbe-4080-b6a0-2c3092adcf5b\") " Nov 01 21:15:18.310490 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:18.310432 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb830eed-0dbe-4080-b6a0-2c3092adcf5b-kube-api-access-l8gjv\" (UniqueName: \"kubernetes.io/projected/cb830eed-0dbe-4080-b6a0-2c3092adcf5b-kube-api-access-l8gjv\") pod \"cb830eed-0dbe-4080-b6a0-2c3092adcf5b\" (UID: \"cb830eed-0dbe-4080-b6a0-2c3092adcf5b\") " Nov 01 21:15:18.310724 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:18.310701 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb830eed-0dbe-4080-b6a0-2c3092adcf5b-maas-api-service-ca" pod "cb830eed-0dbe-4080-b6a0-2c3092adcf5b" (UID: "cb830eed-0dbe-4080-b6a0-2c3092adcf5b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:15:18.310954 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:18.310933 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb830eed-0dbe-4080-b6a0-2c3092adcf5b-maas-api-service-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:18.312624 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:18.312599 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb830eed-0dbe-4080-b6a0-2c3092adcf5b-kube-api-access-l8gjv" pod "cb830eed-0dbe-4080-b6a0-2c3092adcf5b" (UID: "cb830eed-0dbe-4080-b6a0-2c3092adcf5b"). InnerVolumeSpecName "kube-api-access-l8gjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:15:18.411394 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:18.411364 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cff29d06-8657-4d3f-aa13-88d10e0c6532-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cff29d06-8657-4d3f-aa13-88d10e0c6532-maas-api-service-ca\") pod \"cff29d06-8657-4d3f-aa13-88d10e0c6532\" (UID: \"cff29d06-8657-4d3f-aa13-88d10e0c6532\") " Nov 01 21:15:18.411559 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:18.411538 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cff29d06-8657-4d3f-aa13-88d10e0c6532-kube-api-access-2vwwx\" (UniqueName: \"kubernetes.io/projected/cff29d06-8657-4d3f-aa13-88d10e0c6532-kube-api-access-2vwwx\") pod \"cff29d06-8657-4d3f-aa13-88d10e0c6532\" (UID: \"cff29d06-8657-4d3f-aa13-88d10e0c6532\") " Nov 01 21:15:18.411762 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:18.411738 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cff29d06-8657-4d3f-aa13-88d10e0c6532-maas-api-service-ca" pod "cff29d06-8657-4d3f-aa13-88d10e0c6532" (UID: "cff29d06-8657-4d3f-aa13-88d10e0c6532"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:15:18.411961 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:18.411944 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cff29d06-8657-4d3f-aa13-88d10e0c6532-maas-api-service-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:18.412049 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:18.411966 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8gjv\" (UniqueName: \"kubernetes.io/projected/cb830eed-0dbe-4080-b6a0-2c3092adcf5b-kube-api-access-l8gjv\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:18.413624 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:18.413597 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cff29d06-8657-4d3f-aa13-88d10e0c6532-kube-api-access-2vwwx" pod "cff29d06-8657-4d3f-aa13-88d10e0c6532" (UID: "cff29d06-8657-4d3f-aa13-88d10e0c6532"). InnerVolumeSpecName "kube-api-access-2vwwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:15:18.509933 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:18.509903 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892795-hjxd2"] Nov 01 21:15:18.513250 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:18.513223 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vwwx\" (UniqueName: \"kubernetes.io/projected/cff29d06-8657-4d3f-aa13-88d10e0c6532-kube-api-access-2vwwx\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:18.513476 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:18.513455 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dce296-29892795-hjxd2"] Nov 01 21:15:18.577143 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:18.577050 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:18.631074 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:18.631039 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-670df9-29892795-n9d77"] Nov 01 21:15:18.743945 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:18.743899 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:18.913541 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:18.913426 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:19.068373 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:19.068336 4484 generic.go:356] "Generic (PLEG): container finished" podID="59565b69-6f84-415f-b765-223943cd28d1" containerID="2f1c3fd81e32857327a68174b3e4cd0ed9e00d54c6207e669257cb9a71562f68" exitCode=60 Nov 01 21:15:19.068809 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:19.068410 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892795-vj4nc" event={"ID":"59565b69-6f84-415f-b765-223943cd28d1","Type":"ContainerDied","Data":"2f1c3fd81e32857327a68174b3e4cd0ed9e00d54c6207e669257cb9a71562f68"} Nov 01 21:15:19.068809 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:19.068477 4484 scope.go:122] "RemoveContainer" containerID="ce0fe23b48571d9f09d0ead0771db9b723e7d14eb3eec117a32f36c9b5637d2a" Nov 01 21:15:19.068939 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:19.068847 4484 scope.go:122] "RemoveContainer" containerID="2f1c3fd81e32857327a68174b3e4cd0ed9e00d54c6207e669257cb9a71562f68" Nov 01 21:15:19.069172 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:15:19.069149 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892795-vj4nc_odh-ai-gateway-infra(59565b69-6f84-415f-b765-223943cd28d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892795-vj4nc" podUID="59565b69-6f84-415f-b765-223943cd28d1" Nov 01 21:15:19.069972 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:19.069950 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892795-sks9n" Nov 01 21:15:19.070093 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:19.069957 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892795-sks9n" event={"ID":"cb830eed-0dbe-4080-b6a0-2c3092adcf5b","Type":"ContainerDied","Data":"ef12a02e2fac94c17db0332f97520e855592a200f3bd0013ca5387d3b71f846f"} Nov 01 21:15:19.071937 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:19.071916 4484 generic.go:356] "Generic (PLEG): container finished" podID="9464b311-fde8-4875-aafa-ee5c07166a4e" containerID="7fd9992e66f1fb3dad09129b46c8da8b663a8a7786fccb29a9075894754e09fe" exitCode=60 Nov 01 21:15:19.072085 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:19.071939 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892795-8dd2w" event={"ID":"9464b311-fde8-4875-aafa-ee5c07166a4e","Type":"ContainerDied","Data":"7fd9992e66f1fb3dad09129b46c8da8b663a8a7786fccb29a9075894754e09fe"} Nov 01 21:15:19.072338 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:19.072316 4484 scope.go:122] "RemoveContainer" containerID="7fd9992e66f1fb3dad09129b46c8da8b663a8a7786fccb29a9075894754e09fe" Nov 01 21:15:19.072594 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:15:19.072562 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892795-8dd2w_odh-ai-gateway-infra(9464b311-fde8-4875-aafa-ee5c07166a4e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892795-8dd2w" podUID="9464b311-fde8-4875-aafa-ee5c07166a4e" Nov 01 21:15:19.074601 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:19.074582 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a41242-29892795-lg5ct" Nov 01 21:15:19.074825 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:19.074807 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a41242-29892795-lg5ct" event={"ID":"cff29d06-8657-4d3f-aa13-88d10e0c6532","Type":"ContainerDied","Data":"3e41ca261be8349a2ccbf3453b713143e590f5ed6de0a65dd44f929c0c4bf6d7"} Nov 01 21:15:19.078500 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:19.078474 4484 generic.go:356] "Generic (PLEG): container finished" podID="9da77fa2-9d96-4a3d-9a1e-2d23a7d8fb0c" containerID="b9f80c055a86d58a1553a5b6bd191876fd43ed799b0f07aba7672f9c8c46e893" exitCode=60 Nov 01 21:15:19.078600 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:19.078580 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892795-fqldx" event={"ID":"9da77fa2-9d96-4a3d-9a1e-2d23a7d8fb0c","Type":"ContainerDied","Data":"b9f80c055a86d58a1553a5b6bd191876fd43ed799b0f07aba7672f9c8c46e893"} Nov 01 21:15:19.078973 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:19.078953 4484 scope.go:122] "RemoveContainer" containerID="b9f80c055a86d58a1553a5b6bd191876fd43ed799b0f07aba7672f9c8c46e893" Nov 01 21:15:19.079260 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:15:19.079233 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892795-fqldx_odh-ai-gateway-infra(9da77fa2-9d96-4a3d-9a1e-2d23a7d8fb0c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892795-fqldx" podUID="9da77fa2-9d96-4a3d-9a1e-2d23a7d8fb0c" Nov 01 21:15:19.083701 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:19.083683 4484 scope.go:122] "RemoveContainer" containerID="803c39d671cbb9d4c67356acb70ddb16f25c7e2a4d36b14724337751c5f92610" Nov 01 21:15:19.096272 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:19.096243 4484 scope.go:122] "RemoveContainer" containerID="dedc905385dfc2826cd6292fa51cf6619c01581198fbb0b9deffb9365bfb6266" Nov 01 21:15:19.114594 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:19.114515 4484 scope.go:122] "RemoveContainer" containerID="53d57403f7c56e0ed53ff322366cf72b24ddab7817b2e9699390ba3cac1923ff" Nov 01 21:15:19.136918 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:19.136199 4484 scope.go:122] "RemoveContainer" containerID="a587a4d80aeb0ec663f463da45ae60c7c6d41df84ad66d0d736b1772379d90fb" Nov 01 21:15:19.165664 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:19.165591 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892795-fqldx"] Nov 01 21:15:19.259723 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:19.259695 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892795-x5p6l" Nov 01 21:15:19.266229 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:19.266207 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-670df9-29892795-n9d77" Nov 01 21:15:19.287487 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:19.287455 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892795-vj4nc"] Nov 01 21:15:19.371390 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:19.371356 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="425105dd-4b1f-4f03-94a1-121733d0ad62" path="/var/lib/kubelet/pods/425105dd-4b1f-4f03-94a1-121733d0ad62/volumes" Nov 01 21:15:19.422625 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:19.422542 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d44ebf49-b772-485f-a982-055f9dd19efc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d44ebf49-b772-485f-a982-055f9dd19efc-maas-api-service-ca\") pod \"d44ebf49-b772-485f-a982-055f9dd19efc\" (UID: \"d44ebf49-b772-485f-a982-055f9dd19efc\") " Nov 01 21:15:19.422625 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:19.422595 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d44ebf49-b772-485f-a982-055f9dd19efc-kube-api-access-8grt8\" (UniqueName: \"kubernetes.io/projected/d44ebf49-b772-485f-a982-055f9dd19efc-kube-api-access-8grt8\") pod \"d44ebf49-b772-485f-a982-055f9dd19efc\" (UID: \"d44ebf49-b772-485f-a982-055f9dd19efc\") " Nov 01 21:15:19.422870 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:19.422700 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f695676a-20ed-4db3-a4f3-2961bc48be9a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f695676a-20ed-4db3-a4f3-2961bc48be9a-maas-api-service-ca\") pod \"f695676a-20ed-4db3-a4f3-2961bc48be9a\" (UID: \"f695676a-20ed-4db3-a4f3-2961bc48be9a\") " Nov 01 21:15:19.422870 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:19.422722 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f695676a-20ed-4db3-a4f3-2961bc48be9a-kube-api-access-8zgd9\" (UniqueName: \"kubernetes.io/projected/f695676a-20ed-4db3-a4f3-2961bc48be9a-kube-api-access-8zgd9\") pod \"f695676a-20ed-4db3-a4f3-2961bc48be9a\" (UID: \"f695676a-20ed-4db3-a4f3-2961bc48be9a\") " Nov 01 21:15:19.423033 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:19.422982 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d44ebf49-b772-485f-a982-055f9dd19efc-maas-api-service-ca" pod "d44ebf49-b772-485f-a982-055f9dd19efc" (UID: "d44ebf49-b772-485f-a982-055f9dd19efc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:15:19.423113 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:19.423065 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f695676a-20ed-4db3-a4f3-2961bc48be9a-maas-api-service-ca" pod "f695676a-20ed-4db3-a4f3-2961bc48be9a" (UID: "f695676a-20ed-4db3-a4f3-2961bc48be9a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:15:19.424810 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:19.424792 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f695676a-20ed-4db3-a4f3-2961bc48be9a-kube-api-access-8zgd9" pod "f695676a-20ed-4db3-a4f3-2961bc48be9a" (UID: "f695676a-20ed-4db3-a4f3-2961bc48be9a"). InnerVolumeSpecName "kube-api-access-8zgd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:15:19.424917 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:19.424898 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d44ebf49-b772-485f-a982-055f9dd19efc-kube-api-access-8grt8" pod "d44ebf49-b772-485f-a982-055f9dd19efc" (UID: "d44ebf49-b772-485f-a982-055f9dd19efc"). InnerVolumeSpecName "kube-api-access-8grt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:15:19.523662 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:19.523628 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d44ebf49-b772-485f-a982-055f9dd19efc-maas-api-service-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:19.523662 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:19.523656 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8grt8\" (UniqueName: \"kubernetes.io/projected/d44ebf49-b772-485f-a982-055f9dd19efc-kube-api-access-8grt8\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:19.523662 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:19.523666 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f695676a-20ed-4db3-a4f3-2961bc48be9a-maas-api-service-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:19.523906 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:19.523676 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zgd9\" (UniqueName: \"kubernetes.io/projected/f695676a-20ed-4db3-a4f3-2961bc48be9a-kube-api-access-8zgd9\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:19.638951 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:19.638917 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892795-8dd2w"] Nov 01 21:15:19.663095 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:19.663059 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a41242-29892795-lg5ct"] Nov 01 21:15:19.669750 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:19.669702 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a41242-29892795-lg5ct"] Nov 01 21:15:19.898506 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:19.898460 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892795-sks9n"] Nov 01 21:15:19.901331 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:19.901300 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-944263-29892795-sks9n"] Nov 01 21:15:20.090970 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:20.090940 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-670df9-29892795-n9d77" Nov 01 21:15:20.091464 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:20.090972 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-670df9-29892795-n9d77" event={"ID":"d44ebf49-b772-485f-a982-055f9dd19efc","Type":"ContainerDied","Data":"43a8415276a93a2d22c94eaf7dfd46d02fb6d7200f792b02df7e49607f7b764e"} Nov 01 21:15:20.091464 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:20.091100 4484 scope.go:122] "RemoveContainer" containerID="746d53800bccaf849d346c16bcde0f9a0a4b35fef182129ed6073af1b3796fed" Nov 01 21:15:20.094756 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:20.094735 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892795-x5p6l" Nov 01 21:15:20.094880 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:20.094746 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892795-x5p6l" event={"ID":"f695676a-20ed-4db3-a4f3-2961bc48be9a","Type":"ContainerDied","Data":"556a3b3f52e9dc87f4dc2e52bef1e045b6b05b52fd02a0f6f02ed9fc27a9b341"} Nov 01 21:15:20.112182 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:20.112154 4484 scope.go:122] "RemoveContainer" containerID="b3bee6c76122aeed0a46c29b2dbcc706f1cf194d1c9d4bedef1cd610f52b701e" Nov 01 21:15:20.311091 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:20.311068 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892795-8dd2w" Nov 01 21:15:20.315862 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:20.315839 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892795-fqldx" Nov 01 21:15:20.318672 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:20.318650 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892795-vj4nc" Nov 01 21:15:20.438342 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:20.438255 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9da77fa2-9d96-4a3d-9a1e-2d23a7d8fb0c-kube-api-access-5x5mp\" (UniqueName: \"kubernetes.io/projected/9da77fa2-9d96-4a3d-9a1e-2d23a7d8fb0c-kube-api-access-5x5mp\") pod \"9da77fa2-9d96-4a3d-9a1e-2d23a7d8fb0c\" (UID: \"9da77fa2-9d96-4a3d-9a1e-2d23a7d8fb0c\") " Nov 01 21:15:20.438533 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:20.438379 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59565b69-6f84-415f-b765-223943cd28d1-kube-api-access-nm4mf\" (UniqueName: \"kubernetes.io/projected/59565b69-6f84-415f-b765-223943cd28d1-kube-api-access-nm4mf\") pod \"59565b69-6f84-415f-b765-223943cd28d1\" (UID: \"59565b69-6f84-415f-b765-223943cd28d1\") " Nov 01 21:15:20.438533 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:20.438450 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9464b311-fde8-4875-aafa-ee5c07166a4e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9464b311-fde8-4875-aafa-ee5c07166a4e-maas-api-service-ca\") pod \"9464b311-fde8-4875-aafa-ee5c07166a4e\" (UID: \"9464b311-fde8-4875-aafa-ee5c07166a4e\") " Nov 01 21:15:20.438533 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:20.438478 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9464b311-fde8-4875-aafa-ee5c07166a4e-kube-api-access-6pzqv\" (UniqueName: \"kubernetes.io/projected/9464b311-fde8-4875-aafa-ee5c07166a4e-kube-api-access-6pzqv\") pod \"9464b311-fde8-4875-aafa-ee5c07166a4e\" (UID: \"9464b311-fde8-4875-aafa-ee5c07166a4e\") " Nov 01 21:15:20.438533 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:20.438520 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59565b69-6f84-415f-b765-223943cd28d1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59565b69-6f84-415f-b765-223943cd28d1-maas-api-service-ca\") pod \"59565b69-6f84-415f-b765-223943cd28d1\" (UID: \"59565b69-6f84-415f-b765-223943cd28d1\") " Nov 01 21:15:20.438852 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:20.438586 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9da77fa2-9d96-4a3d-9a1e-2d23a7d8fb0c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9da77fa2-9d96-4a3d-9a1e-2d23a7d8fb0c-maas-api-service-ca\") pod \"9da77fa2-9d96-4a3d-9a1e-2d23a7d8fb0c\" (UID: \"9da77fa2-9d96-4a3d-9a1e-2d23a7d8fb0c\") " Nov 01 21:15:20.438920 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:20.438882 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59565b69-6f84-415f-b765-223943cd28d1-maas-api-service-ca" pod "59565b69-6f84-415f-b765-223943cd28d1" (UID: "59565b69-6f84-415f-b765-223943cd28d1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:15:20.438920 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:20.438901 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9464b311-fde8-4875-aafa-ee5c07166a4e-maas-api-service-ca" pod "9464b311-fde8-4875-aafa-ee5c07166a4e" (UID: "9464b311-fde8-4875-aafa-ee5c07166a4e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:15:20.439038 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:20.438983 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9da77fa2-9d96-4a3d-9a1e-2d23a7d8fb0c-maas-api-service-ca" pod "9da77fa2-9d96-4a3d-9a1e-2d23a7d8fb0c" (UID: "9da77fa2-9d96-4a3d-9a1e-2d23a7d8fb0c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:15:20.439163 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:20.439136 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9da77fa2-9d96-4a3d-9a1e-2d23a7d8fb0c-maas-api-service-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:20.439163 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:20.439163 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9464b311-fde8-4875-aafa-ee5c07166a4e-maas-api-service-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:20.439373 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:20.439177 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59565b69-6f84-415f-b765-223943cd28d1-maas-api-service-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:20.440663 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:20.440641 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9da77fa2-9d96-4a3d-9a1e-2d23a7d8fb0c-kube-api-access-5x5mp" pod "9da77fa2-9d96-4a3d-9a1e-2d23a7d8fb0c" (UID: "9da77fa2-9d96-4a3d-9a1e-2d23a7d8fb0c"). InnerVolumeSpecName "kube-api-access-5x5mp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:15:20.440747 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:20.440660 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59565b69-6f84-415f-b765-223943cd28d1-kube-api-access-nm4mf" pod "59565b69-6f84-415f-b765-223943cd28d1" (UID: "59565b69-6f84-415f-b765-223943cd28d1"). InnerVolumeSpecName "kube-api-access-nm4mf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:15:20.440747 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:20.440644 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9464b311-fde8-4875-aafa-ee5c07166a4e-kube-api-access-6pzqv" pod "9464b311-fde8-4875-aafa-ee5c07166a4e" (UID: "9464b311-fde8-4875-aafa-ee5c07166a4e"). InnerVolumeSpecName "kube-api-access-6pzqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:15:20.475832 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:20.475800 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-670df9-29892795-n9d77"] Nov 01 21:15:20.482148 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:20.482091 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-670df9-29892795-n9d77"] Nov 01 21:15:20.540296 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:20.540263 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5x5mp\" (UniqueName: \"kubernetes.io/projected/9da77fa2-9d96-4a3d-9a1e-2d23a7d8fb0c-kube-api-access-5x5mp\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:20.540296 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:20.540292 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm4mf\" (UniqueName: \"kubernetes.io/projected/59565b69-6f84-415f-b765-223943cd28d1-kube-api-access-nm4mf\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:20.540296 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:20.540303 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pzqv\" (UniqueName: \"kubernetes.io/projected/9464b311-fde8-4875-aafa-ee5c07166a4e-kube-api-access-6pzqv\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:20.593113 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:20.593074 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892795-x5p6l"] Nov 01 21:15:20.596286 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:20.596258 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-db0157-29892795-x5p6l"] Nov 01 21:15:21.109970 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:21.109940 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892795-fqldx" Nov 01 21:15:21.110513 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:21.109966 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892795-fqldx" event={"ID":"9da77fa2-9d96-4a3d-9a1e-2d23a7d8fb0c","Type":"ContainerDied","Data":"e792ab85cb6389c3953bc9912337539d15b972299f52753a0f8ef873ff20256f"} Nov 01 21:15:21.110513 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:21.110055 4484 scope.go:122] "RemoveContainer" containerID="b9f80c055a86d58a1553a5b6bd191876fd43ed799b0f07aba7672f9c8c46e893" Nov 01 21:15:21.112650 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:21.112625 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892795-vj4nc" Nov 01 21:15:21.112764 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:21.112621 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892795-vj4nc" event={"ID":"59565b69-6f84-415f-b765-223943cd28d1","Type":"ContainerDied","Data":"f77b3e4df133ad433c4c2f2f3e4021b370b08ad545b8e21e37c5fbdf62763599"} Nov 01 21:15:21.114301 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:21.114276 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892795-8dd2w" event={"ID":"9464b311-fde8-4875-aafa-ee5c07166a4e","Type":"ContainerDied","Data":"ced5156f930cf354feced59cadeaa27ecd7486bfa4c2ecc40707bef49031d026"} Nov 01 21:15:21.114455 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:21.114308 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892795-8dd2w" Nov 01 21:15:21.121641 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:21.121620 4484 scope.go:122] "RemoveContainer" containerID="2f1c3fd81e32857327a68174b3e4cd0ed9e00d54c6207e669257cb9a71562f68" Nov 01 21:15:21.131583 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:21.131565 4484 scope.go:122] "RemoveContainer" containerID="7fd9992e66f1fb3dad09129b46c8da8b663a8a7786fccb29a9075894754e09fe" Nov 01 21:15:21.277668 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:21.277638 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892795-8dd2w"] Nov 01 21:15:21.280264 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:21.280242 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-df700a-29892795-8dd2w"] Nov 01 21:15:21.371328 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:21.371249 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9464b311-fde8-4875-aafa-ee5c07166a4e" path="/var/lib/kubelet/pods/9464b311-fde8-4875-aafa-ee5c07166a4e/volumes" Nov 01 21:15:21.371699 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:21.371682 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb830eed-0dbe-4080-b6a0-2c3092adcf5b" path="/var/lib/kubelet/pods/cb830eed-0dbe-4080-b6a0-2c3092adcf5b/volumes" Nov 01 21:15:21.372218 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:21.372201 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cff29d06-8657-4d3f-aa13-88d10e0c6532" path="/var/lib/kubelet/pods/cff29d06-8657-4d3f-aa13-88d10e0c6532/volumes" Nov 01 21:15:21.372891 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:21.372877 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d44ebf49-b772-485f-a982-055f9dd19efc" path="/var/lib/kubelet/pods/d44ebf49-b772-485f-a982-055f9dd19efc/volumes" Nov 01 21:15:21.373288 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:21.373275 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f695676a-20ed-4db3-a4f3-2961bc48be9a" path="/var/lib/kubelet/pods/f695676a-20ed-4db3-a4f3-2961bc48be9a/volumes" Nov 01 21:15:21.389826 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:21.389792 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892795-fqldx"] Nov 01 21:15:21.394453 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:21.394425 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1a1e9e-29892795-fqldx"] Nov 01 21:15:21.504377 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:21.504340 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892795-vj4nc"] Nov 01 21:15:21.509303 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:21.509272 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf3fa2-29892795-vj4nc"] Nov 01 21:15:21.861066 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:21.861030 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:22.047039 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:22.047008 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9651f055-qb4vx"] Nov 01 21:15:22.050732 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:22.050695 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9651f055-qb4vx"] Nov 01 21:15:22.053230 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:22.053205 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d74f8c1e-sdwfd"] Nov 01 21:15:22.055397 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:22.055378 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d74f8c1e-sdwfd"] Nov 01 21:15:23.258488 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:23.258452 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:23.371976 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:23.371943 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59565b69-6f84-415f-b765-223943cd28d1" path="/var/lib/kubelet/pods/59565b69-6f84-415f-b765-223943cd28d1/volumes" Nov 01 21:15:23.372437 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:23.372423 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69b1f9a7-bdcc-4a23-937e-07957ceca362" path="/var/lib/kubelet/pods/69b1f9a7-bdcc-4a23-937e-07957ceca362/volumes" Nov 01 21:15:23.372802 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:23.372789 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b380568-48c5-4410-8ddc-3537d86ad229" path="/var/lib/kubelet/pods/8b380568-48c5-4410-8ddc-3537d86ad229/volumes" Nov 01 21:15:23.373344 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:23.373325 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9da77fa2-9d96-4a3d-9a1e-2d23a7d8fb0c" path="/var/lib/kubelet/pods/9da77fa2-9d96-4a3d-9a1e-2d23a7d8fb0c/volumes" Nov 01 21:15:23.591137 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:23.590983 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:23.855650 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:23.855543 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:24.063927 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:24.063884 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-dce296-m5bf4"] Nov 01 21:15:24.120070 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:24.116670 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-dce296-m5bf4"] Nov 01 21:15:24.458124 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:24.458043 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:24.907914 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:24.907870 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:25.143053 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:25.143018 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:25.299933 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:25.299903 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:25.371913 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:25.371879 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24b80eae-a972-46e8-bd86-f67c5452d3b1" path="/var/lib/kubelet/pods/24b80eae-a972-46e8-bd86-f67c5452d3b1/volumes" Nov 01 21:15:25.417660 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:25.417624 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-670df9-fplpk"] Nov 01 21:15:25.468338 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:25.468309 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-670df9-fplpk"] Nov 01 21:15:25.812974 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:25.812940 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:27.373591 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:27.373550 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bb7e26a-1e45-4f1d-b40c-bfeb2efce36e" path="/var/lib/kubelet/pods/8bb7e26a-1e45-4f1d-b40c-bfeb2efce36e/volumes" Nov 01 21:15:30.033528 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:30.033485 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:31.096976 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.096944 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-df700a-t5hcp"] Nov 01 21:15:31.097497 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097482 4484 cpu_manager.go:414] "containerMap: removing container" podUID="cb830eed-0dbe-4080-b6a0-2c3092adcf5b" containerName="cleanup" Nov 01 21:15:31.097554 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097498 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb830eed-0dbe-4080-b6a0-2c3092adcf5b" containerName="cleanup" Nov 01 21:15:31.097554 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097510 4484 cpu_manager.go:414] "containerMap: removing container" podUID="6a48915d-f25c-4aae-bd0d-2714992416d1" containerName="istio-proxy" Nov 01 21:15:31.097554 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097516 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a48915d-f25c-4aae-bd0d-2714992416d1" containerName="istio-proxy" Nov 01 21:15:31.097554 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097524 4484 cpu_manager.go:414] "containerMap: removing container" podUID="d44ebf49-b772-485f-a982-055f9dd19efc" containerName="cleanup" Nov 01 21:15:31.097554 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097529 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d44ebf49-b772-485f-a982-055f9dd19efc" containerName="cleanup" Nov 01 21:15:31.097554 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097537 4484 cpu_manager.go:414] "containerMap: removing container" podUID="8da24464-e245-4061-a89b-9e9e9f61036d" containerName="istio-proxy" Nov 01 21:15:31.097554 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097541 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8da24464-e245-4061-a89b-9e9e9f61036d" containerName="istio-proxy" Nov 01 21:15:31.097554 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097554 4484 cpu_manager.go:414] "containerMap: removing container" podUID="59565b69-6f84-415f-b765-223943cd28d1" containerName="cleanup" Nov 01 21:15:31.097798 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097558 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59565b69-6f84-415f-b765-223943cd28d1" containerName="cleanup" Nov 01 21:15:31.097798 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097565 4484 cpu_manager.go:414] "containerMap: removing container" podUID="be6565c5-c4bd-4f2c-9d50-0b5359d589e4" containerName="cleanup" Nov 01 21:15:31.097798 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097569 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be6565c5-c4bd-4f2c-9d50-0b5359d589e4" containerName="cleanup" Nov 01 21:15:31.097798 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097576 4484 cpu_manager.go:414] "containerMap: removing container" podUID="9da77fa2-9d96-4a3d-9a1e-2d23a7d8fb0c" containerName="cleanup" Nov 01 21:15:31.097798 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097580 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9da77fa2-9d96-4a3d-9a1e-2d23a7d8fb0c" containerName="cleanup" Nov 01 21:15:31.097798 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097586 4484 cpu_manager.go:414] "containerMap: removing container" podUID="59565b69-6f84-415f-b765-223943cd28d1" containerName="cleanup" Nov 01 21:15:31.097798 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097590 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59565b69-6f84-415f-b765-223943cd28d1" containerName="cleanup" Nov 01 21:15:31.097798 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097598 4484 cpu_manager.go:414] "containerMap: removing container" podUID="009125b3-fc6d-4cbe-aa45-5cf27cc39a2e" containerName="payload-pre-processing" Nov 01 21:15:31.097798 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097602 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="009125b3-fc6d-4cbe-aa45-5cf27cc39a2e" containerName="payload-pre-processing" Nov 01 21:15:31.097798 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097612 4484 cpu_manager.go:414] "containerMap: removing container" podUID="f695676a-20ed-4db3-a4f3-2961bc48be9a" containerName="cleanup" Nov 01 21:15:31.097798 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097616 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f695676a-20ed-4db3-a4f3-2961bc48be9a" containerName="cleanup" Nov 01 21:15:31.097798 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097624 4484 cpu_manager.go:414] "containerMap: removing container" podUID="9464b311-fde8-4875-aafa-ee5c07166a4e" containerName="cleanup" Nov 01 21:15:31.097798 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097628 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9464b311-fde8-4875-aafa-ee5c07166a4e" containerName="cleanup" Nov 01 21:15:31.097798 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097637 4484 cpu_manager.go:414] "containerMap: removing container" podUID="9da77fa2-9d96-4a3d-9a1e-2d23a7d8fb0c" containerName="cleanup" Nov 01 21:15:31.097798 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097656 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9da77fa2-9d96-4a3d-9a1e-2d23a7d8fb0c" containerName="cleanup" Nov 01 21:15:31.097798 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097663 4484 cpu_manager.go:414] "containerMap: removing container" podUID="cff29d06-8657-4d3f-aa13-88d10e0c6532" containerName="cleanup" Nov 01 21:15:31.097798 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097667 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cff29d06-8657-4d3f-aa13-88d10e0c6532" containerName="cleanup" Nov 01 21:15:31.097798 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097673 4484 cpu_manager.go:414] "containerMap: removing container" podUID="f695676a-20ed-4db3-a4f3-2961bc48be9a" containerName="cleanup" Nov 01 21:15:31.097798 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097677 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f695676a-20ed-4db3-a4f3-2961bc48be9a" containerName="cleanup" Nov 01 21:15:31.097798 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097687 4484 cpu_manager.go:414] "containerMap: removing container" podUID="e23f5f45-a08b-4a20-b61e-d33119765592" containerName="istio-proxy" Nov 01 21:15:31.097798 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097691 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e23f5f45-a08b-4a20-b61e-d33119765592" containerName="istio-proxy" Nov 01 21:15:31.097798 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097703 4484 cpu_manager.go:414] "containerMap: removing container" podUID="9da77fa2-9d96-4a3d-9a1e-2d23a7d8fb0c" containerName="cleanup" Nov 01 21:15:31.097798 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097710 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9da77fa2-9d96-4a3d-9a1e-2d23a7d8fb0c" containerName="cleanup" Nov 01 21:15:31.097798 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097720 4484 cpu_manager.go:414] "containerMap: removing container" podUID="425105dd-4b1f-4f03-94a1-121733d0ad62" containerName="cleanup" Nov 01 21:15:31.097798 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097724 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="425105dd-4b1f-4f03-94a1-121733d0ad62" containerName="cleanup" Nov 01 21:15:31.097798 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097730 4484 cpu_manager.go:414] "containerMap: removing container" podUID="9464b311-fde8-4875-aafa-ee5c07166a4e" containerName="cleanup" Nov 01 21:15:31.097798 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097734 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9464b311-fde8-4875-aafa-ee5c07166a4e" containerName="cleanup" Nov 01 21:15:31.097798 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097743 4484 cpu_manager.go:414] "containerMap: removing container" podUID="f411ea1f-0db8-4748-87e9-4ae94a7a12c4" containerName="istio-proxy" Nov 01 21:15:31.097798 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097747 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f411ea1f-0db8-4748-87e9-4ae94a7a12c4" containerName="istio-proxy" Nov 01 21:15:31.097798 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097753 4484 cpu_manager.go:414] "containerMap: removing container" podUID="9464b311-fde8-4875-aafa-ee5c07166a4e" containerName="cleanup" Nov 01 21:15:31.097798 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097758 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9464b311-fde8-4875-aafa-ee5c07166a4e" containerName="cleanup" Nov 01 21:15:31.097798 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097764 4484 cpu_manager.go:414] "containerMap: removing container" podUID="425105dd-4b1f-4f03-94a1-121733d0ad62" containerName="cleanup" Nov 01 21:15:31.097798 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097768 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="425105dd-4b1f-4f03-94a1-121733d0ad62" containerName="cleanup" Nov 01 21:15:31.097798 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097774 4484 cpu_manager.go:414] "containerMap: removing container" podUID="f9a691db-7b44-40aa-8924-1f2b716da33a" containerName="payload-processing" Nov 01 21:15:31.097798 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097778 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9a691db-7b44-40aa-8924-1f2b716da33a" containerName="payload-processing" Nov 01 21:15:31.097798 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097785 4484 cpu_manager.go:414] "containerMap: removing container" podUID="cff29d06-8657-4d3f-aa13-88d10e0c6532" containerName="cleanup" Nov 01 21:15:31.097798 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097789 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cff29d06-8657-4d3f-aa13-88d10e0c6532" containerName="cleanup" Nov 01 21:15:31.097798 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097795 4484 cpu_manager.go:414] "containerMap: removing container" podUID="425105dd-4b1f-4f03-94a1-121733d0ad62" containerName="cleanup" Nov 01 21:15:31.097798 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097799 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="425105dd-4b1f-4f03-94a1-121733d0ad62" containerName="cleanup" Nov 01 21:15:31.097798 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097806 4484 cpu_manager.go:414] "containerMap: removing container" podUID="cff29d06-8657-4d3f-aa13-88d10e0c6532" containerName="cleanup" Nov 01 21:15:31.097798 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097810 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cff29d06-8657-4d3f-aa13-88d10e0c6532" containerName="cleanup" Nov 01 21:15:31.097798 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097818 4484 cpu_manager.go:414] "containerMap: removing container" podUID="cb830eed-0dbe-4080-b6a0-2c3092adcf5b" containerName="cleanup" Nov 01 21:15:31.099142 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097822 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb830eed-0dbe-4080-b6a0-2c3092adcf5b" containerName="cleanup" Nov 01 21:15:31.099142 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097832 4484 cpu_manager.go:414] "containerMap: removing container" podUID="d44ebf49-b772-485f-a982-055f9dd19efc" containerName="cleanup" Nov 01 21:15:31.099142 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097836 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d44ebf49-b772-485f-a982-055f9dd19efc" containerName="cleanup" Nov 01 21:15:31.099142 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097906 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="9da77fa2-9d96-4a3d-9a1e-2d23a7d8fb0c" containerName="cleanup" Nov 01 21:15:31.099142 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097913 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="d44ebf49-b772-485f-a982-055f9dd19efc" containerName="cleanup" Nov 01 21:15:31.099142 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097919 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="425105dd-4b1f-4f03-94a1-121733d0ad62" containerName="cleanup" Nov 01 21:15:31.099142 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097924 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="9464b311-fde8-4875-aafa-ee5c07166a4e" containerName="cleanup" Nov 01 21:15:31.099142 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097929 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="f411ea1f-0db8-4748-87e9-4ae94a7a12c4" containerName="istio-proxy" Nov 01 21:15:31.099142 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097935 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="9464b311-fde8-4875-aafa-ee5c07166a4e" containerName="cleanup" Nov 01 21:15:31.099142 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097941 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="425105dd-4b1f-4f03-94a1-121733d0ad62" containerName="cleanup" Nov 01 21:15:31.099142 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097945 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9a691db-7b44-40aa-8924-1f2b716da33a" containerName="payload-processing" Nov 01 21:15:31.099142 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097951 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="f695676a-20ed-4db3-a4f3-2961bc48be9a" containerName="cleanup" Nov 01 21:15:31.099142 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097955 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb830eed-0dbe-4080-b6a0-2c3092adcf5b" containerName="cleanup" Nov 01 21:15:31.099142 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097961 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="cff29d06-8657-4d3f-aa13-88d10e0c6532" containerName="cleanup" Nov 01 21:15:31.099142 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097966 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="425105dd-4b1f-4f03-94a1-121733d0ad62" containerName="cleanup" Nov 01 21:15:31.099142 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097971 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="cff29d06-8657-4d3f-aa13-88d10e0c6532" containerName="cleanup" Nov 01 21:15:31.099142 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097976 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb830eed-0dbe-4080-b6a0-2c3092adcf5b" containerName="cleanup" Nov 01 21:15:31.099142 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097981 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="be6565c5-c4bd-4f2c-9d50-0b5359d589e4" containerName="cleanup" Nov 01 21:15:31.099142 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.097986 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="d44ebf49-b772-485f-a982-055f9dd19efc" containerName="cleanup" Nov 01 21:15:31.099142 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.098005 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb830eed-0dbe-4080-b6a0-2c3092adcf5b" containerName="cleanup" Nov 01 21:15:31.099142 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.098013 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a48915d-f25c-4aae-bd0d-2714992416d1" containerName="istio-proxy" Nov 01 21:15:31.099142 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.098018 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="d44ebf49-b772-485f-a982-055f9dd19efc" containerName="cleanup" Nov 01 21:15:31.099142 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.098024 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="8da24464-e245-4061-a89b-9e9e9f61036d" containerName="istio-proxy" Nov 01 21:15:31.099142 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.098031 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="59565b69-6f84-415f-b765-223943cd28d1" containerName="cleanup" Nov 01 21:15:31.099142 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.098036 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="be6565c5-c4bd-4f2c-9d50-0b5359d589e4" containerName="cleanup" Nov 01 21:15:31.099142 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.098041 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="9da77fa2-9d96-4a3d-9a1e-2d23a7d8fb0c" containerName="cleanup" Nov 01 21:15:31.099142 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.098046 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="59565b69-6f84-415f-b765-223943cd28d1" containerName="cleanup" Nov 01 21:15:31.099142 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.098052 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="009125b3-fc6d-4cbe-aa45-5cf27cc39a2e" containerName="payload-pre-processing" Nov 01 21:15:31.099142 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.098058 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="59565b69-6f84-415f-b765-223943cd28d1" containerName="cleanup" Nov 01 21:15:31.099142 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.098063 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="f695676a-20ed-4db3-a4f3-2961bc48be9a" containerName="cleanup" Nov 01 21:15:31.099142 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.098067 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="e23f5f45-a08b-4a20-b61e-d33119765592" containerName="istio-proxy" Nov 01 21:15:31.102956 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.102936 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-df700a-t5hcp" Nov 01 21:15:31.158380 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.158343 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-df700a-t5hcp"] Nov 01 21:15:31.198545 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.198513 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a86e09e-82de-4bcc-9611-eead7fdb2ce9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-df700a-t5hcp\" (UID: \"0a86e09e-82de-4bcc-9611-eead7fdb2ce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-df700a-t5hcp" Nov 01 21:15:31.198739 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.198693 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn9zb\" (UniqueName: \"kubernetes.io/projected/0a86e09e-82de-4bcc-9611-eead7fdb2ce9-kube-api-access-cn9zb\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-df700a-t5hcp\" (UID: \"0a86e09e-82de-4bcc-9611-eead7fdb2ce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-df700a-t5hcp" Nov 01 21:15:31.198809 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.198791 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a86e09e-82de-4bcc-9611-eead7fdb2ce9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-df700a-t5hcp\" (UID: \"0a86e09e-82de-4bcc-9611-eead7fdb2ce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-df700a-t5hcp" Nov 01 21:15:31.206053 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.206025 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:15:31.206125 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.206082 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:15:31.207335 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.207319 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:15:31.207396 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.207345 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:15:31.299731 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.299689 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn9zb\" (UniqueName: \"kubernetes.io/projected/0a86e09e-82de-4bcc-9611-eead7fdb2ce9-kube-api-access-cn9zb\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-df700a-t5hcp\" (UID: \"0a86e09e-82de-4bcc-9611-eead7fdb2ce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-df700a-t5hcp" Nov 01 21:15:31.299948 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.299789 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a86e09e-82de-4bcc-9611-eead7fdb2ce9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-df700a-t5hcp\" (UID: \"0a86e09e-82de-4bcc-9611-eead7fdb2ce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-df700a-t5hcp" Nov 01 21:15:31.299948 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.299823 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a86e09e-82de-4bcc-9611-eead7fdb2ce9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-df700a-t5hcp\" (UID: \"0a86e09e-82de-4bcc-9611-eead7fdb2ce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-df700a-t5hcp" Nov 01 21:15:31.300583 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.300539 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a86e09e-82de-4bcc-9611-eead7fdb2ce9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-df700a-t5hcp\" (UID: \"0a86e09e-82de-4bcc-9611-eead7fdb2ce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-df700a-t5hcp" Nov 01 21:15:31.302538 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.302516 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a86e09e-82de-4bcc-9611-eead7fdb2ce9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-df700a-t5hcp\" (UID: \"0a86e09e-82de-4bcc-9611-eead7fdb2ce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-df700a-t5hcp" Nov 01 21:15:31.360034 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.359928 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn9zb\" (UniqueName: \"kubernetes.io/projected/0a86e09e-82de-4bcc-9611-eead7fdb2ce9-kube-api-access-cn9zb\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-df700a-t5hcp\" (UID: \"0a86e09e-82de-4bcc-9611-eead7fdb2ce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-df700a-t5hcp" Nov 01 21:15:31.416856 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.416822 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-df700a-t5hcp" Nov 01 21:15:31.790924 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:31.790894 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-df700a-t5hcp"] Nov 01 21:15:31.792299 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:15:31.792269 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a86e09e_82de_4bcc_9611_eead7fdb2ce9.slice/crio-6ceea352e7e4f4556e427e307a298cec8e9d4d5bc0c26e816bcf21f9a384153b WatchSource:0}: Error finding container 6ceea352e7e4f4556e427e307a298cec8e9d4d5bc0c26e816bcf21f9a384153b: Status 404 returned error can't find the container with id 6ceea352e7e4f4556e427e307a298cec8e9d4d5bc0c26e816bcf21f9a384153b Nov 01 21:15:32.132885 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:32.132856 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk"] Nov 01 21:15:32.133258 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:32.133080 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk" podUID="8a5b2e73-9f47-47aa-9ed7-c75d17b1f658" containerName="istio-proxy" containerID="cri-o://40529fb5a9715b9494e878075c25a0959062796221255c40dfd8cac30095adee" gracePeriod=30 Nov 01 21:15:32.198872 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:32.198815 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-df700a-t5hcp" event={"ID":"0a86e09e-82de-4bcc-9611-eead7fdb2ce9","Type":"ContainerStarted","Data":"8344400d481c0a87bc25f773a889fbe380d00e8ee9d2c3c84d8a8a3cba54c1b7"} Nov 01 21:15:32.199103 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:32.198892 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-df700a-t5hcp" event={"ID":"0a86e09e-82de-4bcc-9611-eead7fdb2ce9","Type":"ContainerStarted","Data":"6ceea352e7e4f4556e427e307a298cec8e9d4d5bc0c26e816bcf21f9a384153b"} Nov 01 21:15:32.202357 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:32.202312 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:32.377736 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:32.377637 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-df700a-t5hcp" podStartSLOduration=1.377622676 podStartE2EDuration="1.377622676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:15:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:15:32.376196667 +0000 UTC m=+2683.153836802" watchObservedRunningTime="2026-11-01 21:15:32.377622676 +0000 UTC m=+2683.155262835" Nov 01 21:15:32.650343 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:32.650238 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:33.098341 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:33.098305 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-208f2b-jdcwz"] Nov 01 21:15:33.098893 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:33.098879 4484 cpu_manager.go:414] "containerMap: removing container" podUID="cb830eed-0dbe-4080-b6a0-2c3092adcf5b" containerName="cleanup" Nov 01 21:15:33.098955 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:33.098895 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb830eed-0dbe-4080-b6a0-2c3092adcf5b" containerName="cleanup" Nov 01 21:15:33.098955 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:33.098911 4484 cpu_manager.go:414] "containerMap: removing container" podUID="d44ebf49-b772-485f-a982-055f9dd19efc" containerName="cleanup" Nov 01 21:15:33.098955 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:33.098917 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d44ebf49-b772-485f-a982-055f9dd19efc" containerName="cleanup" Nov 01 21:15:33.098955 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:33.098929 4484 cpu_manager.go:414] "containerMap: removing container" podUID="be6565c5-c4bd-4f2c-9d50-0b5359d589e4" containerName="cleanup" Nov 01 21:15:33.098955 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:33.098934 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be6565c5-c4bd-4f2c-9d50-0b5359d589e4" containerName="cleanup" Nov 01 21:15:33.098955 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:33.098941 4484 cpu_manager.go:414] "containerMap: removing container" podUID="59565b69-6f84-415f-b765-223943cd28d1" containerName="cleanup" Nov 01 21:15:33.098955 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:33.098946 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59565b69-6f84-415f-b765-223943cd28d1" containerName="cleanup" Nov 01 21:15:33.099197 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:33.098961 4484 cpu_manager.go:414] "containerMap: removing container" podUID="f695676a-20ed-4db3-a4f3-2961bc48be9a" containerName="cleanup" Nov 01 21:15:33.099197 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:33.098966 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f695676a-20ed-4db3-a4f3-2961bc48be9a" containerName="cleanup" Nov 01 21:15:33.099197 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:33.099073 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="f695676a-20ed-4db3-a4f3-2961bc48be9a" containerName="cleanup" Nov 01 21:15:33.099197 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:33.099082 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="9da77fa2-9d96-4a3d-9a1e-2d23a7d8fb0c" containerName="cleanup" Nov 01 21:15:33.099197 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:33.099090 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="9464b311-fde8-4875-aafa-ee5c07166a4e" containerName="cleanup" Nov 01 21:15:33.099197 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:33.099097 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="cff29d06-8657-4d3f-aa13-88d10e0c6532" containerName="cleanup" Nov 01 21:15:33.102390 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:33.102373 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-208f2b-jdcwz" Nov 01 21:15:33.158699 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:33.158671 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-208f2b-jdcwz"] Nov 01 21:15:33.210466 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:33.210431 4484 generic.go:356] "Generic (PLEG): container finished" podID="0a86e09e-82de-4bcc-9611-eead7fdb2ce9" containerID="8344400d481c0a87bc25f773a889fbe380d00e8ee9d2c3c84d8a8a3cba54c1b7" exitCode=0 Nov 01 21:15:33.210663 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:33.210521 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-df700a-t5hcp" event={"ID":"0a86e09e-82de-4bcc-9611-eead7fdb2ce9","Type":"ContainerDied","Data":"8344400d481c0a87bc25f773a889fbe380d00e8ee9d2c3c84d8a8a3cba54c1b7"} Nov 01 21:15:33.228081 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:33.228049 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxv6r\" (UniqueName: \"kubernetes.io/projected/6fa4342c-78ad-493c-8709-f7e5b2ceaa3b-kube-api-access-rxv6r\") pod \"maas-api-revoke-keys-e2e-worker-w0-208f2b-jdcwz\" (UID: \"6fa4342c-78ad-493c-8709-f7e5b2ceaa3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-208f2b-jdcwz" Nov 01 21:15:33.228254 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:33.228234 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fa4342c-78ad-493c-8709-f7e5b2ceaa3b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-208f2b-jdcwz\" (UID: \"6fa4342c-78ad-493c-8709-f7e5b2ceaa3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-208f2b-jdcwz" Nov 01 21:15:33.228306 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:33.228275 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fa4342c-78ad-493c-8709-f7e5b2ceaa3b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-208f2b-jdcwz\" (UID: \"6fa4342c-78ad-493c-8709-f7e5b2ceaa3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-208f2b-jdcwz" Nov 01 21:15:33.329756 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:33.329719 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fa4342c-78ad-493c-8709-f7e5b2ceaa3b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-208f2b-jdcwz\" (UID: \"6fa4342c-78ad-493c-8709-f7e5b2ceaa3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-208f2b-jdcwz" Nov 01 21:15:33.329944 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:33.329780 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fa4342c-78ad-493c-8709-f7e5b2ceaa3b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-208f2b-jdcwz\" (UID: \"6fa4342c-78ad-493c-8709-f7e5b2ceaa3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-208f2b-jdcwz" Nov 01 21:15:33.329944 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:33.329875 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxv6r\" (UniqueName: \"kubernetes.io/projected/6fa4342c-78ad-493c-8709-f7e5b2ceaa3b-kube-api-access-rxv6r\") pod \"maas-api-revoke-keys-e2e-worker-w0-208f2b-jdcwz\" (UID: \"6fa4342c-78ad-493c-8709-f7e5b2ceaa3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-208f2b-jdcwz" Nov 01 21:15:33.330551 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:33.330523 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fa4342c-78ad-493c-8709-f7e5b2ceaa3b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-208f2b-jdcwz\" (UID: \"6fa4342c-78ad-493c-8709-f7e5b2ceaa3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-208f2b-jdcwz" Nov 01 21:15:33.335283 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:33.332401 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fa4342c-78ad-493c-8709-f7e5b2ceaa3b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-208f2b-jdcwz\" (UID: \"6fa4342c-78ad-493c-8709-f7e5b2ceaa3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-208f2b-jdcwz" Nov 01 21:15:33.387611 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:33.387541 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxv6r\" (UniqueName: \"kubernetes.io/projected/6fa4342c-78ad-493c-8709-f7e5b2ceaa3b-kube-api-access-rxv6r\") pod \"maas-api-revoke-keys-e2e-worker-w0-208f2b-jdcwz\" (UID: \"6fa4342c-78ad-493c-8709-f7e5b2ceaa3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-208f2b-jdcwz" Nov 01 21:15:33.415663 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:33.415622 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-208f2b-jdcwz" Nov 01 21:15:33.789570 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:33.789538 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-208f2b-jdcwz"] Nov 01 21:15:33.790407 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:15:33.790382 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fa4342c_78ad_493c_8709_f7e5b2ceaa3b.slice/crio-4bec2b93cf79e40c24b79bfd3b809f1bfee5f1120c50f6903d675384aac20a95 WatchSource:0}: Error finding container 4bec2b93cf79e40c24b79bfd3b809f1bfee5f1120c50f6903d675384aac20a95: Status 404 returned error can't find the container with id 4bec2b93cf79e40c24b79bfd3b809f1bfee5f1120c50f6903d675384aac20a95 Nov 01 21:15:34.219731 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:34.219692 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-208f2b-jdcwz" event={"ID":"6fa4342c-78ad-493c-8709-f7e5b2ceaa3b","Type":"ContainerStarted","Data":"31aa46d356be0b2610bf24d726b7f2d9214181a100f4c127131be981cad380cc"} Nov 01 21:15:34.220147 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:34.219749 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-208f2b-jdcwz" event={"ID":"6fa4342c-78ad-493c-8709-f7e5b2ceaa3b","Type":"ContainerStarted","Data":"4bec2b93cf79e40c24b79bfd3b809f1bfee5f1120c50f6903d675384aac20a95"} Nov 01 21:15:34.346118 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:34.336841 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-208f2b-jdcwz" podStartSLOduration=1.3368222969999999 podStartE2EDuration="1.336822297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:15:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:15:34.333579528 +0000 UTC m=+2685.111219664" watchObservedRunningTime="2026-11-01 21:15:34.336822297 +0000 UTC m=+2685.114462434" Nov 01 21:15:34.398485 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:34.398457 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-df700a-t5hcp" Nov 01 21:15:34.544395 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:34.544357 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a86e09e-82de-4bcc-9611-eead7fdb2ce9-kube-api-access-cn9zb\" (UniqueName: \"kubernetes.io/projected/0a86e09e-82de-4bcc-9611-eead7fdb2ce9-kube-api-access-cn9zb\") pod \"0a86e09e-82de-4bcc-9611-eead7fdb2ce9\" (UID: \"0a86e09e-82de-4bcc-9611-eead7fdb2ce9\") " Nov 01 21:15:34.544567 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:34.544416 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a86e09e-82de-4bcc-9611-eead7fdb2ce9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a86e09e-82de-4bcc-9611-eead7fdb2ce9-maas-api-xks-ca\") pod \"0a86e09e-82de-4bcc-9611-eead7fdb2ce9\" (UID: \"0a86e09e-82de-4bcc-9611-eead7fdb2ce9\") " Nov 01 21:15:34.544567 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:34.544512 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a86e09e-82de-4bcc-9611-eead7fdb2ce9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a86e09e-82de-4bcc-9611-eead7fdb2ce9-maas-api-openshift-ca\") pod \"0a86e09e-82de-4bcc-9611-eead7fdb2ce9\" (UID: \"0a86e09e-82de-4bcc-9611-eead7fdb2ce9\") " Nov 01 21:15:34.544885 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:34.544848 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a86e09e-82de-4bcc-9611-eead7fdb2ce9-maas-api-openshift-ca" pod "0a86e09e-82de-4bcc-9611-eead7fdb2ce9" (UID: "0a86e09e-82de-4bcc-9611-eead7fdb2ce9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:15:34.545047 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:34.545027 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a86e09e-82de-4bcc-9611-eead7fdb2ce9-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:34.546590 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:34.546556 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a86e09e-82de-4bcc-9611-eead7fdb2ce9-maas-api-xks-ca" pod "0a86e09e-82de-4bcc-9611-eead7fdb2ce9" (UID: "0a86e09e-82de-4bcc-9611-eead7fdb2ce9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:15:34.546715 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:34.546670 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a86e09e-82de-4bcc-9611-eead7fdb2ce9-kube-api-access-cn9zb" pod "0a86e09e-82de-4bcc-9611-eead7fdb2ce9" (UID: "0a86e09e-82de-4bcc-9611-eead7fdb2ce9"). InnerVolumeSpecName "kube-api-access-cn9zb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:15:34.646489 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:34.646410 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cn9zb\" (UniqueName: \"kubernetes.io/projected/0a86e09e-82de-4bcc-9611-eead7fdb2ce9-kube-api-access-cn9zb\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:34.646489 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:34.646441 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a86e09e-82de-4bcc-9611-eead7fdb2ce9-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:35.014865 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:35.014828 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:35.228608 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:35.228579 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-df700a-t5hcp" Nov 01 21:15:35.229034 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:35.228578 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-df700a-t5hcp" event={"ID":"0a86e09e-82de-4bcc-9611-eead7fdb2ce9","Type":"ContainerDied","Data":"6ceea352e7e4f4556e427e307a298cec8e9d4d5bc0c26e816bcf21f9a384153b"} Nov 01 21:15:35.229034 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:35.228706 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ceea352e7e4f4556e427e307a298cec8e9d4d5bc0c26e816bcf21f9a384153b" Nov 01 21:15:35.230110 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:35.230080 4484 generic.go:356] "Generic (PLEG): container finished" podID="6fa4342c-78ad-493c-8709-f7e5b2ceaa3b" containerID="31aa46d356be0b2610bf24d726b7f2d9214181a100f4c127131be981cad380cc" exitCode=0 Nov 01 21:15:35.230253 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:35.230167 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-208f2b-jdcwz" event={"ID":"6fa4342c-78ad-493c-8709-f7e5b2ceaa3b","Type":"ContainerDied","Data":"31aa46d356be0b2610bf24d726b7f2d9214181a100f4c127131be981cad380cc"} Nov 01 21:15:36.391559 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:36.391529 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-208f2b-jdcwz" Nov 01 21:15:36.568874 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:36.568838 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fa4342c-78ad-493c-8709-f7e5b2ceaa3b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fa4342c-78ad-493c-8709-f7e5b2ceaa3b-maas-api-openshift-ca\") pod \"6fa4342c-78ad-493c-8709-f7e5b2ceaa3b\" (UID: \"6fa4342c-78ad-493c-8709-f7e5b2ceaa3b\") " Nov 01 21:15:36.569094 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:36.568895 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fa4342c-78ad-493c-8709-f7e5b2ceaa3b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fa4342c-78ad-493c-8709-f7e5b2ceaa3b-maas-api-xks-ca\") pod \"6fa4342c-78ad-493c-8709-f7e5b2ceaa3b\" (UID: \"6fa4342c-78ad-493c-8709-f7e5b2ceaa3b\") " Nov 01 21:15:36.569094 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:36.568946 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fa4342c-78ad-493c-8709-f7e5b2ceaa3b-kube-api-access-rxv6r\" (UniqueName: \"kubernetes.io/projected/6fa4342c-78ad-493c-8709-f7e5b2ceaa3b-kube-api-access-rxv6r\") pod \"6fa4342c-78ad-493c-8709-f7e5b2ceaa3b\" (UID: \"6fa4342c-78ad-493c-8709-f7e5b2ceaa3b\") " Nov 01 21:15:36.569348 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:36.569324 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fa4342c-78ad-493c-8709-f7e5b2ceaa3b-maas-api-openshift-ca" pod "6fa4342c-78ad-493c-8709-f7e5b2ceaa3b" (UID: "6fa4342c-78ad-493c-8709-f7e5b2ceaa3b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:15:36.569469 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:36.569453 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fa4342c-78ad-493c-8709-f7e5b2ceaa3b-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:36.571130 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:36.571108 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fa4342c-78ad-493c-8709-f7e5b2ceaa3b-maas-api-xks-ca" pod "6fa4342c-78ad-493c-8709-f7e5b2ceaa3b" (UID: "6fa4342c-78ad-493c-8709-f7e5b2ceaa3b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:15:36.571232 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:36.571216 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fa4342c-78ad-493c-8709-f7e5b2ceaa3b-kube-api-access-rxv6r" pod "6fa4342c-78ad-493c-8709-f7e5b2ceaa3b" (UID: "6fa4342c-78ad-493c-8709-f7e5b2ceaa3b"). InnerVolumeSpecName "kube-api-access-rxv6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:15:36.595930 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:36.595894 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:36.670679 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:36.670630 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fa4342c-78ad-493c-8709-f7e5b2ceaa3b-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:36.670679 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:36.670677 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxv6r\" (UniqueName: \"kubernetes.io/projected/6fa4342c-78ad-493c-8709-f7e5b2ceaa3b-kube-api-access-rxv6r\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:36.968206 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:36.968125 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:37.137876 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:37.137836 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:37.179116 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:37.179080 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a41242-dgcgt"] Nov 01 21:15:37.182119 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:37.182097 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a41242-dgcgt"] Nov 01 21:15:37.245666 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:37.245641 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-208f2b-jdcwz" Nov 01 21:15:37.245821 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:37.245676 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-208f2b-jdcwz" event={"ID":"6fa4342c-78ad-493c-8709-f7e5b2ceaa3b","Type":"ContainerDied","Data":"4bec2b93cf79e40c24b79bfd3b809f1bfee5f1120c50f6903d675384aac20a95"} Nov 01 21:15:37.245821 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:37.245726 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4bec2b93cf79e40c24b79bfd3b809f1bfee5f1120c50f6903d675384aac20a95" Nov 01 21:15:37.375943 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:37.375916 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c259f9d2-8a0f-411d-bc9d-61644916daee" path="/var/lib/kubelet/pods/c259f9d2-8a0f-411d-bc9d-61644916daee/volumes" Nov 01 21:15:37.403627 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:37.403601 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk" Nov 01 21:15:37.580823 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:37.580741 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-credential-socket\") pod \"8a5b2e73-9f47-47aa-9ed7-c75d17b1f658\" (UID: \"8a5b2e73-9f47-47aa-9ed7-c75d17b1f658\") " Nov 01 21:15:37.580975 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:37.580835 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-workload-socket\") pod \"8a5b2e73-9f47-47aa-9ed7-c75d17b1f658\" (UID: \"8a5b2e73-9f47-47aa-9ed7-c75d17b1f658\") " Nov 01 21:15:37.580975 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:37.580872 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-istio-data\") pod \"8a5b2e73-9f47-47aa-9ed7-c75d17b1f658\" (UID: \"8a5b2e73-9f47-47aa-9ed7-c75d17b1f658\") " Nov 01 21:15:37.580975 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:37.580910 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-istio-podinfo\") pod \"8a5b2e73-9f47-47aa-9ed7-c75d17b1f658\" (UID: \"8a5b2e73-9f47-47aa-9ed7-c75d17b1f658\") " Nov 01 21:15:37.580975 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:37.580935 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-istio-envoy\") pod \"8a5b2e73-9f47-47aa-9ed7-c75d17b1f658\" (UID: \"8a5b2e73-9f47-47aa-9ed7-c75d17b1f658\") " Nov 01 21:15:37.580975 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:37.580957 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-kube-api-access-rhcp5\" (UniqueName: \"kubernetes.io/projected/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-kube-api-access-rhcp5\") pod \"8a5b2e73-9f47-47aa-9ed7-c75d17b1f658\" (UID: \"8a5b2e73-9f47-47aa-9ed7-c75d17b1f658\") " Nov 01 21:15:37.581273 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:37.580979 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-workload-certs\") pod \"8a5b2e73-9f47-47aa-9ed7-c75d17b1f658\" (UID: \"8a5b2e73-9f47-47aa-9ed7-c75d17b1f658\") " Nov 01 21:15:37.581273 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:37.581051 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-istio-token\" (UniqueName: \"kubernetes.io/projected/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-istio-token\") pod \"8a5b2e73-9f47-47aa-9ed7-c75d17b1f658\" (UID: \"8a5b2e73-9f47-47aa-9ed7-c75d17b1f658\") " Nov 01 21:15:37.581273 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:37.581097 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-istiod-ca-cert\") pod \"8a5b2e73-9f47-47aa-9ed7-c75d17b1f658\" (UID: \"8a5b2e73-9f47-47aa-9ed7-c75d17b1f658\") " Nov 01 21:15:37.581273 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:37.581105 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-credential-socket" pod "8a5b2e73-9f47-47aa-9ed7-c75d17b1f658" (UID: "8a5b2e73-9f47-47aa-9ed7-c75d17b1f658"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:15:37.581273 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:37.581174 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-workload-socket" pod "8a5b2e73-9f47-47aa-9ed7-c75d17b1f658" (UID: "8a5b2e73-9f47-47aa-9ed7-c75d17b1f658"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:15:37.581525 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:37.581290 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-istio-data" pod "8a5b2e73-9f47-47aa-9ed7-c75d17b1f658" (UID: "8a5b2e73-9f47-47aa-9ed7-c75d17b1f658"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:15:37.581525 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:37.581327 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-workload-certs" pod "8a5b2e73-9f47-47aa-9ed7-c75d17b1f658" (UID: "8a5b2e73-9f47-47aa-9ed7-c75d17b1f658"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:15:37.581637 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:37.581536 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-istiod-ca-cert" pod "8a5b2e73-9f47-47aa-9ed7-c75d17b1f658" (UID: "8a5b2e73-9f47-47aa-9ed7-c75d17b1f658"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:15:37.581637 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:37.581623 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:37.581727 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:37.581646 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:37.581727 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:37.581661 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:37.581727 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:37.581675 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:37.583311 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:37.583281 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-kube-api-access-rhcp5" pod "8a5b2e73-9f47-47aa-9ed7-c75d17b1f658" (UID: "8a5b2e73-9f47-47aa-9ed7-c75d17b1f658"). InnerVolumeSpecName "kube-api-access-rhcp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:15:37.583439 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:37.583374 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-istio-podinfo" pod "8a5b2e73-9f47-47aa-9ed7-c75d17b1f658" (UID: "8a5b2e73-9f47-47aa-9ed7-c75d17b1f658"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:15:37.583671 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:37.583654 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-istio-envoy" pod "8a5b2e73-9f47-47aa-9ed7-c75d17b1f658" (UID: "8a5b2e73-9f47-47aa-9ed7-c75d17b1f658"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:15:37.583734 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:37.583681 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-istio-token" pod "8a5b2e73-9f47-47aa-9ed7-c75d17b1f658" (UID: "8a5b2e73-9f47-47aa-9ed7-c75d17b1f658"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:15:37.603231 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:37.603203 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:37.682704 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:37.682674 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:37.682704 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:37.682700 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:37.682704 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:37.682710 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhcp5\" (UniqueName: \"kubernetes.io/projected/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-kube-api-access-rhcp5\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:37.682914 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:37.682720 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:37.682914 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:37.682728 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:37.762165 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:37.762126 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:38.254902 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:38.254871 4484 generic.go:356] "Generic (PLEG): container finished" podID="8a5b2e73-9f47-47aa-9ed7-c75d17b1f658" containerID="40529fb5a9715b9494e878075c25a0959062796221255c40dfd8cac30095adee" exitCode=0 Nov 01 21:15:38.255100 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:38.254941 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk" Nov 01 21:15:38.255100 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:38.254955 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk" event={"ID":"8a5b2e73-9f47-47aa-9ed7-c75d17b1f658","Type":"ContainerDied","Data":"40529fb5a9715b9494e878075c25a0959062796221255c40dfd8cac30095adee"} Nov 01 21:15:38.255100 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:38.255064 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk" event={"ID":"8a5b2e73-9f47-47aa-9ed7-c75d17b1f658","Type":"ContainerDied","Data":"eaec53cd0ebea9ebd6e54848169ed6803a78420901f55e631b0514fe70a9416a"} Nov 01 21:15:38.255100 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:38.255090 4484 scope.go:122] "RemoveContainer" containerID="40529fb5a9715b9494e878075c25a0959062796221255c40dfd8cac30095adee" Nov 01 21:15:38.266029 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:38.265991 4484 scope.go:122] "RemoveContainer" containerID="4696272076f0d71ea68d0e91f545e2f24273c5a522684fc28be907d79aab81f7" Nov 01 21:15:38.275603 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:38.275581 4484 scope.go:122] "RemoveContainer" containerID="40529fb5a9715b9494e878075c25a0959062796221255c40dfd8cac30095adee" Nov 01 21:15:38.275890 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:15:38.275870 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40529fb5a9715b9494e878075c25a0959062796221255c40dfd8cac30095adee\": container with ID starting with 40529fb5a9715b9494e878075c25a0959062796221255c40dfd8cac30095adee not found: ID does not exist" containerID="40529fb5a9715b9494e878075c25a0959062796221255c40dfd8cac30095adee" Nov 01 21:15:38.275948 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:38.275899 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40529fb5a9715b9494e878075c25a0959062796221255c40dfd8cac30095adee"} err="failed to get container status \"40529fb5a9715b9494e878075c25a0959062796221255c40dfd8cac30095adee\": rpc error: code = NotFound desc = could not find container \"40529fb5a9715b9494e878075c25a0959062796221255c40dfd8cac30095adee\": container with ID starting with 40529fb5a9715b9494e878075c25a0959062796221255c40dfd8cac30095adee not found: ID does not exist" Nov 01 21:15:38.275948 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:38.275916 4484 scope.go:122] "RemoveContainer" containerID="4696272076f0d71ea68d0e91f545e2f24273c5a522684fc28be907d79aab81f7" Nov 01 21:15:38.276310 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:15:38.276254 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4696272076f0d71ea68d0e91f545e2f24273c5a522684fc28be907d79aab81f7\": container with ID starting with 4696272076f0d71ea68d0e91f545e2f24273c5a522684fc28be907d79aab81f7 not found: ID does not exist" containerID="4696272076f0d71ea68d0e91f545e2f24273c5a522684fc28be907d79aab81f7" Nov 01 21:15:38.276412 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:38.276318 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4696272076f0d71ea68d0e91f545e2f24273c5a522684fc28be907d79aab81f7"} err="failed to get container status \"4696272076f0d71ea68d0e91f545e2f24273c5a522684fc28be907d79aab81f7\": rpc error: code = NotFound desc = could not find container \"4696272076f0d71ea68d0e91f545e2f24273c5a522684fc28be907d79aab81f7\": container with ID starting with 4696272076f0d71ea68d0e91f545e2f24273c5a522684fc28be907d79aab81f7 not found: ID does not exist" Nov 01 21:15:38.425528 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:38.425492 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk"] Nov 01 21:15:38.476585 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:38.476553 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-670df9-openshift-default-6c5f7449f5-zvvdk"] Nov 01 21:15:39.371197 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:39.371163 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a5b2e73-9f47-47aa-9ed7-c75d17b1f658" path="/var/lib/kubelet/pods/8a5b2e73-9f47-47aa-9ed7-c75d17b1f658/volumes" Nov 01 21:15:41.839552 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:41.839521 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:42.619224 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:42.619188 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-df700a-788d57d789-n6jlx"] Nov 01 21:15:42.725015 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:42.724970 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-df700a-789b6c9547-g9ps7"] Nov 01 21:15:43.010020 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:43.009958 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:43.264765 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:43.264661 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:43.712964 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:43.712876 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:43.872177 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:43.872128 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:43.965802 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:43.965709 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2"] Nov 01 21:15:43.966037 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:43.966009 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2" podUID="297fe96b-76ad-499e-b9b7-f536d1851712" containerName="istio-proxy" containerID="cri-o://c5598745e8a990a6fa1b4835b95fdb9aae63eeffbdac8f7b48ea236ade0a472f" gracePeriod=30 Nov 01 21:15:44.039083 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:44.039030 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:44.475849 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:44.475820 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:44.738898 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:44.738859 4484 patch_prober.go:29] interesting pod/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 21:15:44.739092 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:44.738928 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2" podUID="297fe96b-76ad-499e-b9b7-f536d1851712" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 21:15:44.941124 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:44.941088 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1a1e9e-p4bw8"] Nov 01 21:15:44.941609 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:44.941597 4484 cpu_manager.go:414] "containerMap: removing container" podUID="6fa4342c-78ad-493c-8709-f7e5b2ceaa3b" containerName="revoke-keys" Nov 01 21:15:44.941662 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:44.941610 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fa4342c-78ad-493c-8709-f7e5b2ceaa3b" containerName="revoke-keys" Nov 01 21:15:44.941662 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:44.941626 4484 cpu_manager.go:414] "containerMap: removing container" podUID="8a5b2e73-9f47-47aa-9ed7-c75d17b1f658" containerName="istio-proxy" Nov 01 21:15:44.941662 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:44.941631 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a5b2e73-9f47-47aa-9ed7-c75d17b1f658" containerName="istio-proxy" Nov 01 21:15:44.941662 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:44.941655 4484 cpu_manager.go:414] "containerMap: removing container" podUID="0a86e09e-82de-4bcc-9611-eead7fdb2ce9" containerName="revoke-keys" Nov 01 21:15:44.941662 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:44.941660 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a86e09e-82de-4bcc-9611-eead7fdb2ce9" containerName="revoke-keys" Nov 01 21:15:44.941847 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:44.941726 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fa4342c-78ad-493c-8709-f7e5b2ceaa3b" containerName="revoke-keys" Nov 01 21:15:44.941847 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:44.941737 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a5b2e73-9f47-47aa-9ed7-c75d17b1f658" containerName="istio-proxy" Nov 01 21:15:44.941847 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:44.941746 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a5b2e73-9f47-47aa-9ed7-c75d17b1f658" containerName="istio-proxy" Nov 01 21:15:44.941847 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:44.941753 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a86e09e-82de-4bcc-9611-eead7fdb2ce9" containerName="revoke-keys" Nov 01 21:15:44.946597 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:44.946579 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1a1e9e-p4bw8" Nov 01 21:15:44.962860 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:44.962835 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqkk5\" (UniqueName: \"kubernetes.io/projected/395ed984-c93b-415a-8f5b-72c79c4e4f1a-kube-api-access-sqkk5\") pod \"maas-api-revoke-keys-e2e-worker-w4-1a1e9e-p4bw8\" (UID: \"395ed984-c93b-415a-8f5b-72c79c4e4f1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1a1e9e-p4bw8" Nov 01 21:15:44.962987 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:44.962956 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/395ed984-c93b-415a-8f5b-72c79c4e4f1a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-1a1e9e-p4bw8\" (UID: \"395ed984-c93b-415a-8f5b-72c79c4e4f1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1a1e9e-p4bw8" Nov 01 21:15:44.963053 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:44.962987 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/395ed984-c93b-415a-8f5b-72c79c4e4f1a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-1a1e9e-p4bw8\" (UID: \"395ed984-c93b-415a-8f5b-72c79c4e4f1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1a1e9e-p4bw8" Nov 01 21:15:45.002512 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:45.002442 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1a1e9e-p4bw8"] Nov 01 21:15:45.049515 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:45.049478 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:15:45.049515 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:45.049482 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:15:45.049954 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:45.049521 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:15:45.049954 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:45.049570 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:15:45.063840 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:45.063811 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqkk5\" (UniqueName: \"kubernetes.io/projected/395ed984-c93b-415a-8f5b-72c79c4e4f1a-kube-api-access-sqkk5\") pod \"maas-api-revoke-keys-e2e-worker-w4-1a1e9e-p4bw8\" (UID: \"395ed984-c93b-415a-8f5b-72c79c4e4f1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1a1e9e-p4bw8" Nov 01 21:15:45.064046 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:45.063916 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/395ed984-c93b-415a-8f5b-72c79c4e4f1a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-1a1e9e-p4bw8\" (UID: \"395ed984-c93b-415a-8f5b-72c79c4e4f1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1a1e9e-p4bw8" Nov 01 21:15:45.064046 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:45.063945 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/395ed984-c93b-415a-8f5b-72c79c4e4f1a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-1a1e9e-p4bw8\" (UID: \"395ed984-c93b-415a-8f5b-72c79c4e4f1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1a1e9e-p4bw8" Nov 01 21:15:45.064657 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:45.064638 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/395ed984-c93b-415a-8f5b-72c79c4e4f1a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-1a1e9e-p4bw8\" (UID: \"395ed984-c93b-415a-8f5b-72c79c4e4f1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1a1e9e-p4bw8" Nov 01 21:15:45.066293 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:45.066263 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/395ed984-c93b-415a-8f5b-72c79c4e4f1a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-1a1e9e-p4bw8\" (UID: \"395ed984-c93b-415a-8f5b-72c79c4e4f1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1a1e9e-p4bw8" Nov 01 21:15:45.121249 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:45.121209 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqkk5\" (UniqueName: \"kubernetes.io/projected/395ed984-c93b-415a-8f5b-72c79c4e4f1a-kube-api-access-sqkk5\") pod \"maas-api-revoke-keys-e2e-worker-w4-1a1e9e-p4bw8\" (UID: \"395ed984-c93b-415a-8f5b-72c79c4e4f1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1a1e9e-p4bw8" Nov 01 21:15:45.259888 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:45.259795 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1a1e9e-p4bw8" Nov 01 21:15:45.633888 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:45.633864 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1a1e9e-p4bw8"] Nov 01 21:15:45.635450 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:15:45.635418 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod395ed984_c93b_415a_8f5b_72c79c4e4f1a.slice/crio-1ac8caa545de2ba0432a47af7e47892455d9b6c6e7a6192939afce7d58478176 WatchSource:0}: Error finding container 1ac8caa545de2ba0432a47af7e47892455d9b6c6e7a6192939afce7d58478176: Status 404 returned error can't find the container with id 1ac8caa545de2ba0432a47af7e47892455d9b6c6e7a6192939afce7d58478176 Nov 01 21:15:45.767091 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:15:45.767051 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:15:46.222639 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:46.222543 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:46.341609 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:46.341571 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1a1e9e-p4bw8" event={"ID":"395ed984-c93b-415a-8f5b-72c79c4e4f1a","Type":"ContainerStarted","Data":"e57aabf609e00da2996a6d6a9cd3cd110e72dd3626958c81b2d0a395e0030d36"} Nov 01 21:15:46.341817 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:46.341625 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1a1e9e-p4bw8" event={"ID":"395ed984-c93b-415a-8f5b-72c79c4e4f1a","Type":"ContainerStarted","Data":"1ac8caa545de2ba0432a47af7e47892455d9b6c6e7a6192939afce7d58478176"} Nov 01 21:15:46.457380 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:46.457317 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1a1e9e-p4bw8" podStartSLOduration=2.45730411 podStartE2EDuration="2.45730411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:15:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:15:46.4565874 +0000 UTC m=+2697.234227555" watchObservedRunningTime="2026-11-01 21:15:46.45730411 +0000 UTC m=+2697.234944244" Nov 01 21:15:46.510366 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:46.510332 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-208f2b-7fb7755ddc-lx76l"] Nov 01 21:15:46.576625 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:46.576590 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-208f2b-76b7b8c9f-fxd8f"] Nov 01 21:15:46.857352 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:46.857277 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:47.015787 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:47.015756 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:47.350406 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:47.350375 4484 generic.go:356] "Generic (PLEG): container finished" podID="395ed984-c93b-415a-8f5b-72c79c4e4f1a" containerID="e57aabf609e00da2996a6d6a9cd3cd110e72dd3626958c81b2d0a395e0030d36" exitCode=0 Nov 01 21:15:47.350885 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:47.350467 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1a1e9e-p4bw8" event={"ID":"395ed984-c93b-415a-8f5b-72c79c4e4f1a","Type":"ContainerDied","Data":"e57aabf609e00da2996a6d6a9cd3cd110e72dd3626958c81b2d0a395e0030d36"} Nov 01 21:15:47.619643 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:47.619543 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-df700a-788d57d789-n6jlx" podUID="37709aaf-dd72-454f-9131-5334513b7c2a" containerName="payload-processing" containerID="cri-o://857571fd4d8abedfa9086909cf9ced28262842dd4bc18e92cdb45ded376bb1aa" gracePeriod=25 Nov 01 21:15:47.725818 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:47.725774 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-df700a-789b6c9547-g9ps7" podUID="5b0aa3a9-007c-45e7-a21b-3fd0f2f8dbf7" containerName="payload-pre-processing" containerID="cri-o://1f70884702ed43d3e600d5300d5b56f68cd10d8d87417d5b27a9738ac21ddd99" gracePeriod=25 Nov 01 21:15:47.808693 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:47.808661 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:47.894399 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:47.894222 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-df700a-788d57d789-n6jlx" Nov 01 21:15:48.005813 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:48.005778 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-df700a-789b6c9547-g9ps7" Nov 01 21:15:48.009802 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:48.009774 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37709aaf-dd72-454f-9131-5334513b7c2a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37709aaf-dd72-454f-9131-5334513b7c2a-plugins-config-volume\") pod \"37709aaf-dd72-454f-9131-5334513b7c2a\" (UID: \"37709aaf-dd72-454f-9131-5334513b7c2a\") " Nov 01 21:15:48.009974 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:48.009959 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37709aaf-dd72-454f-9131-5334513b7c2a-kube-api-access-xxhrv\" (UniqueName: \"kubernetes.io/projected/37709aaf-dd72-454f-9131-5334513b7c2a-kube-api-access-xxhrv\") pod \"37709aaf-dd72-454f-9131-5334513b7c2a\" (UID: \"37709aaf-dd72-454f-9131-5334513b7c2a\") " Nov 01 21:15:48.010263 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:48.010240 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37709aaf-dd72-454f-9131-5334513b7c2a-plugins-config-volume" pod "37709aaf-dd72-454f-9131-5334513b7c2a" (UID: "37709aaf-dd72-454f-9131-5334513b7c2a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:15:48.010381 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:48.010365 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37709aaf-dd72-454f-9131-5334513b7c2a-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:48.012041 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:48.012020 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37709aaf-dd72-454f-9131-5334513b7c2a-kube-api-access-xxhrv" pod "37709aaf-dd72-454f-9131-5334513b7c2a" (UID: "37709aaf-dd72-454f-9131-5334513b7c2a"). InnerVolumeSpecName "kube-api-access-xxhrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:15:48.086959 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:48.086925 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:48.111713 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:48.111685 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b0aa3a9-007c-45e7-a21b-3fd0f2f8dbf7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b0aa3a9-007c-45e7-a21b-3fd0f2f8dbf7-plugins-config-volume\") pod \"5b0aa3a9-007c-45e7-a21b-3fd0f2f8dbf7\" (UID: \"5b0aa3a9-007c-45e7-a21b-3fd0f2f8dbf7\") " Nov 01 21:15:48.111902 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:48.111877 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b0aa3a9-007c-45e7-a21b-3fd0f2f8dbf7-kube-api-access-gx8vd\" (UniqueName: \"kubernetes.io/projected/5b0aa3a9-007c-45e7-a21b-3fd0f2f8dbf7-kube-api-access-gx8vd\") pod \"5b0aa3a9-007c-45e7-a21b-3fd0f2f8dbf7\" (UID: \"5b0aa3a9-007c-45e7-a21b-3fd0f2f8dbf7\") " Nov 01 21:15:48.112123 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:48.112107 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxhrv\" (UniqueName: \"kubernetes.io/projected/37709aaf-dd72-454f-9131-5334513b7c2a-kube-api-access-xxhrv\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:48.112214 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:48.112120 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b0aa3a9-007c-45e7-a21b-3fd0f2f8dbf7-plugins-config-volume" pod "5b0aa3a9-007c-45e7-a21b-3fd0f2f8dbf7" (UID: "5b0aa3a9-007c-45e7-a21b-3fd0f2f8dbf7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:15:48.113916 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:48.113897 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b0aa3a9-007c-45e7-a21b-3fd0f2f8dbf7-kube-api-access-gx8vd" pod "5b0aa3a9-007c-45e7-a21b-3fd0f2f8dbf7" (UID: "5b0aa3a9-007c-45e7-a21b-3fd0f2f8dbf7"). InnerVolumeSpecName "kube-api-access-gx8vd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:15:48.212991 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:48.212907 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx8vd\" (UniqueName: \"kubernetes.io/projected/5b0aa3a9-007c-45e7-a21b-3fd0f2f8dbf7-kube-api-access-gx8vd\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:48.212991 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:48.212937 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b0aa3a9-007c-45e7-a21b-3fd0f2f8dbf7-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:48.244160 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:48.244122 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:48.358598 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:48.358569 4484 generic.go:356] "Generic (PLEG): container finished" podID="37709aaf-dd72-454f-9131-5334513b7c2a" containerID="857571fd4d8abedfa9086909cf9ced28262842dd4bc18e92cdb45ded376bb1aa" exitCode=0 Nov 01 21:15:48.359121 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:48.358638 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-df700a-788d57d789-n6jlx" Nov 01 21:15:48.359121 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:48.358650 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-df700a-788d57d789-n6jlx" event={"ID":"37709aaf-dd72-454f-9131-5334513b7c2a","Type":"ContainerDied","Data":"857571fd4d8abedfa9086909cf9ced28262842dd4bc18e92cdb45ded376bb1aa"} Nov 01 21:15:48.359121 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:48.358722 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-df700a-788d57d789-n6jlx" event={"ID":"37709aaf-dd72-454f-9131-5334513b7c2a","Type":"ContainerDied","Data":"989d0fa577b2ad523a46922156a0b6ab03d7a821de983f3cd09ba36cdae72b2b"} Nov 01 21:15:48.359121 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:48.358751 4484 scope.go:122] "RemoveContainer" containerID="857571fd4d8abedfa9086909cf9ced28262842dd4bc18e92cdb45ded376bb1aa" Nov 01 21:15:48.360740 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:48.360364 4484 generic.go:356] "Generic (PLEG): container finished" podID="5b0aa3a9-007c-45e7-a21b-3fd0f2f8dbf7" containerID="1f70884702ed43d3e600d5300d5b56f68cd10d8d87417d5b27a9738ac21ddd99" exitCode=0 Nov 01 21:15:48.360740 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:48.360417 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-df700a-789b6c9547-g9ps7" Nov 01 21:15:48.360740 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:48.360437 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-df700a-789b6c9547-g9ps7" event={"ID":"5b0aa3a9-007c-45e7-a21b-3fd0f2f8dbf7","Type":"ContainerDied","Data":"1f70884702ed43d3e600d5300d5b56f68cd10d8d87417d5b27a9738ac21ddd99"} Nov 01 21:15:48.360740 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:48.360492 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-df700a-789b6c9547-g9ps7" event={"ID":"5b0aa3a9-007c-45e7-a21b-3fd0f2f8dbf7","Type":"ContainerDied","Data":"0197eceb99bfacfbfc67b8750bf896090e173d469508fda8930023166fc9d488"} Nov 01 21:15:48.370861 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:48.370836 4484 scope.go:122] "RemoveContainer" containerID="857571fd4d8abedfa9086909cf9ced28262842dd4bc18e92cdb45ded376bb1aa" Nov 01 21:15:48.371576 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:15:48.371550 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"857571fd4d8abedfa9086909cf9ced28262842dd4bc18e92cdb45ded376bb1aa\": container with ID starting with 857571fd4d8abedfa9086909cf9ced28262842dd4bc18e92cdb45ded376bb1aa not found: ID does not exist" containerID="857571fd4d8abedfa9086909cf9ced28262842dd4bc18e92cdb45ded376bb1aa" Nov 01 21:15:48.371660 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:48.371587 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"857571fd4d8abedfa9086909cf9ced28262842dd4bc18e92cdb45ded376bb1aa"} err="failed to get container status \"857571fd4d8abedfa9086909cf9ced28262842dd4bc18e92cdb45ded376bb1aa\": rpc error: code = NotFound desc = could not find container \"857571fd4d8abedfa9086909cf9ced28262842dd4bc18e92cdb45ded376bb1aa\": container with ID starting with 857571fd4d8abedfa9086909cf9ced28262842dd4bc18e92cdb45ded376bb1aa not found: ID does not exist" Nov 01 21:15:48.371660 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:48.371606 4484 scope.go:122] "RemoveContainer" containerID="1f70884702ed43d3e600d5300d5b56f68cd10d8d87417d5b27a9738ac21ddd99" Nov 01 21:15:48.381528 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:48.381504 4484 scope.go:122] "RemoveContainer" containerID="1f70884702ed43d3e600d5300d5b56f68cd10d8d87417d5b27a9738ac21ddd99" Nov 01 21:15:48.381974 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:15:48.381944 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f70884702ed43d3e600d5300d5b56f68cd10d8d87417d5b27a9738ac21ddd99\": container with ID starting with 1f70884702ed43d3e600d5300d5b56f68cd10d8d87417d5b27a9738ac21ddd99 not found: ID does not exist" containerID="1f70884702ed43d3e600d5300d5b56f68cd10d8d87417d5b27a9738ac21ddd99" Nov 01 21:15:48.382087 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:48.381984 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f70884702ed43d3e600d5300d5b56f68cd10d8d87417d5b27a9738ac21ddd99"} err="failed to get container status \"1f70884702ed43d3e600d5300d5b56f68cd10d8d87417d5b27a9738ac21ddd99\": rpc error: code = NotFound desc = could not find container \"1f70884702ed43d3e600d5300d5b56f68cd10d8d87417d5b27a9738ac21ddd99\": container with ID starting with 1f70884702ed43d3e600d5300d5b56f68cd10d8d87417d5b27a9738ac21ddd99 not found: ID does not exist" Nov 01 21:15:48.385746 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:48.385722 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-24f923-stmpt"] Nov 01 21:15:48.388167 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:48.388143 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-24f923-stmpt"] Nov 01 21:15:48.414407 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:48.414378 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:48.482475 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:48.482451 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1a1e9e-p4bw8" Nov 01 21:15:48.515735 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:48.515702 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/395ed984-c93b-415a-8f5b-72c79c4e4f1a-kube-api-access-sqkk5\" (UniqueName: \"kubernetes.io/projected/395ed984-c93b-415a-8f5b-72c79c4e4f1a-kube-api-access-sqkk5\") pod \"395ed984-c93b-415a-8f5b-72c79c4e4f1a\" (UID: \"395ed984-c93b-415a-8f5b-72c79c4e4f1a\") " Nov 01 21:15:48.515915 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:48.515802 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/395ed984-c93b-415a-8f5b-72c79c4e4f1a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/395ed984-c93b-415a-8f5b-72c79c4e4f1a-maas-api-xks-ca\") pod \"395ed984-c93b-415a-8f5b-72c79c4e4f1a\" (UID: \"395ed984-c93b-415a-8f5b-72c79c4e4f1a\") " Nov 01 21:15:48.515915 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:48.515880 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/395ed984-c93b-415a-8f5b-72c79c4e4f1a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/395ed984-c93b-415a-8f5b-72c79c4e4f1a-maas-api-openshift-ca\") pod \"395ed984-c93b-415a-8f5b-72c79c4e4f1a\" (UID: \"395ed984-c93b-415a-8f5b-72c79c4e4f1a\") " Nov 01 21:15:48.516296 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:48.516272 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/395ed984-c93b-415a-8f5b-72c79c4e4f1a-maas-api-openshift-ca" pod "395ed984-c93b-415a-8f5b-72c79c4e4f1a" (UID: "395ed984-c93b-415a-8f5b-72c79c4e4f1a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:15:48.517890 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:48.517868 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/395ed984-c93b-415a-8f5b-72c79c4e4f1a-maas-api-xks-ca" pod "395ed984-c93b-415a-8f5b-72c79c4e4f1a" (UID: "395ed984-c93b-415a-8f5b-72c79c4e4f1a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:15:48.518025 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:48.517907 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/395ed984-c93b-415a-8f5b-72c79c4e4f1a-kube-api-access-sqkk5" pod "395ed984-c93b-415a-8f5b-72c79c4e4f1a" (UID: "395ed984-c93b-415a-8f5b-72c79c4e4f1a"). InnerVolumeSpecName "kube-api-access-sqkk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:15:48.530850 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:48.530826 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-df700a-788d57d789-n6jlx"] Nov 01 21:15:48.534434 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:48.534404 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-df700a-788d57d789-n6jlx"] Nov 01 21:15:48.616707 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:48.616670 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqkk5\" (UniqueName: \"kubernetes.io/projected/395ed984-c93b-415a-8f5b-72c79c4e4f1a-kube-api-access-sqkk5\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:48.616707 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:48.616700 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/395ed984-c93b-415a-8f5b-72c79c4e4f1a-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:48.616707 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:48.616716 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/395ed984-c93b-415a-8f5b-72c79c4e4f1a-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:48.645240 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:48.645210 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-df700a-789b6c9547-g9ps7"] Nov 01 21:15:48.647521 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:48.647494 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-df700a-789b6c9547-g9ps7"] Nov 01 21:15:48.761474 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:48.761389 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:49.228148 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:49.228123 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2" Nov 01 21:15:49.323880 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:49.323838 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/297fe96b-76ad-499e-b9b7-f536d1851712-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/297fe96b-76ad-499e-b9b7-f536d1851712-istio-data\") pod \"297fe96b-76ad-499e-b9b7-f536d1851712\" (UID: \"297fe96b-76ad-499e-b9b7-f536d1851712\") " Nov 01 21:15:49.324126 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:49.323902 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/297fe96b-76ad-499e-b9b7-f536d1851712-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/297fe96b-76ad-499e-b9b7-f536d1851712-credential-socket\") pod \"297fe96b-76ad-499e-b9b7-f536d1851712\" (UID: \"297fe96b-76ad-499e-b9b7-f536d1851712\") " Nov 01 21:15:49.324126 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:49.323984 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/297fe96b-76ad-499e-b9b7-f536d1851712-kube-api-access-9mh25\" (UniqueName: \"kubernetes.io/projected/297fe96b-76ad-499e-b9b7-f536d1851712-kube-api-access-9mh25\") pod \"297fe96b-76ad-499e-b9b7-f536d1851712\" (UID: \"297fe96b-76ad-499e-b9b7-f536d1851712\") " Nov 01 21:15:49.324126 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:49.324064 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/297fe96b-76ad-499e-b9b7-f536d1851712-istio-token\" (UniqueName: \"kubernetes.io/projected/297fe96b-76ad-499e-b9b7-f536d1851712-istio-token\") pod \"297fe96b-76ad-499e-b9b7-f536d1851712\" (UID: \"297fe96b-76ad-499e-b9b7-f536d1851712\") " Nov 01 21:15:49.324126 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:49.324100 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/297fe96b-76ad-499e-b9b7-f536d1851712-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/297fe96b-76ad-499e-b9b7-f536d1851712-istio-podinfo\") pod \"297fe96b-76ad-499e-b9b7-f536d1851712\" (UID: \"297fe96b-76ad-499e-b9b7-f536d1851712\") " Nov 01 21:15:49.324376 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:49.324194 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/297fe96b-76ad-499e-b9b7-f536d1851712-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/297fe96b-76ad-499e-b9b7-f536d1851712-workload-socket\") pod \"297fe96b-76ad-499e-b9b7-f536d1851712\" (UID: \"297fe96b-76ad-499e-b9b7-f536d1851712\") " Nov 01 21:15:49.324376 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:49.324194 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/297fe96b-76ad-499e-b9b7-f536d1851712-istio-data" pod "297fe96b-76ad-499e-b9b7-f536d1851712" (UID: "297fe96b-76ad-499e-b9b7-f536d1851712"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:15:49.324376 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:49.324224 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/297fe96b-76ad-499e-b9b7-f536d1851712-credential-socket" pod "297fe96b-76ad-499e-b9b7-f536d1851712" (UID: "297fe96b-76ad-499e-b9b7-f536d1851712"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:15:49.324376 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:49.324226 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/297fe96b-76ad-499e-b9b7-f536d1851712-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/297fe96b-76ad-499e-b9b7-f536d1851712-istio-envoy\") pod \"297fe96b-76ad-499e-b9b7-f536d1851712\" (UID: \"297fe96b-76ad-499e-b9b7-f536d1851712\") " Nov 01 21:15:49.324376 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:49.324317 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/297fe96b-76ad-499e-b9b7-f536d1851712-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/297fe96b-76ad-499e-b9b7-f536d1851712-workload-certs\") pod \"297fe96b-76ad-499e-b9b7-f536d1851712\" (UID: \"297fe96b-76ad-499e-b9b7-f536d1851712\") " Nov 01 21:15:49.324671 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:49.324387 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/297fe96b-76ad-499e-b9b7-f536d1851712-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/297fe96b-76ad-499e-b9b7-f536d1851712-istiod-ca-cert\") pod \"297fe96b-76ad-499e-b9b7-f536d1851712\" (UID: \"297fe96b-76ad-499e-b9b7-f536d1851712\") " Nov 01 21:15:49.324671 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:49.324416 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/297fe96b-76ad-499e-b9b7-f536d1851712-workload-socket" pod "297fe96b-76ad-499e-b9b7-f536d1851712" (UID: "297fe96b-76ad-499e-b9b7-f536d1851712"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:15:49.324671 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:49.324582 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/297fe96b-76ad-499e-b9b7-f536d1851712-workload-certs" pod "297fe96b-76ad-499e-b9b7-f536d1851712" (UID: "297fe96b-76ad-499e-b9b7-f536d1851712"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:15:49.324850 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:49.324751 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/297fe96b-76ad-499e-b9b7-f536d1851712-istiod-ca-cert" pod "297fe96b-76ad-499e-b9b7-f536d1851712" (UID: "297fe96b-76ad-499e-b9b7-f536d1851712"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:15:49.324910 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:49.324884 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/297fe96b-76ad-499e-b9b7-f536d1851712-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:49.324910 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:49.324898 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/297fe96b-76ad-499e-b9b7-f536d1851712-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:49.324910 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:49.324907 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/297fe96b-76ad-499e-b9b7-f536d1851712-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:49.325120 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:49.324915 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/297fe96b-76ad-499e-b9b7-f536d1851712-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:49.325120 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:49.324924 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/297fe96b-76ad-499e-b9b7-f536d1851712-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:49.326287 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:49.326264 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/297fe96b-76ad-499e-b9b7-f536d1851712-kube-api-access-9mh25" pod "297fe96b-76ad-499e-b9b7-f536d1851712" (UID: "297fe96b-76ad-499e-b9b7-f536d1851712"). InnerVolumeSpecName "kube-api-access-9mh25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:15:49.326417 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:49.326387 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/297fe96b-76ad-499e-b9b7-f536d1851712-istio-envoy" pod "297fe96b-76ad-499e-b9b7-f536d1851712" (UID: "297fe96b-76ad-499e-b9b7-f536d1851712"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:15:49.326715 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:49.326693 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/297fe96b-76ad-499e-b9b7-f536d1851712-istio-podinfo" pod "297fe96b-76ad-499e-b9b7-f536d1851712" (UID: "297fe96b-76ad-499e-b9b7-f536d1851712"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:15:49.326785 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:49.326731 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/297fe96b-76ad-499e-b9b7-f536d1851712-istio-token" pod "297fe96b-76ad-499e-b9b7-f536d1851712" (UID: "297fe96b-76ad-499e-b9b7-f536d1851712"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:15:49.372158 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:49.372067 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1a1e9e-p4bw8" Nov 01 21:15:49.374066 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:49.374032 4484 generic.go:356] "Generic (PLEG): container finished" podID="297fe96b-76ad-499e-b9b7-f536d1851712" containerID="c5598745e8a990a6fa1b4835b95fdb9aae63eeffbdac8f7b48ea236ade0a472f" exitCode=0 Nov 01 21:15:49.374242 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:49.374225 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2" Nov 01 21:15:49.374698 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:49.374671 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37709aaf-dd72-454f-9131-5334513b7c2a" path="/var/lib/kubelet/pods/37709aaf-dd72-454f-9131-5334513b7c2a/volumes" Nov 01 21:15:49.375304 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:49.375280 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b0aa3a9-007c-45e7-a21b-3fd0f2f8dbf7" path="/var/lib/kubelet/pods/5b0aa3a9-007c-45e7-a21b-3fd0f2f8dbf7/volumes" Nov 01 21:15:49.375863 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:49.375843 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdfb78fd-83db-44e6-adeb-903d034934ab" path="/var/lib/kubelet/pods/fdfb78fd-83db-44e6-adeb-903d034934ab/volumes" Nov 01 21:15:49.376585 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:49.376558 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1a1e9e-p4bw8" event={"ID":"395ed984-c93b-415a-8f5b-72c79c4e4f1a","Type":"ContainerDied","Data":"1ac8caa545de2ba0432a47af7e47892455d9b6c6e7a6192939afce7d58478176"} Nov 01 21:15:49.376715 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:49.376607 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ac8caa545de2ba0432a47af7e47892455d9b6c6e7a6192939afce7d58478176" Nov 01 21:15:49.376715 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:49.376620 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2" event={"ID":"297fe96b-76ad-499e-b9b7-f536d1851712","Type":"ContainerDied","Data":"c5598745e8a990a6fa1b4835b95fdb9aae63eeffbdac8f7b48ea236ade0a472f"} Nov 01 21:15:49.376715 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:49.376649 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2" event={"ID":"297fe96b-76ad-499e-b9b7-f536d1851712","Type":"ContainerDied","Data":"2819b88d9211839434141b0623eeceffa5ba014cda4f1bb3c8d391c4cf5626a2"} Nov 01 21:15:49.376715 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:49.376680 4484 scope.go:122] "RemoveContainer" containerID="c5598745e8a990a6fa1b4835b95fdb9aae63eeffbdac8f7b48ea236ade0a472f" Nov 01 21:15:49.391219 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:49.391198 4484 scope.go:122] "RemoveContainer" containerID="0bb6eab4f96a0a4c0b346f8fa050f8575cf18099997779b566985b9b7a370e1b" Nov 01 21:15:49.400934 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:49.400913 4484 scope.go:122] "RemoveContainer" containerID="c5598745e8a990a6fa1b4835b95fdb9aae63eeffbdac8f7b48ea236ade0a472f" Nov 01 21:15:49.401331 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:15:49.401301 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5598745e8a990a6fa1b4835b95fdb9aae63eeffbdac8f7b48ea236ade0a472f\": container with ID starting with c5598745e8a990a6fa1b4835b95fdb9aae63eeffbdac8f7b48ea236ade0a472f not found: ID does not exist" containerID="c5598745e8a990a6fa1b4835b95fdb9aae63eeffbdac8f7b48ea236ade0a472f" Nov 01 21:15:49.401465 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:49.401370 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5598745e8a990a6fa1b4835b95fdb9aae63eeffbdac8f7b48ea236ade0a472f"} err="failed to get container status \"c5598745e8a990a6fa1b4835b95fdb9aae63eeffbdac8f7b48ea236ade0a472f\": rpc error: code = NotFound desc = could not find container \"c5598745e8a990a6fa1b4835b95fdb9aae63eeffbdac8f7b48ea236ade0a472f\": container with ID starting with c5598745e8a990a6fa1b4835b95fdb9aae63eeffbdac8f7b48ea236ade0a472f not found: ID does not exist" Nov 01 21:15:49.401465 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:49.401408 4484 scope.go:122] "RemoveContainer" containerID="0bb6eab4f96a0a4c0b346f8fa050f8575cf18099997779b566985b9b7a370e1b" Nov 01 21:15:49.401725 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:15:49.401704 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bb6eab4f96a0a4c0b346f8fa050f8575cf18099997779b566985b9b7a370e1b\": container with ID starting with 0bb6eab4f96a0a4c0b346f8fa050f8575cf18099997779b566985b9b7a370e1b not found: ID does not exist" containerID="0bb6eab4f96a0a4c0b346f8fa050f8575cf18099997779b566985b9b7a370e1b" Nov 01 21:15:49.401828 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:49.401729 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bb6eab4f96a0a4c0b346f8fa050f8575cf18099997779b566985b9b7a370e1b"} err="failed to get container status \"0bb6eab4f96a0a4c0b346f8fa050f8575cf18099997779b566985b9b7a370e1b\": rpc error: code = NotFound desc = could not find container \"0bb6eab4f96a0a4c0b346f8fa050f8575cf18099997779b566985b9b7a370e1b\": container with ID starting with 0bb6eab4f96a0a4c0b346f8fa050f8575cf18099997779b566985b9b7a370e1b not found: ID does not exist" Nov 01 21:15:49.424591 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:49.424564 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q"] Nov 01 21:15:49.425561 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:49.425536 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mh25\" (UniqueName: \"kubernetes.io/projected/297fe96b-76ad-499e-b9b7-f536d1851712-kube-api-access-9mh25\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:49.425561 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:49.425561 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/297fe96b-76ad-499e-b9b7-f536d1851712-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:49.425753 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:49.425572 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/297fe96b-76ad-499e-b9b7-f536d1851712-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:49.425753 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:49.425580 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/297fe96b-76ad-499e-b9b7-f536d1851712-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:49.434480 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:49.434458 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws"] Nov 01 21:15:49.600304 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:49.600266 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2"] Nov 01 21:15:49.603954 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:49.603927 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-a41242-openshift-default-86bb88f59-hn6s2"] Nov 01 21:15:49.634304 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:49.634218 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:51.372614 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:51.372580 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="297fe96b-76ad-499e-b9b7-f536d1851712" path="/var/lib/kubelet/pods/297fe96b-76ad-499e-b9b7-f536d1851712/volumes" Nov 01 21:15:51.511461 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:51.511403 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-208f2b-7fb7755ddc-lx76l" podUID="63adc0cc-05ea-4b6a-9f41-cde67a045b02" containerName="payload-processing" containerID="cri-o://854406f946db48c116ed03210fd1fedcda582d58c9168c829649feb819a89755" gracePeriod=25 Nov 01 21:15:52.351188 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:52.351145 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-208f2b-76b7b8c9f-fxd8f" podUID="2801b6dc-8797-4099-81fa-5fed21c2de2e" containerName="payload-pre-processing" containerID="cri-o://61f951c2913180db98752f89fbbb4d4af848cce81f6e65e421fe55d201799d2b" gracePeriod=25 Nov 01 21:15:52.402371 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:52.402325 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-208f2b-7fb7755ddc-lx76l" podUID="63adc0cc-05ea-4b6a-9f41-cde67a045b02" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 21:15:53.379305 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:53.379271 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:54.715695 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:54.715641 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:54.875783 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:54.875733 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:55.045393 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:55.045359 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:55.732785 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:55.732750 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bf3fa2-7xf2w"] Nov 01 21:15:55.733242 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:55.733228 4484 cpu_manager.go:414] "containerMap: removing container" podUID="297fe96b-76ad-499e-b9b7-f536d1851712" containerName="istio-proxy" Nov 01 21:15:55.733294 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:55.733243 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="297fe96b-76ad-499e-b9b7-f536d1851712" containerName="istio-proxy" Nov 01 21:15:55.733294 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:55.733270 4484 cpu_manager.go:414] "containerMap: removing container" podUID="395ed984-c93b-415a-8f5b-72c79c4e4f1a" containerName="revoke-keys" Nov 01 21:15:55.733294 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:55.733276 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="395ed984-c93b-415a-8f5b-72c79c4e4f1a" containerName="revoke-keys" Nov 01 21:15:55.733294 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:55.733286 4484 cpu_manager.go:414] "containerMap: removing container" podUID="297fe96b-76ad-499e-b9b7-f536d1851712" containerName="istio-proxy" Nov 01 21:15:55.733294 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:55.733291 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="297fe96b-76ad-499e-b9b7-f536d1851712" containerName="istio-proxy" Nov 01 21:15:55.733470 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:55.733301 4484 cpu_manager.go:414] "containerMap: removing container" podUID="8a5b2e73-9f47-47aa-9ed7-c75d17b1f658" containerName="istio-proxy" Nov 01 21:15:55.733470 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:55.733307 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a5b2e73-9f47-47aa-9ed7-c75d17b1f658" containerName="istio-proxy" Nov 01 21:15:55.733470 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:55.733325 4484 cpu_manager.go:414] "containerMap: removing container" podUID="5b0aa3a9-007c-45e7-a21b-3fd0f2f8dbf7" containerName="payload-pre-processing" Nov 01 21:15:55.733470 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:55.733330 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b0aa3a9-007c-45e7-a21b-3fd0f2f8dbf7" containerName="payload-pre-processing" Nov 01 21:15:55.733470 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:55.733337 4484 cpu_manager.go:414] "containerMap: removing container" podUID="37709aaf-dd72-454f-9131-5334513b7c2a" containerName="payload-processing" Nov 01 21:15:55.733470 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:55.733341 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37709aaf-dd72-454f-9131-5334513b7c2a" containerName="payload-processing" Nov 01 21:15:55.733470 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:55.733404 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="297fe96b-76ad-499e-b9b7-f536d1851712" containerName="istio-proxy" Nov 01 21:15:55.733470 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:55.733417 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="395ed984-c93b-415a-8f5b-72c79c4e4f1a" containerName="revoke-keys" Nov 01 21:15:55.733470 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:55.733424 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="297fe96b-76ad-499e-b9b7-f536d1851712" containerName="istio-proxy" Nov 01 21:15:55.733470 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:55.733432 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b0aa3a9-007c-45e7-a21b-3fd0f2f8dbf7" containerName="payload-pre-processing" Nov 01 21:15:55.733470 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:55.733438 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="37709aaf-dd72-454f-9131-5334513b7c2a" containerName="payload-processing" Nov 01 21:15:55.738049 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:55.738033 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bf3fa2-7xf2w" Nov 01 21:15:55.792177 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:55.792146 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bf3fa2-7xf2w"] Nov 01 21:15:55.798471 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:55.798438 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09664937-66a9-4626-83a6-3a4667d4dc68-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-bf3fa2-7xf2w\" (UID: \"09664937-66a9-4626-83a6-3a4667d4dc68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bf3fa2-7xf2w" Nov 01 21:15:55.798707 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:55.798680 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvxg8\" (UniqueName: \"kubernetes.io/projected/09664937-66a9-4626-83a6-3a4667d4dc68-kube-api-access-fvxg8\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-bf3fa2-7xf2w\" (UID: \"09664937-66a9-4626-83a6-3a4667d4dc68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bf3fa2-7xf2w" Nov 01 21:15:55.798857 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:55.798726 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09664937-66a9-4626-83a6-3a4667d4dc68-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-bf3fa2-7xf2w\" (UID: \"09664937-66a9-4626-83a6-3a4667d4dc68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bf3fa2-7xf2w" Nov 01 21:15:55.840106 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:55.840065 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:15:55.840317 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:55.840181 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:15:55.840317 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:55.840204 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:15:55.841839 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:55.841815 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:15:55.899840 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:55.899802 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvxg8\" (UniqueName: \"kubernetes.io/projected/09664937-66a9-4626-83a6-3a4667d4dc68-kube-api-access-fvxg8\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-bf3fa2-7xf2w\" (UID: \"09664937-66a9-4626-83a6-3a4667d4dc68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bf3fa2-7xf2w" Nov 01 21:15:55.899840 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:55.899842 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09664937-66a9-4626-83a6-3a4667d4dc68-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-bf3fa2-7xf2w\" (UID: \"09664937-66a9-4626-83a6-3a4667d4dc68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bf3fa2-7xf2w" Nov 01 21:15:55.900237 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:55.899888 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09664937-66a9-4626-83a6-3a4667d4dc68-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-bf3fa2-7xf2w\" (UID: \"09664937-66a9-4626-83a6-3a4667d4dc68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bf3fa2-7xf2w" Nov 01 21:15:55.900643 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:55.900622 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09664937-66a9-4626-83a6-3a4667d4dc68-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-bf3fa2-7xf2w\" (UID: \"09664937-66a9-4626-83a6-3a4667d4dc68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bf3fa2-7xf2w" Nov 01 21:15:55.902385 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:55.902356 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09664937-66a9-4626-83a6-3a4667d4dc68-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-bf3fa2-7xf2w\" (UID: \"09664937-66a9-4626-83a6-3a4667d4dc68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bf3fa2-7xf2w" Nov 01 21:15:55.957606 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:55.957571 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvxg8\" (UniqueName: \"kubernetes.io/projected/09664937-66a9-4626-83a6-3a4667d4dc68-kube-api-access-fvxg8\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-bf3fa2-7xf2w\" (UID: \"09664937-66a9-4626-83a6-3a4667d4dc68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bf3fa2-7xf2w" Nov 01 21:15:56.051612 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:56.051569 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bf3fa2-7xf2w" Nov 01 21:15:56.427051 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:56.427022 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bf3fa2-7xf2w"] Nov 01 21:15:56.428072 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:15:56.428040 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09664937_66a9_4626_83a6_3a4667d4dc68.slice/crio-11690eb3cdf5d1ecb0a63d0696937dab1528ea018201082c62f71c02aed25e31 WatchSource:0}: Error finding container 11690eb3cdf5d1ecb0a63d0696937dab1528ea018201082c62f71c02aed25e31: Status 404 returned error can't find the container with id 11690eb3cdf5d1ecb0a63d0696937dab1528ea018201082c62f71c02aed25e31 Nov 01 21:15:57.427432 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:57.427398 4484 generic.go:356] "Generic (PLEG): container finished" podID="09664937-66a9-4626-83a6-3a4667d4dc68" containerID="7d361c6ae6b1328446fbe82824322f7de911699b2f3f40a1b69d7bc5088813fe" exitCode=0 Nov 01 21:15:57.427862 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:57.427488 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bf3fa2-7xf2w" event={"ID":"09664937-66a9-4626-83a6-3a4667d4dc68","Type":"ContainerDied","Data":"7d361c6ae6b1328446fbe82824322f7de911699b2f3f40a1b69d7bc5088813fe"} Nov 01 21:15:57.427862 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:57.427550 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bf3fa2-7xf2w" event={"ID":"09664937-66a9-4626-83a6-3a4667d4dc68","Type":"ContainerStarted","Data":"11690eb3cdf5d1ecb0a63d0696937dab1528ea018201082c62f71c02aed25e31"} Nov 01 21:15:57.669968 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:57.669893 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-1a1e9e-66d4f8c5f-cr9fp"] Nov 01 21:15:57.689741 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:57.689697 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56018f3e-dbs8g"] Nov 01 21:15:57.693521 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:57.693498 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56018f3e-dbs8g" Nov 01 21:15:57.751172 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:57.751125 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56018f3e-dbs8g"] Nov 01 21:15:57.753874 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:57.753843 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee43e1ee-cq2d9"] Nov 01 21:15:57.757734 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:57.757714 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee43e1ee-cq2d9" Nov 01 21:15:57.777166 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:57.777135 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-1a1e9e-8fc6dcb7-bs6sm"] Nov 01 21:15:57.813679 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:57.813643 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee43e1ee-cq2d9"] Nov 01 21:15:57.816783 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:57.816754 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmxqk\" (UniqueName: \"kubernetes.io/projected/e61432fd-9976-4b09-b225-b9750d67aa3a-kube-api-access-jmxqk\") pod \"maas-api-revoke-sub-ee43e1ee-cq2d9\" (UID: \"e61432fd-9976-4b09-b225-b9750d67aa3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee43e1ee-cq2d9" Nov 01 21:15:57.816960 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:57.816847 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/684eee5e-908a-4f3c-9c04-782732b39560-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56018f3e-dbs8g\" (UID: \"684eee5e-908a-4f3c-9c04-782732b39560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56018f3e-dbs8g" Nov 01 21:15:57.816960 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:57.816914 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e61432fd-9976-4b09-b225-b9750d67aa3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee43e1ee-cq2d9\" (UID: \"e61432fd-9976-4b09-b225-b9750d67aa3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee43e1ee-cq2d9" Nov 01 21:15:57.817131 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:57.816964 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/684eee5e-908a-4f3c-9c04-782732b39560-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56018f3e-dbs8g\" (UID: \"684eee5e-908a-4f3c-9c04-782732b39560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56018f3e-dbs8g" Nov 01 21:15:57.817131 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:57.817007 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hr5d\" (UniqueName: \"kubernetes.io/projected/684eee5e-908a-4f3c-9c04-782732b39560-kube-api-access-6hr5d\") pod \"maas-api-revoke-sub-56018f3e-dbs8g\" (UID: \"684eee5e-908a-4f3c-9c04-782732b39560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56018f3e-dbs8g" Nov 01 21:15:57.817251 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:57.817149 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e61432fd-9976-4b09-b225-b9750d67aa3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee43e1ee-cq2d9\" (UID: \"e61432fd-9976-4b09-b225-b9750d67aa3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee43e1ee-cq2d9" Nov 01 21:15:57.918663 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:57.918633 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/684eee5e-908a-4f3c-9c04-782732b39560-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56018f3e-dbs8g\" (UID: \"684eee5e-908a-4f3c-9c04-782732b39560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56018f3e-dbs8g" Nov 01 21:15:57.918830 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:57.918669 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e61432fd-9976-4b09-b225-b9750d67aa3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee43e1ee-cq2d9\" (UID: \"e61432fd-9976-4b09-b225-b9750d67aa3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee43e1ee-cq2d9" Nov 01 21:15:57.918830 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:57.918700 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/684eee5e-908a-4f3c-9c04-782732b39560-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56018f3e-dbs8g\" (UID: \"684eee5e-908a-4f3c-9c04-782732b39560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56018f3e-dbs8g" Nov 01 21:15:57.918830 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:57.918728 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hr5d\" (UniqueName: \"kubernetes.io/projected/684eee5e-908a-4f3c-9c04-782732b39560-kube-api-access-6hr5d\") pod \"maas-api-revoke-sub-56018f3e-dbs8g\" (UID: \"684eee5e-908a-4f3c-9c04-782732b39560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56018f3e-dbs8g" Nov 01 21:15:57.918830 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:57.918764 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e61432fd-9976-4b09-b225-b9750d67aa3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee43e1ee-cq2d9\" (UID: \"e61432fd-9976-4b09-b225-b9750d67aa3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee43e1ee-cq2d9" Nov 01 21:15:57.919107 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:57.918845 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmxqk\" (UniqueName: \"kubernetes.io/projected/e61432fd-9976-4b09-b225-b9750d67aa3a-kube-api-access-jmxqk\") pod \"maas-api-revoke-sub-ee43e1ee-cq2d9\" (UID: \"e61432fd-9976-4b09-b225-b9750d67aa3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee43e1ee-cq2d9" Nov 01 21:15:57.919475 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:57.919450 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/684eee5e-908a-4f3c-9c04-782732b39560-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56018f3e-dbs8g\" (UID: \"684eee5e-908a-4f3c-9c04-782732b39560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56018f3e-dbs8g" Nov 01 21:15:57.919574 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:57.919457 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e61432fd-9976-4b09-b225-b9750d67aa3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee43e1ee-cq2d9\" (UID: \"e61432fd-9976-4b09-b225-b9750d67aa3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee43e1ee-cq2d9" Nov 01 21:15:57.921062 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:57.921008 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/684eee5e-908a-4f3c-9c04-782732b39560-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56018f3e-dbs8g\" (UID: \"684eee5e-908a-4f3c-9c04-782732b39560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56018f3e-dbs8g" Nov 01 21:15:57.921153 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:57.921082 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e61432fd-9976-4b09-b225-b9750d67aa3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee43e1ee-cq2d9\" (UID: \"e61432fd-9976-4b09-b225-b9750d67aa3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee43e1ee-cq2d9" Nov 01 21:15:57.956201 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:57.956161 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:57.979055 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:57.978735 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hr5d\" (UniqueName: \"kubernetes.io/projected/684eee5e-908a-4f3c-9c04-782732b39560-kube-api-access-6hr5d\") pod \"maas-api-revoke-sub-56018f3e-dbs8g\" (UID: \"684eee5e-908a-4f3c-9c04-782732b39560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56018f3e-dbs8g" Nov 01 21:15:57.979055 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:57.979024 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmxqk\" (UniqueName: \"kubernetes.io/projected/e61432fd-9976-4b09-b225-b9750d67aa3a-kube-api-access-jmxqk\") pod \"maas-api-revoke-sub-ee43e1ee-cq2d9\" (UID: \"e61432fd-9976-4b09-b225-b9750d67aa3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee43e1ee-cq2d9" Nov 01 21:15:58.006463 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:58.006435 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56018f3e-dbs8g" Nov 01 21:15:58.072562 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:58.072535 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee43e1ee-cq2d9" Nov 01 21:15:58.110313 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:58.110274 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:58.275424 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:58.275394 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:58.388850 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:58.382331 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56018f3e-dbs8g"] Nov 01 21:15:58.426741 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:58.426636 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:58.436154 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:58.436125 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56018f3e-dbs8g" event={"ID":"684eee5e-908a-4f3c-9c04-782732b39560","Type":"ContainerStarted","Data":"bdd91976f3b03c3dd24cce2c0ae2ea2448e84024ca4cd85bebd78e20bacb5e03"} Nov 01 21:15:58.489954 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:58.489920 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee43e1ee-cq2d9"] Nov 01 21:15:58.490641 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:15:58.490598 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode61432fd_9976_4b09_b225_b9750d67aa3a.slice/crio-05d8b9b5c5c050f1b278783201e66403b29cd6e934a3f62734651c2afb2c6509 WatchSource:0}: Error finding container 05d8b9b5c5c050f1b278783201e66403b29cd6e934a3f62734651c2afb2c6509: Status 404 returned error can't find the container with id 05d8b9b5c5c050f1b278783201e66403b29cd6e934a3f62734651c2afb2c6509 Nov 01 21:15:58.560208 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:58.560184 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bf3fa2-7xf2w" Nov 01 21:15:58.626310 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:58.626279 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09664937-66a9-4626-83a6-3a4667d4dc68-kube-api-access-fvxg8\" (UniqueName: \"kubernetes.io/projected/09664937-66a9-4626-83a6-3a4667d4dc68-kube-api-access-fvxg8\") pod \"09664937-66a9-4626-83a6-3a4667d4dc68\" (UID: \"09664937-66a9-4626-83a6-3a4667d4dc68\") " Nov 01 21:15:58.626478 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:58.626329 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09664937-66a9-4626-83a6-3a4667d4dc68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09664937-66a9-4626-83a6-3a4667d4dc68-maas-api-xks-ca\") pod \"09664937-66a9-4626-83a6-3a4667d4dc68\" (UID: \"09664937-66a9-4626-83a6-3a4667d4dc68\") " Nov 01 21:15:58.626478 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:58.626449 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09664937-66a9-4626-83a6-3a4667d4dc68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09664937-66a9-4626-83a6-3a4667d4dc68-maas-api-openshift-ca\") pod \"09664937-66a9-4626-83a6-3a4667d4dc68\" (UID: \"09664937-66a9-4626-83a6-3a4667d4dc68\") " Nov 01 21:15:58.626805 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:58.626782 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09664937-66a9-4626-83a6-3a4667d4dc68-maas-api-openshift-ca" pod "09664937-66a9-4626-83a6-3a4667d4dc68" (UID: "09664937-66a9-4626-83a6-3a4667d4dc68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:15:58.626991 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:58.626972 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09664937-66a9-4626-83a6-3a4667d4dc68-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:58.628585 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:58.628563 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09664937-66a9-4626-83a6-3a4667d4dc68-maas-api-xks-ca" pod "09664937-66a9-4626-83a6-3a4667d4dc68" (UID: "09664937-66a9-4626-83a6-3a4667d4dc68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:15:58.628585 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:58.628573 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09664937-66a9-4626-83a6-3a4667d4dc68-kube-api-access-fvxg8" pod "09664937-66a9-4626-83a6-3a4667d4dc68" (UID: "09664937-66a9-4626-83a6-3a4667d4dc68"). InnerVolumeSpecName "kube-api-access-fvxg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:15:58.727986 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:58.727960 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvxg8\" (UniqueName: \"kubernetes.io/projected/09664937-66a9-4626-83a6-3a4667d4dc68-kube-api-access-fvxg8\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:58.727986 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:58.727986 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09664937-66a9-4626-83a6-3a4667d4dc68-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:59.447949 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:59.447898 4484 generic.go:356] "Generic (PLEG): container finished" podID="684eee5e-908a-4f3c-9c04-782732b39560" containerID="2c07bac71e1347752f1cc099eb3c05ebd66434cc23a0417037cbeb6bd854760c" exitCode=0 Nov 01 21:15:59.448378 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:59.448042 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56018f3e-dbs8g" event={"ID":"684eee5e-908a-4f3c-9c04-782732b39560","Type":"ContainerDied","Data":"2c07bac71e1347752f1cc099eb3c05ebd66434cc23a0417037cbeb6bd854760c"} Nov 01 21:15:59.450411 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:59.450376 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee43e1ee-cq2d9" event={"ID":"e61432fd-9976-4b09-b225-b9750d67aa3a","Type":"ContainerStarted","Data":"122624c96d30804d0282e0b7ddf0f9666781ce9580db37fc59f0f35c367a5632"} Nov 01 21:15:59.450648 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:59.450436 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee43e1ee-cq2d9" event={"ID":"e61432fd-9976-4b09-b225-b9750d67aa3a","Type":"ContainerStarted","Data":"05d8b9b5c5c050f1b278783201e66403b29cd6e934a3f62734651c2afb2c6509"} Nov 01 21:15:59.451881 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:59.451862 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bf3fa2-7xf2w" Nov 01 21:15:59.452035 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:59.451897 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bf3fa2-7xf2w" event={"ID":"09664937-66a9-4626-83a6-3a4667d4dc68","Type":"ContainerDied","Data":"11690eb3cdf5d1ecb0a63d0696937dab1528ea018201082c62f71c02aed25e31"} Nov 01 21:15:59.452035 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:59.451935 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11690eb3cdf5d1ecb0a63d0696937dab1528ea018201082c62f71c02aed25e31" Nov 01 21:15:59.617630 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:59.617547 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:59.678331 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:59.678274 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee43e1ee-cq2d9" podStartSLOduration=2.678259825 podStartE2EDuration="2.678259825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:15:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:15:59.676742525 +0000 UTC m=+2710.454382660" watchObservedRunningTime="2026-11-01 21:15:59.678259825 +0000 UTC m=+2710.455899959" Nov 01 21:15:59.884570 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:15:59.884492 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:00.043260 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:00.043224 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:00.221967 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:00.221879 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-df700a-t5hcp"] Nov 01 21:16:00.224596 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:00.224575 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-df700a-t5hcp"] Nov 01 21:16:00.461160 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:00.461125 4484 generic.go:356] "Generic (PLEG): container finished" podID="e61432fd-9976-4b09-b225-b9750d67aa3a" containerID="122624c96d30804d0282e0b7ddf0f9666781ce9580db37fc59f0f35c367a5632" exitCode=0 Nov 01 21:16:00.461575 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:00.461218 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee43e1ee-cq2d9" event={"ID":"e61432fd-9976-4b09-b225-b9750d67aa3a","Type":"ContainerDied","Data":"122624c96d30804d0282e0b7ddf0f9666781ce9580db37fc59f0f35c367a5632"} Nov 01 21:16:00.613386 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:00.613352 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56018f3e-dbs8g" Nov 01 21:16:00.627322 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:00.627284 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:00.760537 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:00.760503 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/684eee5e-908a-4f3c-9c04-782732b39560-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/684eee5e-908a-4f3c-9c04-782732b39560-maas-api-xks-ca\") pod \"684eee5e-908a-4f3c-9c04-782732b39560\" (UID: \"684eee5e-908a-4f3c-9c04-782732b39560\") " Nov 01 21:16:00.760695 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:00.760603 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/684eee5e-908a-4f3c-9c04-782732b39560-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/684eee5e-908a-4f3c-9c04-782732b39560-maas-api-openshift-ca\") pod \"684eee5e-908a-4f3c-9c04-782732b39560\" (UID: \"684eee5e-908a-4f3c-9c04-782732b39560\") " Nov 01 21:16:00.760695 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:00.760660 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/684eee5e-908a-4f3c-9c04-782732b39560-kube-api-access-6hr5d\" (UniqueName: \"kubernetes.io/projected/684eee5e-908a-4f3c-9c04-782732b39560-kube-api-access-6hr5d\") pod \"684eee5e-908a-4f3c-9c04-782732b39560\" (UID: \"684eee5e-908a-4f3c-9c04-782732b39560\") " Nov 01 21:16:00.761026 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:00.760966 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/684eee5e-908a-4f3c-9c04-782732b39560-maas-api-openshift-ca" pod "684eee5e-908a-4f3c-9c04-782732b39560" (UID: "684eee5e-908a-4f3c-9c04-782732b39560"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:16:00.761175 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:00.761102 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/684eee5e-908a-4f3c-9c04-782732b39560-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:00.762686 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:00.762661 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/684eee5e-908a-4f3c-9c04-782732b39560-kube-api-access-6hr5d" pod "684eee5e-908a-4f3c-9c04-782732b39560" (UID: "684eee5e-908a-4f3c-9c04-782732b39560"). InnerVolumeSpecName "kube-api-access-6hr5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:16:00.762796 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:00.762725 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/684eee5e-908a-4f3c-9c04-782732b39560-maas-api-xks-ca" pod "684eee5e-908a-4f3c-9c04-782732b39560" (UID: "684eee5e-908a-4f3c-9c04-782732b39560"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:16:00.862413 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:00.862382 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/684eee5e-908a-4f3c-9c04-782732b39560-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:00.862413 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:00.862412 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hr5d\" (UniqueName: \"kubernetes.io/projected/684eee5e-908a-4f3c-9c04-782732b39560-kube-api-access-6hr5d\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:01.187773 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:01.187694 4484 scope.go:122] "RemoveContainer" containerID="da84c929870134471bed08ccb9e480182d572a25708a2e2493781d7cf52e844e" Nov 01 21:16:01.197906 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:01.197885 4484 scope.go:122] "RemoveContainer" containerID="91c1faf51b1598e1b9d909446f90217ec650012b77aee07fc176cf543e3b5e4d" Nov 01 21:16:01.372450 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:01.372415 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a86e09e-82de-4bcc-9611-eead7fdb2ce9" path="/var/lib/kubelet/pods/0a86e09e-82de-4bcc-9611-eead7fdb2ce9/volumes" Nov 01 21:16:01.396525 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:01.396487 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-208f2b-76b7b8c9f-fxd8f" podUID="2801b6dc-8797-4099-81fa-5fed21c2de2e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 21:16:01.471253 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:01.471174 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56018f3e-dbs8g" Nov 01 21:16:01.471253 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:01.471213 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56018f3e-dbs8g" event={"ID":"684eee5e-908a-4f3c-9c04-782732b39560","Type":"ContainerDied","Data":"bdd91976f3b03c3dd24cce2c0ae2ea2448e84024ca4cd85bebd78e20bacb5e03"} Nov 01 21:16:01.471695 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:01.471262 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bdd91976f3b03c3dd24cce2c0ae2ea2448e84024ca4cd85bebd78e20bacb5e03" Nov 01 21:16:01.622461 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:01.622432 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee43e1ee-cq2d9" Nov 01 21:16:01.771541 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:01.771500 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e61432fd-9976-4b09-b225-b9750d67aa3a-kube-api-access-jmxqk\" (UniqueName: \"kubernetes.io/projected/e61432fd-9976-4b09-b225-b9750d67aa3a-kube-api-access-jmxqk\") pod \"e61432fd-9976-4b09-b225-b9750d67aa3a\" (UID: \"e61432fd-9976-4b09-b225-b9750d67aa3a\") " Nov 01 21:16:01.771760 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:01.771603 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e61432fd-9976-4b09-b225-b9750d67aa3a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e61432fd-9976-4b09-b225-b9750d67aa3a-maas-api-openshift-ca\") pod \"e61432fd-9976-4b09-b225-b9750d67aa3a\" (UID: \"e61432fd-9976-4b09-b225-b9750d67aa3a\") " Nov 01 21:16:01.771760 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:01.771697 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e61432fd-9976-4b09-b225-b9750d67aa3a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e61432fd-9976-4b09-b225-b9750d67aa3a-maas-api-xks-ca\") pod \"e61432fd-9976-4b09-b225-b9750d67aa3a\" (UID: \"e61432fd-9976-4b09-b225-b9750d67aa3a\") " Nov 01 21:16:01.772093 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:01.772067 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e61432fd-9976-4b09-b225-b9750d67aa3a-maas-api-openshift-ca" pod "e61432fd-9976-4b09-b225-b9750d67aa3a" (UID: "e61432fd-9976-4b09-b225-b9750d67aa3a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:16:01.773801 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:01.773770 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e61432fd-9976-4b09-b225-b9750d67aa3a-maas-api-xks-ca" pod "e61432fd-9976-4b09-b225-b9750d67aa3a" (UID: "e61432fd-9976-4b09-b225-b9750d67aa3a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:16:01.773931 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:01.773910 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e61432fd-9976-4b09-b225-b9750d67aa3a-kube-api-access-jmxqk" pod "e61432fd-9976-4b09-b225-b9750d67aa3a" (UID: "e61432fd-9976-4b09-b225-b9750d67aa3a"). InnerVolumeSpecName "kube-api-access-jmxqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:16:01.873091 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:01.873048 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e61432fd-9976-4b09-b225-b9750d67aa3a-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:01.873091 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:01.873083 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e61432fd-9976-4b09-b225-b9750d67aa3a-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:01.873091 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:01.873099 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmxqk\" (UniqueName: \"kubernetes.io/projected/e61432fd-9976-4b09-b225-b9750d67aa3a-kube-api-access-jmxqk\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:02.402037 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:02.401975 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-208f2b-7fb7755ddc-lx76l" podUID="63adc0cc-05ea-4b6a-9f41-cde67a045b02" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 21:16:02.479141 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:02.479111 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee43e1ee-cq2d9" Nov 01 21:16:02.479546 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:02.479141 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee43e1ee-cq2d9" event={"ID":"e61432fd-9976-4b09-b225-b9750d67aa3a","Type":"ContainerDied","Data":"05d8b9b5c5c050f1b278783201e66403b29cd6e934a3f62734651c2afb2c6509"} Nov 01 21:16:02.479546 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:02.479194 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05d8b9b5c5c050f1b278783201e66403b29cd6e934a3f62734651c2afb2c6509" Nov 01 21:16:02.671207 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:02.671099 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-1a1e9e-66d4f8c5f-cr9fp" podUID="8a9e967b-5887-4361-95ad-9a4122698833" containerName="payload-processing" containerID="cri-o://964c4d7650f59a65aa7a22dc715d5f3763ebf12890a3c110838a4f3f43ca558c" gracePeriod=25 Nov 01 21:16:02.778407 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:02.778331 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1a1e9e-8fc6dcb7-bs6sm" podUID="903377a5-cfac-4d8a-adf0-aa729a3cfdde" containerName="payload-pre-processing" containerID="cri-o://c18048df1d8d86a50759e6376198bc7db34272c3c122c1e3070a9671b7b67572" gracePeriod=25 Nov 01 21:16:04.445962 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:04.445923 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" podUID="9f19be29-2c84-4ae4-9674-acca4905f4e0" containerName="main" containerID="cri-o://018a8111200584e1a07c782d1e392d72ba8219e4b5fbb40d840cec6b2b77f167" gracePeriod=45 Nov 01 21:16:04.456427 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:04.456387 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" podUID="fe8307db-264d-4b8a-b097-f180bc255a9e" containerName="main" containerID="cri-o://b5fb24eb8274831d9a71f6e345a01ef166c31017df29796bd931accc2d74eb85" gracePeriod=45 Nov 01 21:16:04.732463 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:04.732433 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" Nov 01 21:16:04.754492 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:04.754466 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" Nov 01 21:16:04.905891 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:04.905861 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f19be29-2c84-4ae4-9674-acca4905f4e0-kube-api-access-9rjtz\" (UniqueName: \"kubernetes.io/projected/9f19be29-2c84-4ae4-9674-acca4905f4e0-kube-api-access-9rjtz\") pod \"9f19be29-2c84-4ae4-9674-acca4905f4e0\" (UID: \"9f19be29-2c84-4ae4-9674-acca4905f4e0\") " Nov 01 21:16:04.906157 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:04.905896 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe8307db-264d-4b8a-b097-f180bc255a9e-kube-api-access-p542x\" (UniqueName: \"kubernetes.io/projected/fe8307db-264d-4b8a-b097-f180bc255a9e-kube-api-access-p542x\") pod \"fe8307db-264d-4b8a-b097-f180bc255a9e\" (UID: \"fe8307db-264d-4b8a-b097-f180bc255a9e\") " Nov 01 21:16:04.906157 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:04.905930 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f19be29-2c84-4ae4-9674-acca4905f4e0-tls-certs\" (UniqueName: \"kubernetes.io/secret/9f19be29-2c84-4ae4-9674-acca4905f4e0-tls-certs\") pod \"9f19be29-2c84-4ae4-9674-acca4905f4e0\" (UID: \"9f19be29-2c84-4ae4-9674-acca4905f4e0\") " Nov 01 21:16:04.906157 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:04.906053 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f19be29-2c84-4ae4-9674-acca4905f4e0-home\" (UniqueName: \"kubernetes.io/empty-dir/9f19be29-2c84-4ae4-9674-acca4905f4e0-home\") pod \"9f19be29-2c84-4ae4-9674-acca4905f4e0\" (UID: \"9f19be29-2c84-4ae4-9674-acca4905f4e0\") " Nov 01 21:16:04.906157 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:04.906098 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe8307db-264d-4b8a-b097-f180bc255a9e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe8307db-264d-4b8a-b097-f180bc255a9e-dshm\") pod \"fe8307db-264d-4b8a-b097-f180bc255a9e\" (UID: \"fe8307db-264d-4b8a-b097-f180bc255a9e\") " Nov 01 21:16:04.906397 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:04.906172 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe8307db-264d-4b8a-b097-f180bc255a9e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe8307db-264d-4b8a-b097-f180bc255a9e-tmp-dir\") pod \"fe8307db-264d-4b8a-b097-f180bc255a9e\" (UID: \"fe8307db-264d-4b8a-b097-f180bc255a9e\") " Nov 01 21:16:04.906397 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:04.906242 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe8307db-264d-4b8a-b097-f180bc255a9e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe8307db-264d-4b8a-b097-f180bc255a9e-model-cache\") pod \"fe8307db-264d-4b8a-b097-f180bc255a9e\" (UID: \"fe8307db-264d-4b8a-b097-f180bc255a9e\") " Nov 01 21:16:04.906397 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:04.906280 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f19be29-2c84-4ae4-9674-acca4905f4e0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f19be29-2c84-4ae4-9674-acca4905f4e0-model-cache\") pod \"9f19be29-2c84-4ae4-9674-acca4905f4e0\" (UID: \"9f19be29-2c84-4ae4-9674-acca4905f4e0\") " Nov 01 21:16:04.906397 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:04.906319 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f19be29-2c84-4ae4-9674-acca4905f4e0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f19be29-2c84-4ae4-9674-acca4905f4e0-dshm\") pod \"9f19be29-2c84-4ae4-9674-acca4905f4e0\" (UID: \"9f19be29-2c84-4ae4-9674-acca4905f4e0\") " Nov 01 21:16:04.906397 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:04.906340 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f19be29-2c84-4ae4-9674-acca4905f4e0-home" pod "9f19be29-2c84-4ae4-9674-acca4905f4e0" (UID: "9f19be29-2c84-4ae4-9674-acca4905f4e0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:16:04.906397 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:04.906375 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe8307db-264d-4b8a-b097-f180bc255a9e-tls-certs\" (UniqueName: \"kubernetes.io/secret/fe8307db-264d-4b8a-b097-f180bc255a9e-tls-certs\") pod \"fe8307db-264d-4b8a-b097-f180bc255a9e\" (UID: \"fe8307db-264d-4b8a-b097-f180bc255a9e\") " Nov 01 21:16:04.906768 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:04.906452 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f19be29-2c84-4ae4-9674-acca4905f4e0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f19be29-2c84-4ae4-9674-acca4905f4e0-tmp-dir\") pod \"9f19be29-2c84-4ae4-9674-acca4905f4e0\" (UID: \"9f19be29-2c84-4ae4-9674-acca4905f4e0\") " Nov 01 21:16:04.906768 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:04.906491 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe8307db-264d-4b8a-b097-f180bc255a9e-home\" (UniqueName: \"kubernetes.io/empty-dir/fe8307db-264d-4b8a-b097-f180bc255a9e-home\") pod \"fe8307db-264d-4b8a-b097-f180bc255a9e\" (UID: \"fe8307db-264d-4b8a-b097-f180bc255a9e\") " Nov 01 21:16:04.906768 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:04.906540 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe8307db-264d-4b8a-b097-f180bc255a9e-model-cache" pod "fe8307db-264d-4b8a-b097-f180bc255a9e" (UID: "fe8307db-264d-4b8a-b097-f180bc255a9e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:16:04.907189 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:04.906944 4484 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f19be29-2c84-4ae4-9674-acca4905f4e0-home\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:04.907189 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:04.906967 4484 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe8307db-264d-4b8a-b097-f180bc255a9e-model-cache\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:04.907372 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:04.907211 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe8307db-264d-4b8a-b097-f180bc255a9e-home" pod "fe8307db-264d-4b8a-b097-f180bc255a9e" (UID: "fe8307db-264d-4b8a-b097-f180bc255a9e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:16:04.908059 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:04.907896 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f19be29-2c84-4ae4-9674-acca4905f4e0-tmp-dir" pod "9f19be29-2c84-4ae4-9674-acca4905f4e0" (UID: "9f19be29-2c84-4ae4-9674-acca4905f4e0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:16:04.908059 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:04.907979 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f19be29-2c84-4ae4-9674-acca4905f4e0-model-cache" pod "9f19be29-2c84-4ae4-9674-acca4905f4e0" (UID: "9f19be29-2c84-4ae4-9674-acca4905f4e0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:16:04.908059 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:04.908006 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe8307db-264d-4b8a-b097-f180bc255a9e-tmp-dir" pod "fe8307db-264d-4b8a-b097-f180bc255a9e" (UID: "fe8307db-264d-4b8a-b097-f180bc255a9e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:16:04.908950 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:04.908924 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f19be29-2c84-4ae4-9674-acca4905f4e0-tls-certs" pod "9f19be29-2c84-4ae4-9674-acca4905f4e0" (UID: "9f19be29-2c84-4ae4-9674-acca4905f4e0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:16:04.909169 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:04.909150 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f19be29-2c84-4ae4-9674-acca4905f4e0-kube-api-access-9rjtz" pod "9f19be29-2c84-4ae4-9674-acca4905f4e0" (UID: "9f19be29-2c84-4ae4-9674-acca4905f4e0"). InnerVolumeSpecName "kube-api-access-9rjtz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:16:04.909974 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:04.909940 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe8307db-264d-4b8a-b097-f180bc255a9e-dshm" pod "fe8307db-264d-4b8a-b097-f180bc255a9e" (UID: "fe8307db-264d-4b8a-b097-f180bc255a9e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:16:04.910649 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:04.910625 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f19be29-2c84-4ae4-9674-acca4905f4e0-dshm" pod "9f19be29-2c84-4ae4-9674-acca4905f4e0" (UID: "9f19be29-2c84-4ae4-9674-acca4905f4e0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:16:04.910952 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:04.910930 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe8307db-264d-4b8a-b097-f180bc255a9e-tls-certs" pod "fe8307db-264d-4b8a-b097-f180bc255a9e" (UID: "fe8307db-264d-4b8a-b097-f180bc255a9e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:16:04.911284 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:04.911267 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe8307db-264d-4b8a-b097-f180bc255a9e-kube-api-access-p542x" pod "fe8307db-264d-4b8a-b097-f180bc255a9e" (UID: "fe8307db-264d-4b8a-b097-f180bc255a9e"). InnerVolumeSpecName "kube-api-access-p542x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:16:05.007883 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:05.007848 4484 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f19be29-2c84-4ae4-9674-acca4905f4e0-model-cache\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:05.007883 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:05.007880 4484 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f19be29-2c84-4ae4-9674-acca4905f4e0-dshm\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:05.008134 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:05.007893 4484 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe8307db-264d-4b8a-b097-f180bc255a9e-tls-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:05.008134 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:05.007905 4484 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f19be29-2c84-4ae4-9674-acca4905f4e0-tmp-dir\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:05.008134 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:05.007916 4484 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe8307db-264d-4b8a-b097-f180bc255a9e-home\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:05.008134 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:05.007927 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rjtz\" (UniqueName: \"kubernetes.io/projected/9f19be29-2c84-4ae4-9674-acca4905f4e0-kube-api-access-9rjtz\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:05.008134 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:05.007940 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p542x\" (UniqueName: \"kubernetes.io/projected/fe8307db-264d-4b8a-b097-f180bc255a9e-kube-api-access-p542x\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:05.008134 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:05.007954 4484 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f19be29-2c84-4ae4-9674-acca4905f4e0-tls-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:05.008134 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:05.007965 4484 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe8307db-264d-4b8a-b097-f180bc255a9e-dshm\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:05.008134 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:05.007976 4484 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe8307db-264d-4b8a-b097-f180bc255a9e-tmp-dir\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:05.080783 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:05.080748 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:05.508695 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:05.508648 4484 generic.go:356] "Generic (PLEG): container finished" podID="9f19be29-2c84-4ae4-9674-acca4905f4e0" containerID="018a8111200584e1a07c782d1e392d72ba8219e4b5fbb40d840cec6b2b77f167" exitCode=0 Nov 01 21:16:05.509225 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:05.508714 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" Nov 01 21:16:05.509225 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:05.508741 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" event={"ID":"9f19be29-2c84-4ae4-9674-acca4905f4e0","Type":"ContainerDied","Data":"018a8111200584e1a07c782d1e392d72ba8219e4b5fbb40d840cec6b2b77f167"} Nov 01 21:16:05.509225 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:05.508785 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" event={"ID":"9f19be29-2c84-4ae4-9674-acca4905f4e0","Type":"ContainerDied","Data":"c1e8d907efbd3712c3bafcf5bc419865375e826060e678f8ebf7ccad8e7ab00e"} Nov 01 21:16:05.509225 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:05.508808 4484 scope.go:122] "RemoveContainer" containerID="018a8111200584e1a07c782d1e392d72ba8219e4b5fbb40d840cec6b2b77f167" Nov 01 21:16:05.510328 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:05.510261 4484 generic.go:356] "Generic (PLEG): container finished" podID="fe8307db-264d-4b8a-b097-f180bc255a9e" containerID="b5fb24eb8274831d9a71f6e345a01ef166c31017df29796bd931accc2d74eb85" exitCode=0 Nov 01 21:16:05.510328 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:05.510323 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" Nov 01 21:16:05.510493 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:05.510349 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" event={"ID":"fe8307db-264d-4b8a-b097-f180bc255a9e","Type":"ContainerDied","Data":"b5fb24eb8274831d9a71f6e345a01ef166c31017df29796bd931accc2d74eb85"} Nov 01 21:16:05.510493 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:05.510382 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws" event={"ID":"fe8307db-264d-4b8a-b097-f180bc255a9e","Type":"ContainerDied","Data":"ce4d6d7c36f55f57858e9ce63198be9f53a8c46b22dad9aebd173cca6c3300bd"} Nov 01 21:16:05.518840 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:05.518824 4484 scope.go:122] "RemoveContainer" containerID="018a8111200584e1a07c782d1e392d72ba8219e4b5fbb40d840cec6b2b77f167" Nov 01 21:16:05.519116 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:16:05.519096 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"018a8111200584e1a07c782d1e392d72ba8219e4b5fbb40d840cec6b2b77f167\": container with ID starting with 018a8111200584e1a07c782d1e392d72ba8219e4b5fbb40d840cec6b2b77f167 not found: ID does not exist" containerID="018a8111200584e1a07c782d1e392d72ba8219e4b5fbb40d840cec6b2b77f167" Nov 01 21:16:05.519177 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:05.519136 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"018a8111200584e1a07c782d1e392d72ba8219e4b5fbb40d840cec6b2b77f167"} err="failed to get container status \"018a8111200584e1a07c782d1e392d72ba8219e4b5fbb40d840cec6b2b77f167\": rpc error: code = NotFound desc = could not find container \"018a8111200584e1a07c782d1e392d72ba8219e4b5fbb40d840cec6b2b77f167\": container with ID starting with 018a8111200584e1a07c782d1e392d72ba8219e4b5fbb40d840cec6b2b77f167 not found: ID does not exist" Nov 01 21:16:05.519177 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:05.519157 4484 scope.go:122] "RemoveContainer" containerID="b5fb24eb8274831d9a71f6e345a01ef166c31017df29796bd931accc2d74eb85" Nov 01 21:16:05.528316 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:05.528293 4484 scope.go:122] "RemoveContainer" containerID="b5fb24eb8274831d9a71f6e345a01ef166c31017df29796bd931accc2d74eb85" Nov 01 21:16:05.528585 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:16:05.528566 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5fb24eb8274831d9a71f6e345a01ef166c31017df29796bd931accc2d74eb85\": container with ID starting with b5fb24eb8274831d9a71f6e345a01ef166c31017df29796bd931accc2d74eb85 not found: ID does not exist" containerID="b5fb24eb8274831d9a71f6e345a01ef166c31017df29796bd931accc2d74eb85" Nov 01 21:16:05.528654 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:05.528592 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5fb24eb8274831d9a71f6e345a01ef166c31017df29796bd931accc2d74eb85"} err="failed to get container status \"b5fb24eb8274831d9a71f6e345a01ef166c31017df29796bd931accc2d74eb85\": rpc error: code = NotFound desc = could not find container \"b5fb24eb8274831d9a71f6e345a01ef166c31017df29796bd931accc2d74eb85\": container with ID starting with b5fb24eb8274831d9a71f6e345a01ef166c31017df29796bd931accc2d74eb85 not found: ID does not exist" Nov 01 21:16:05.650260 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:05.650159 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q" podUID="9f19be29-2c84-4ae4-9674-acca4905f4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 01 21:16:05.676072 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:05.676037 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws"] Nov 01 21:16:05.726582 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:05.726550 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-1a1e9e/premium-w4-1a1e9e-kserve-f68c4c5d4-qbxws"] Nov 01 21:16:05.790967 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:05.790932 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q"] Nov 01 21:16:05.794306 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:05.794278 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-1a1e9e/distinct-w4-1a1e9e-kserve-754444b64c-z8h6q"] Nov 01 21:16:06.576778 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:06.576727 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:06.735567 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:06.735531 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:06.890229 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:06.890145 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822"] Nov 01 21:16:06.890398 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:06.890362 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822" podUID="4048751a-034d-4ab2-937f-6161505e4166" containerName="istio-proxy" containerID="cri-o://429a4ca14aa787dd0f7b09db887feee8c5d69c7e08a2f527007420308f3cd47e" gracePeriod=30 Nov 01 21:16:06.921281 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:06.921234 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:07.343443 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:07.343399 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:07.374065 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:07.374024 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f19be29-2c84-4ae4-9674-acca4905f4e0" path="/var/lib/kubelet/pods/9f19be29-2c84-4ae4-9674-acca4905f4e0/volumes" Nov 01 21:16:07.374736 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:07.374711 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe8307db-264d-4b8a-b097-f180bc255a9e" path="/var/lib/kubelet/pods/fe8307db-264d-4b8a-b097-f180bc255a9e/volumes" Nov 01 21:16:07.401146 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:07.401099 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-bf3fa2-79cb8b8f-w5wfb"] Nov 01 21:16:07.504517 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:07.504484 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bf3fa2-5f4bb58487-jrpbx"] Nov 01 21:16:07.519599 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:07.519561 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:07.566098 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:07.566046 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1a1e9e-8fc6dcb7-bs6sm" podUID="903377a5-cfac-4d8a-adf0-aa729a3cfdde" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 21:16:07.566272 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:07.566160 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-1a1e9e-66d4f8c5f-cr9fp" podUID="8a9e967b-5887-4361-95ad-9a4122698833" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 21:16:07.804617 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:07.804576 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:07.812697 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:07.812667 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-db0157-7d8ls"] Nov 01 21:16:07.813196 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:07.813182 4484 cpu_manager.go:414] "containerMap: removing container" podUID="09664937-66a9-4626-83a6-3a4667d4dc68" containerName="revoke-keys" Nov 01 21:16:07.813253 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:07.813198 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09664937-66a9-4626-83a6-3a4667d4dc68" containerName="revoke-keys" Nov 01 21:16:07.813253 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:07.813208 4484 cpu_manager.go:414] "containerMap: removing container" podUID="9f19be29-2c84-4ae4-9674-acca4905f4e0" containerName="main" Nov 01 21:16:07.813253 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:07.813213 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f19be29-2c84-4ae4-9674-acca4905f4e0" containerName="main" Nov 01 21:16:07.813253 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:07.813251 4484 cpu_manager.go:414] "containerMap: removing container" podUID="684eee5e-908a-4f3c-9c04-782732b39560" containerName="revoke-keys" Nov 01 21:16:07.813375 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:07.813256 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="684eee5e-908a-4f3c-9c04-782732b39560" containerName="revoke-keys" Nov 01 21:16:07.813375 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:07.813265 4484 cpu_manager.go:414] "containerMap: removing container" podUID="e61432fd-9976-4b09-b225-b9750d67aa3a" containerName="revoke-keys" Nov 01 21:16:07.813375 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:07.813270 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e61432fd-9976-4b09-b225-b9750d67aa3a" containerName="revoke-keys" Nov 01 21:16:07.813375 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:07.813278 4484 cpu_manager.go:414] "containerMap: removing container" podUID="fe8307db-264d-4b8a-b097-f180bc255a9e" containerName="main" Nov 01 21:16:07.813375 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:07.813284 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe8307db-264d-4b8a-b097-f180bc255a9e" containerName="main" Nov 01 21:16:07.813375 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:07.813361 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="684eee5e-908a-4f3c-9c04-782732b39560" containerName="revoke-keys" Nov 01 21:16:07.813375 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:07.813372 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="e61432fd-9976-4b09-b225-b9750d67aa3a" containerName="revoke-keys" Nov 01 21:16:07.813626 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:07.813380 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe8307db-264d-4b8a-b097-f180bc255a9e" containerName="main" Nov 01 21:16:07.813626 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:07.813393 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="09664937-66a9-4626-83a6-3a4667d4dc68" containerName="revoke-keys" Nov 01 21:16:07.813626 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:07.813399 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f19be29-2c84-4ae4-9674-acca4905f4e0" containerName="main" Nov 01 21:16:07.818328 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:07.818307 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-db0157-7d8ls" Nov 01 21:16:07.843417 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:07.843078 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7vgv\" (UniqueName: \"kubernetes.io/projected/48639132-0d2c-4597-b056-68e1766437dd-kube-api-access-z7vgv\") pod \"maas-api-revoke-keys-e2e-worker-w1-db0157-7d8ls\" (UID: \"48639132-0d2c-4597-b056-68e1766437dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-db0157-7d8ls" Nov 01 21:16:07.843417 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:07.843167 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48639132-0d2c-4597-b056-68e1766437dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-db0157-7d8ls\" (UID: \"48639132-0d2c-4597-b056-68e1766437dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-db0157-7d8ls" Nov 01 21:16:07.843417 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:07.843271 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48639132-0d2c-4597-b056-68e1766437dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-db0157-7d8ls\" (UID: \"48639132-0d2c-4597-b056-68e1766437dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-db0157-7d8ls" Nov 01 21:16:07.871965 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:07.871872 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:16:07.871965 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:07.871951 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:16:07.872240 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:07.872031 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:16:07.872240 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:07.872144 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:16:07.873026 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:07.872964 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-db0157-7d8ls"] Nov 01 21:16:07.947067 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:07.944382 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48639132-0d2c-4597-b056-68e1766437dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-db0157-7d8ls\" (UID: \"48639132-0d2c-4597-b056-68e1766437dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-db0157-7d8ls" Nov 01 21:16:07.947067 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:07.944480 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48639132-0d2c-4597-b056-68e1766437dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-db0157-7d8ls\" (UID: \"48639132-0d2c-4597-b056-68e1766437dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-db0157-7d8ls" Nov 01 21:16:07.947067 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:07.944612 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7vgv\" (UniqueName: \"kubernetes.io/projected/48639132-0d2c-4597-b056-68e1766437dd-kube-api-access-z7vgv\") pod \"maas-api-revoke-keys-e2e-worker-w1-db0157-7d8ls\" (UID: \"48639132-0d2c-4597-b056-68e1766437dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-db0157-7d8ls" Nov 01 21:16:07.960530 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:07.959423 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48639132-0d2c-4597-b056-68e1766437dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-db0157-7d8ls\" (UID: \"48639132-0d2c-4597-b056-68e1766437dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-db0157-7d8ls" Nov 01 21:16:07.970073 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:07.967769 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48639132-0d2c-4597-b056-68e1766437dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-db0157-7d8ls\" (UID: \"48639132-0d2c-4597-b056-68e1766437dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-db0157-7d8ls" Nov 01 21:16:07.970817 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:07.970781 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:08.005990 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:08.005961 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7vgv\" (UniqueName: \"kubernetes.io/projected/48639132-0d2c-4597-b056-68e1766437dd-kube-api-access-z7vgv\") pod \"maas-api-revoke-keys-e2e-worker-w1-db0157-7d8ls\" (UID: \"48639132-0d2c-4597-b056-68e1766437dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-db0157-7d8ls" Nov 01 21:16:08.158741 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:08.158645 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-db0157-7d8ls" Nov 01 21:16:08.402303 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:08.402267 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:08.552024 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:08.551981 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-db0157-7d8ls"] Nov 01 21:16:08.553764 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:16:08.553734 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48639132_0d2c_4597_b056_68e1766437dd.slice/crio-16f189b6b65c624e8db316cfe75c4f362b516daeadc555a9c36cedb47397e544 WatchSource:0}: Error finding container 16f189b6b65c624e8db316cfe75c4f362b516daeadc555a9c36cedb47397e544: Status 404 returned error can't find the container with id 16f189b6b65c624e8db316cfe75c4f362b516daeadc555a9c36cedb47397e544 Nov 01 21:16:08.561761 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:08.561735 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:09.555304 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:09.555265 4484 generic.go:356] "Generic (PLEG): container finished" podID="48639132-0d2c-4597-b056-68e1766437dd" containerID="49e41a02b1d29dc4d48163884adc3dcbc1167f6b93e337cfca19325f494ec07a" exitCode=0 Nov 01 21:16:09.555677 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:09.555321 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-db0157-7d8ls" event={"ID":"48639132-0d2c-4597-b056-68e1766437dd","Type":"ContainerDied","Data":"49e41a02b1d29dc4d48163884adc3dcbc1167f6b93e337cfca19325f494ec07a"} Nov 01 21:16:09.555677 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:09.555367 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-db0157-7d8ls" event={"ID":"48639132-0d2c-4597-b056-68e1766437dd","Type":"ContainerStarted","Data":"16f189b6b65c624e8db316cfe75c4f362b516daeadc555a9c36cedb47397e544"} Nov 01 21:16:09.658457 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:09.658414 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:09.873647 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:09.873567 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e67bc43-bs7hs"] Nov 01 21:16:09.877288 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:09.877270 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e67bc43-bs7hs" Nov 01 21:16:09.912052 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:09.912019 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76e99db3-4z8lg"] Nov 01 21:16:09.916050 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:09.916027 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e99db3-4z8lg" Nov 01 21:16:09.932560 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:09.932530 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e67bc43-bs7hs"] Nov 01 21:16:09.964483 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:09.964442 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39899bb0-5323-441a-8267-e70ae6fe4e5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e67bc43-bs7hs\" (UID: \"39899bb0-5323-441a-8267-e70ae6fe4e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e67bc43-bs7hs" Nov 01 21:16:09.964483 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:09.964485 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqr52\" (UniqueName: \"kubernetes.io/projected/39899bb0-5323-441a-8267-e70ae6fe4e5e-kube-api-access-dqr52\") pod \"maas-api-revoke-sub-2e67bc43-bs7hs\" (UID: \"39899bb0-5323-441a-8267-e70ae6fe4e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e67bc43-bs7hs" Nov 01 21:16:09.964737 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:09.964510 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64efda00-ea04-476c-b80e-9deba0bc9827-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76e99db3-4z8lg\" (UID: \"64efda00-ea04-476c-b80e-9deba0bc9827\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e99db3-4z8lg" Nov 01 21:16:09.964737 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:09.964663 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64efda00-ea04-476c-b80e-9deba0bc9827-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76e99db3-4z8lg\" (UID: \"64efda00-ea04-476c-b80e-9deba0bc9827\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e99db3-4z8lg" Nov 01 21:16:09.964737 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:09.964730 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnvxp\" (UniqueName: \"kubernetes.io/projected/64efda00-ea04-476c-b80e-9deba0bc9827-kube-api-access-xnvxp\") pod \"maas-api-revoke-sub-76e99db3-4z8lg\" (UID: \"64efda00-ea04-476c-b80e-9deba0bc9827\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e99db3-4z8lg" Nov 01 21:16:09.964900 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:09.964836 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39899bb0-5323-441a-8267-e70ae6fe4e5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e67bc43-bs7hs\" (UID: \"39899bb0-5323-441a-8267-e70ae6fe4e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e67bc43-bs7hs" Nov 01 21:16:09.981642 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:09.981605 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76e99db3-4z8lg"] Nov 01 21:16:10.065724 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:10.065678 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39899bb0-5323-441a-8267-e70ae6fe4e5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e67bc43-bs7hs\" (UID: \"39899bb0-5323-441a-8267-e70ae6fe4e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e67bc43-bs7hs" Nov 01 21:16:10.065925 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:10.065737 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqr52\" (UniqueName: \"kubernetes.io/projected/39899bb0-5323-441a-8267-e70ae6fe4e5e-kube-api-access-dqr52\") pod \"maas-api-revoke-sub-2e67bc43-bs7hs\" (UID: \"39899bb0-5323-441a-8267-e70ae6fe4e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e67bc43-bs7hs" Nov 01 21:16:10.065925 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:10.065780 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64efda00-ea04-476c-b80e-9deba0bc9827-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76e99db3-4z8lg\" (UID: \"64efda00-ea04-476c-b80e-9deba0bc9827\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e99db3-4z8lg" Nov 01 21:16:10.065925 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:10.065847 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64efda00-ea04-476c-b80e-9deba0bc9827-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76e99db3-4z8lg\" (UID: \"64efda00-ea04-476c-b80e-9deba0bc9827\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e99db3-4z8lg" Nov 01 21:16:10.065925 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:10.065918 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnvxp\" (UniqueName: \"kubernetes.io/projected/64efda00-ea04-476c-b80e-9deba0bc9827-kube-api-access-xnvxp\") pod \"maas-api-revoke-sub-76e99db3-4z8lg\" (UID: \"64efda00-ea04-476c-b80e-9deba0bc9827\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e99db3-4z8lg" Nov 01 21:16:10.066214 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:10.065955 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39899bb0-5323-441a-8267-e70ae6fe4e5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e67bc43-bs7hs\" (UID: \"39899bb0-5323-441a-8267-e70ae6fe4e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e67bc43-bs7hs" Nov 01 21:16:10.066943 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:10.066920 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64efda00-ea04-476c-b80e-9deba0bc9827-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76e99db3-4z8lg\" (UID: \"64efda00-ea04-476c-b80e-9deba0bc9827\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e99db3-4z8lg" Nov 01 21:16:10.067164 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:10.066983 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39899bb0-5323-441a-8267-e70ae6fe4e5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e67bc43-bs7hs\" (UID: \"39899bb0-5323-441a-8267-e70ae6fe4e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e67bc43-bs7hs" Nov 01 21:16:10.068846 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:10.068824 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64efda00-ea04-476c-b80e-9deba0bc9827-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76e99db3-4z8lg\" (UID: \"64efda00-ea04-476c-b80e-9deba0bc9827\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e99db3-4z8lg" Nov 01 21:16:10.068846 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:10.068839 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39899bb0-5323-441a-8267-e70ae6fe4e5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e67bc43-bs7hs\" (UID: \"39899bb0-5323-441a-8267-e70ae6fe4e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e67bc43-bs7hs" Nov 01 21:16:10.086899 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:10.086870 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:10.123895 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:10.123827 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqr52\" (UniqueName: \"kubernetes.io/projected/39899bb0-5323-441a-8267-e70ae6fe4e5e-kube-api-access-dqr52\") pod \"maas-api-revoke-sub-2e67bc43-bs7hs\" (UID: \"39899bb0-5323-441a-8267-e70ae6fe4e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e67bc43-bs7hs" Nov 01 21:16:10.124070 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:10.123951 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnvxp\" (UniqueName: \"kubernetes.io/projected/64efda00-ea04-476c-b80e-9deba0bc9827-kube-api-access-xnvxp\") pod \"maas-api-revoke-sub-76e99db3-4z8lg\" (UID: \"64efda00-ea04-476c-b80e-9deba0bc9827\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e99db3-4z8lg" Nov 01 21:16:10.197157 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:10.197124 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e67bc43-bs7hs" Nov 01 21:16:10.229931 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:10.229896 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e99db3-4z8lg" Nov 01 21:16:10.250288 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:10.250257 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:10.579374 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:10.579337 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e67bc43-bs7hs"] Nov 01 21:16:10.630218 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:10.630183 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76e99db3-4z8lg"] Nov 01 21:16:10.719952 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:10.719927 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-db0157-7d8ls" Nov 01 21:16:10.781158 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:10.781125 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48639132-0d2c-4597-b056-68e1766437dd-kube-api-access-z7vgv\" (UniqueName: \"kubernetes.io/projected/48639132-0d2c-4597-b056-68e1766437dd-kube-api-access-z7vgv\") pod \"48639132-0d2c-4597-b056-68e1766437dd\" (UID: \"48639132-0d2c-4597-b056-68e1766437dd\") " Nov 01 21:16:10.781325 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:10.781275 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48639132-0d2c-4597-b056-68e1766437dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48639132-0d2c-4597-b056-68e1766437dd-maas-api-openshift-ca\") pod \"48639132-0d2c-4597-b056-68e1766437dd\" (UID: \"48639132-0d2c-4597-b056-68e1766437dd\") " Nov 01 21:16:10.781325 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:10.781317 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48639132-0d2c-4597-b056-68e1766437dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48639132-0d2c-4597-b056-68e1766437dd-maas-api-xks-ca\") pod \"48639132-0d2c-4597-b056-68e1766437dd\" (UID: \"48639132-0d2c-4597-b056-68e1766437dd\") " Nov 01 21:16:10.781636 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:10.781612 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48639132-0d2c-4597-b056-68e1766437dd-maas-api-openshift-ca" pod "48639132-0d2c-4597-b056-68e1766437dd" (UID: "48639132-0d2c-4597-b056-68e1766437dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:16:10.783336 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:10.783303 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48639132-0d2c-4597-b056-68e1766437dd-kube-api-access-z7vgv" pod "48639132-0d2c-4597-b056-68e1766437dd" (UID: "48639132-0d2c-4597-b056-68e1766437dd"). InnerVolumeSpecName "kube-api-access-z7vgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:16:10.783475 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:10.783356 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48639132-0d2c-4597-b056-68e1766437dd-maas-api-xks-ca" pod "48639132-0d2c-4597-b056-68e1766437dd" (UID: "48639132-0d2c-4597-b056-68e1766437dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:16:10.882888 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:10.882778 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48639132-0d2c-4597-b056-68e1766437dd-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:10.882888 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:10.882813 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48639132-0d2c-4597-b056-68e1766437dd-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:10.882888 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:10.882822 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7vgv\" (UniqueName: \"kubernetes.io/projected/48639132-0d2c-4597-b056-68e1766437dd-kube-api-access-z7vgv\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:11.396612 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:11.396569 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-208f2b-76b7b8c9f-fxd8f" podUID="2801b6dc-8797-4099-81fa-5fed21c2de2e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 21:16:11.581150 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:11.581121 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-db0157-7d8ls" Nov 01 21:16:11.581484 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:11.581118 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-db0157-7d8ls" event={"ID":"48639132-0d2c-4597-b056-68e1766437dd","Type":"ContainerDied","Data":"16f189b6b65c624e8db316cfe75c4f362b516daeadc555a9c36cedb47397e544"} Nov 01 21:16:11.581484 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:11.581275 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16f189b6b65c624e8db316cfe75c4f362b516daeadc555a9c36cedb47397e544" Nov 01 21:16:11.582823 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:11.582798 4484 generic.go:356] "Generic (PLEG): container finished" podID="64efda00-ea04-476c-b80e-9deba0bc9827" containerID="09021bf263260cd19389c2a6b85e9df13034f21c56eabe380a191b97c4cb058d" exitCode=0 Nov 01 21:16:11.582928 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:11.582865 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e99db3-4z8lg" event={"ID":"64efda00-ea04-476c-b80e-9deba0bc9827","Type":"ContainerDied","Data":"09021bf263260cd19389c2a6b85e9df13034f21c56eabe380a191b97c4cb058d"} Nov 01 21:16:11.582928 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:11.582908 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e99db3-4z8lg" event={"ID":"64efda00-ea04-476c-b80e-9deba0bc9827","Type":"ContainerStarted","Data":"e5a94e8866e5ea37736d5e2931051a8b7ac6654337c68e5a8131f029a1b0d615"} Nov 01 21:16:11.584558 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:11.584536 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e67bc43-bs7hs" event={"ID":"39899bb0-5323-441a-8267-e70ae6fe4e5e","Type":"ContainerStarted","Data":"6f27b48ef9958eb2b6ffa1d3db2ad0ccc8a036a6becf65fbc03a3652873725a2"} Nov 01 21:16:11.584642 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:11.584580 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e67bc43-bs7hs" event={"ID":"39899bb0-5323-441a-8267-e70ae6fe4e5e","Type":"ContainerStarted","Data":"e5dd60efd1e58e6840371a9e73f97cbdd122f0ab1e4b89c59f5eb59dbd99958d"} Nov 01 21:16:11.780532 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:11.780505 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-208f2b-7fb7755ddc-lx76l" Nov 01 21:16:11.799133 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:11.799103 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:11.877304 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:11.877245 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e67bc43-bs7hs" podStartSLOduration=2.8772321720000003 podStartE2EDuration="2.877232172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:16:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:16:11.875950882 +0000 UTC m=+2722.653591017" watchObservedRunningTime="2026-11-01 21:16:11.877232172 +0000 UTC m=+2722.654872307" Nov 01 21:16:11.894577 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:11.894357 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63adc0cc-05ea-4b6a-9f41-cde67a045b02-kube-api-access-mf8dq\" (UniqueName: \"kubernetes.io/projected/63adc0cc-05ea-4b6a-9f41-cde67a045b02-kube-api-access-mf8dq\") pod \"63adc0cc-05ea-4b6a-9f41-cde67a045b02\" (UID: \"63adc0cc-05ea-4b6a-9f41-cde67a045b02\") " Nov 01 21:16:11.894577 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:11.894430 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63adc0cc-05ea-4b6a-9f41-cde67a045b02-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63adc0cc-05ea-4b6a-9f41-cde67a045b02-plugins-config-volume\") pod \"63adc0cc-05ea-4b6a-9f41-cde67a045b02\" (UID: \"63adc0cc-05ea-4b6a-9f41-cde67a045b02\") " Nov 01 21:16:11.895042 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:11.894983 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63adc0cc-05ea-4b6a-9f41-cde67a045b02-plugins-config-volume" pod "63adc0cc-05ea-4b6a-9f41-cde67a045b02" (UID: "63adc0cc-05ea-4b6a-9f41-cde67a045b02"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:16:11.897274 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:11.897249 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63adc0cc-05ea-4b6a-9f41-cde67a045b02-kube-api-access-mf8dq" pod "63adc0cc-05ea-4b6a-9f41-cde67a045b02" (UID: "63adc0cc-05ea-4b6a-9f41-cde67a045b02"). InnerVolumeSpecName "kube-api-access-mf8dq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:16:11.995859 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:11.995827 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mf8dq\" (UniqueName: \"kubernetes.io/projected/63adc0cc-05ea-4b6a-9f41-cde67a045b02-kube-api-access-mf8dq\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:11.995859 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:11.995856 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63adc0cc-05ea-4b6a-9f41-cde67a045b02-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:12.015352 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.015316 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:12.145550 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.145485 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822" Nov 01 21:16:12.164673 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.164640 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:12.197909 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.197868 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4048751a-034d-4ab2-937f-6161505e4166-istio-token\" (UniqueName: \"kubernetes.io/projected/4048751a-034d-4ab2-937f-6161505e4166-istio-token\") pod \"4048751a-034d-4ab2-937f-6161505e4166\" (UID: \"4048751a-034d-4ab2-937f-6161505e4166\") " Nov 01 21:16:12.198123 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.197918 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4048751a-034d-4ab2-937f-6161505e4166-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4048751a-034d-4ab2-937f-6161505e4166-workload-socket\") pod \"4048751a-034d-4ab2-937f-6161505e4166\" (UID: \"4048751a-034d-4ab2-937f-6161505e4166\") " Nov 01 21:16:12.198123 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.197970 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4048751a-034d-4ab2-937f-6161505e4166-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4048751a-034d-4ab2-937f-6161505e4166-istio-data\") pod \"4048751a-034d-4ab2-937f-6161505e4166\" (UID: \"4048751a-034d-4ab2-937f-6161505e4166\") " Nov 01 21:16:12.198123 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.198026 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4048751a-034d-4ab2-937f-6161505e4166-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4048751a-034d-4ab2-937f-6161505e4166-istiod-ca-cert\") pod \"4048751a-034d-4ab2-937f-6161505e4166\" (UID: \"4048751a-034d-4ab2-937f-6161505e4166\") " Nov 01 21:16:12.198123 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.198077 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4048751a-034d-4ab2-937f-6161505e4166-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4048751a-034d-4ab2-937f-6161505e4166-workload-certs\") pod \"4048751a-034d-4ab2-937f-6161505e4166\" (UID: \"4048751a-034d-4ab2-937f-6161505e4166\") " Nov 01 21:16:12.198123 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.198110 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4048751a-034d-4ab2-937f-6161505e4166-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4048751a-034d-4ab2-937f-6161505e4166-istio-podinfo\") pod \"4048751a-034d-4ab2-937f-6161505e4166\" (UID: \"4048751a-034d-4ab2-937f-6161505e4166\") " Nov 01 21:16:12.198475 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.198218 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4048751a-034d-4ab2-937f-6161505e4166-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4048751a-034d-4ab2-937f-6161505e4166-istio-envoy\") pod \"4048751a-034d-4ab2-937f-6161505e4166\" (UID: \"4048751a-034d-4ab2-937f-6161505e4166\") " Nov 01 21:16:12.198475 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.198246 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4048751a-034d-4ab2-937f-6161505e4166-workload-socket" pod "4048751a-034d-4ab2-937f-6161505e4166" (UID: "4048751a-034d-4ab2-937f-6161505e4166"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:16:12.198475 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.198293 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4048751a-034d-4ab2-937f-6161505e4166-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4048751a-034d-4ab2-937f-6161505e4166-credential-socket\") pod \"4048751a-034d-4ab2-937f-6161505e4166\" (UID: \"4048751a-034d-4ab2-937f-6161505e4166\") " Nov 01 21:16:12.198475 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.198326 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4048751a-034d-4ab2-937f-6161505e4166-kube-api-access-9lrw2\" (UniqueName: \"kubernetes.io/projected/4048751a-034d-4ab2-937f-6161505e4166-kube-api-access-9lrw2\") pod \"4048751a-034d-4ab2-937f-6161505e4166\" (UID: \"4048751a-034d-4ab2-937f-6161505e4166\") " Nov 01 21:16:12.198683 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.198530 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4048751a-034d-4ab2-937f-6161505e4166-workload-certs" pod "4048751a-034d-4ab2-937f-6161505e4166" (UID: "4048751a-034d-4ab2-937f-6161505e4166"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:16:12.198683 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.198545 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4048751a-034d-4ab2-937f-6161505e4166-istio-data" pod "4048751a-034d-4ab2-937f-6161505e4166" (UID: "4048751a-034d-4ab2-937f-6161505e4166"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:16:12.198683 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.198535 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4048751a-034d-4ab2-937f-6161505e4166-istiod-ca-cert" pod "4048751a-034d-4ab2-937f-6161505e4166" (UID: "4048751a-034d-4ab2-937f-6161505e4166"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:16:12.198839 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.198779 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4048751a-034d-4ab2-937f-6161505e4166-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:12.198839 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.198801 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4048751a-034d-4ab2-937f-6161505e4166-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:12.198839 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.198809 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4048751a-034d-4ab2-937f-6161505e4166-credential-socket" pod "4048751a-034d-4ab2-937f-6161505e4166" (UID: "4048751a-034d-4ab2-937f-6161505e4166"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:16:12.198839 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.198815 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4048751a-034d-4ab2-937f-6161505e4166-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:12.199019 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.198857 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4048751a-034d-4ab2-937f-6161505e4166-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:12.200637 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.200602 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4048751a-034d-4ab2-937f-6161505e4166-istio-token" pod "4048751a-034d-4ab2-937f-6161505e4166" (UID: "4048751a-034d-4ab2-937f-6161505e4166"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:16:12.200756 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.200645 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4048751a-034d-4ab2-937f-6161505e4166-kube-api-access-9lrw2" pod "4048751a-034d-4ab2-937f-6161505e4166" (UID: "4048751a-034d-4ab2-937f-6161505e4166"). InnerVolumeSpecName "kube-api-access-9lrw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:16:12.200897 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.200880 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4048751a-034d-4ab2-937f-6161505e4166-istio-envoy" pod "4048751a-034d-4ab2-937f-6161505e4166" (UID: "4048751a-034d-4ab2-937f-6161505e4166"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:16:12.200897 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.200888 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4048751a-034d-4ab2-937f-6161505e4166-istio-podinfo" pod "4048751a-034d-4ab2-937f-6161505e4166" (UID: "4048751a-034d-4ab2-937f-6161505e4166"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:16:12.250148 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.250116 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bf3fa2-7xf2w"] Nov 01 21:16:12.252349 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.252328 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bf3fa2-7xf2w"] Nov 01 21:16:12.283206 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.283174 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4"] Nov 01 21:16:12.299966 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.299937 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4048751a-034d-4ab2-937f-6161505e4166-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:12.299966 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.299965 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4048751a-034d-4ab2-937f-6161505e4166-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:12.300166 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.299980 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lrw2\" (UniqueName: \"kubernetes.io/projected/4048751a-034d-4ab2-937f-6161505e4166-kube-api-access-9lrw2\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:12.300166 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.300007 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4048751a-034d-4ab2-937f-6161505e4166-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:12.300166 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.300021 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4048751a-034d-4ab2-937f-6161505e4166-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:12.401534 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.401432 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bf3fa2-79cb8b8f-w5wfb" podUID="1aca0497-55c4-4de2-8a73-5a72ba9054b2" containerName="payload-processing" containerID="cri-o://878105ee2f079c980b4969f87d59112adea8e0795fde80d710b749e15ad6af99" gracePeriod=25 Nov 01 21:16:12.402840 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.402754 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:12.505585 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.505531 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bf3fa2-5f4bb58487-jrpbx" podUID="84fa582f-9eaf-4d54-9684-87e33f3f8b81" containerName="payload-pre-processing" containerID="cri-o://c3d0df010b25403eda246c44b289954a35c4bd0e39d76fd94b11d915951d7b91" gracePeriod=25 Nov 01 21:16:12.595718 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.595678 4484 generic.go:356] "Generic (PLEG): container finished" podID="63adc0cc-05ea-4b6a-9f41-cde67a045b02" containerID="854406f946db48c116ed03210fd1fedcda582d58c9168c829649feb819a89755" exitCode=0 Nov 01 21:16:12.596213 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.595772 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-208f2b-7fb7755ddc-lx76l" event={"ID":"63adc0cc-05ea-4b6a-9f41-cde67a045b02","Type":"ContainerDied","Data":"854406f946db48c116ed03210fd1fedcda582d58c9168c829649feb819a89755"} Nov 01 21:16:12.596213 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.595828 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-208f2b-7fb7755ddc-lx76l" event={"ID":"63adc0cc-05ea-4b6a-9f41-cde67a045b02","Type":"ContainerDied","Data":"f31a3842b344296b2a48c23b617d5b94aa4171179e3fb73db05d01a59e4015a0"} Nov 01 21:16:12.596213 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.595859 4484 scope.go:122] "RemoveContainer" containerID="854406f946db48c116ed03210fd1fedcda582d58c9168c829649feb819a89755" Nov 01 21:16:12.596213 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.596080 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-208f2b-7fb7755ddc-lx76l" Nov 01 21:16:12.602563 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.602150 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:12.603714 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.603347 4484 generic.go:356] "Generic (PLEG): container finished" podID="39899bb0-5323-441a-8267-e70ae6fe4e5e" containerID="6f27b48ef9958eb2b6ffa1d3db2ad0ccc8a036a6becf65fbc03a3652873725a2" exitCode=0 Nov 01 21:16:12.603714 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.603420 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e67bc43-bs7hs" event={"ID":"39899bb0-5323-441a-8267-e70ae6fe4e5e","Type":"ContainerDied","Data":"6f27b48ef9958eb2b6ffa1d3db2ad0ccc8a036a6becf65fbc03a3652873725a2"} Nov 01 21:16:12.607513 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.607486 4484 generic.go:356] "Generic (PLEG): container finished" podID="2801b6dc-8797-4099-81fa-5fed21c2de2e" containerID="61f951c2913180db98752f89fbbb4d4af848cce81f6e65e421fe55d201799d2b" exitCode=0 Nov 01 21:16:12.607633 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.607607 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-208f2b-76b7b8c9f-fxd8f" event={"ID":"2801b6dc-8797-4099-81fa-5fed21c2de2e","Type":"ContainerDied","Data":"61f951c2913180db98752f89fbbb4d4af848cce81f6e65e421fe55d201799d2b"} Nov 01 21:16:12.612396 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.610304 4484 generic.go:356] "Generic (PLEG): container finished" podID="1aca0497-55c4-4de2-8a73-5a72ba9054b2" containerID="878105ee2f079c980b4969f87d59112adea8e0795fde80d710b749e15ad6af99" exitCode=0 Nov 01 21:16:12.612396 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.610384 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bf3fa2-79cb8b8f-w5wfb" event={"ID":"1aca0497-55c4-4de2-8a73-5a72ba9054b2","Type":"ContainerDied","Data":"878105ee2f079c980b4969f87d59112adea8e0795fde80d710b749e15ad6af99"} Nov 01 21:16:12.618019 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.613957 4484 generic.go:356] "Generic (PLEG): container finished" podID="4048751a-034d-4ab2-937f-6161505e4166" containerID="429a4ca14aa787dd0f7b09db887feee8c5d69c7e08a2f527007420308f3cd47e" exitCode=0 Nov 01 21:16:12.618019 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.614277 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822" Nov 01 21:16:12.618369 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.618266 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822" event={"ID":"4048751a-034d-4ab2-937f-6161505e4166","Type":"ContainerDied","Data":"429a4ca14aa787dd0f7b09db887feee8c5d69c7e08a2f527007420308f3cd47e"} Nov 01 21:16:12.618369 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.618328 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822" event={"ID":"4048751a-034d-4ab2-937f-6161505e4166","Type":"ContainerDied","Data":"de1e21d1da6c4685d968730c466b672600e5c1c820fcee804503625619455ad8"} Nov 01 21:16:12.630252 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.630229 4484 scope.go:122] "RemoveContainer" containerID="854406f946db48c116ed03210fd1fedcda582d58c9168c829649feb819a89755" Nov 01 21:16:12.630834 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:16:12.630800 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"854406f946db48c116ed03210fd1fedcda582d58c9168c829649feb819a89755\": container with ID starting with 854406f946db48c116ed03210fd1fedcda582d58c9168c829649feb819a89755 not found: ID does not exist" containerID="854406f946db48c116ed03210fd1fedcda582d58c9168c829649feb819a89755" Nov 01 21:16:12.631024 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.630846 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"854406f946db48c116ed03210fd1fedcda582d58c9168c829649feb819a89755"} err="failed to get container status \"854406f946db48c116ed03210fd1fedcda582d58c9168c829649feb819a89755\": rpc error: code = NotFound desc = could not find container \"854406f946db48c116ed03210fd1fedcda582d58c9168c829649feb819a89755\": container with ID starting with 854406f946db48c116ed03210fd1fedcda582d58c9168c829649feb819a89755 not found: ID does not exist" Nov 01 21:16:12.631024 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.630871 4484 scope.go:122] "RemoveContainer" containerID="429a4ca14aa787dd0f7b09db887feee8c5d69c7e08a2f527007420308f3cd47e" Nov 01 21:16:12.647554 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.647527 4484 scope.go:122] "RemoveContainer" containerID="20c6f9f9c84adcb86aad2f2a31f2cfad160d309e9a25f92974e5c257bf5bfc04" Nov 01 21:16:12.669301 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.668292 4484 scope.go:122] "RemoveContainer" containerID="429a4ca14aa787dd0f7b09db887feee8c5d69c7e08a2f527007420308f3cd47e" Nov 01 21:16:12.670915 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:16:12.670829 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"429a4ca14aa787dd0f7b09db887feee8c5d69c7e08a2f527007420308f3cd47e\": container with ID starting with 429a4ca14aa787dd0f7b09db887feee8c5d69c7e08a2f527007420308f3cd47e not found: ID does not exist" containerID="429a4ca14aa787dd0f7b09db887feee8c5d69c7e08a2f527007420308f3cd47e" Nov 01 21:16:12.670915 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.670868 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"429a4ca14aa787dd0f7b09db887feee8c5d69c7e08a2f527007420308f3cd47e"} err="failed to get container status \"429a4ca14aa787dd0f7b09db887feee8c5d69c7e08a2f527007420308f3cd47e\": rpc error: code = NotFound desc = could not find container \"429a4ca14aa787dd0f7b09db887feee8c5d69c7e08a2f527007420308f3cd47e\": container with ID starting with 429a4ca14aa787dd0f7b09db887feee8c5d69c7e08a2f527007420308f3cd47e not found: ID does not exist" Nov 01 21:16:12.670915 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.670895 4484 scope.go:122] "RemoveContainer" containerID="20c6f9f9c84adcb86aad2f2a31f2cfad160d309e9a25f92974e5c257bf5bfc04" Nov 01 21:16:12.671392 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:16:12.671312 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20c6f9f9c84adcb86aad2f2a31f2cfad160d309e9a25f92974e5c257bf5bfc04\": container with ID starting with 20c6f9f9c84adcb86aad2f2a31f2cfad160d309e9a25f92974e5c257bf5bfc04 not found: ID does not exist" containerID="20c6f9f9c84adcb86aad2f2a31f2cfad160d309e9a25f92974e5c257bf5bfc04" Nov 01 21:16:12.671392 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.671346 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20c6f9f9c84adcb86aad2f2a31f2cfad160d309e9a25f92974e5c257bf5bfc04"} err="failed to get container status \"20c6f9f9c84adcb86aad2f2a31f2cfad160d309e9a25f92974e5c257bf5bfc04\": rpc error: code = NotFound desc = could not find container \"20c6f9f9c84adcb86aad2f2a31f2cfad160d309e9a25f92974e5c257bf5bfc04\": container with ID starting with 20c6f9f9c84adcb86aad2f2a31f2cfad160d309e9a25f92974e5c257bf5bfc04 not found: ID does not exist" Nov 01 21:16:12.687828 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.687800 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-208f2b-76b7b8c9f-fxd8f" Nov 01 21:16:12.706927 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.705080 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bf3fa2-79cb8b8f-w5wfb" Nov 01 21:16:12.771347 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.771313 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-208f2b-7fb7755ddc-lx76l"] Nov 01 21:16:12.775252 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.775219 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-208f2b-7fb7755ddc-lx76l"] Nov 01 21:16:12.798931 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.798905 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e99db3-4z8lg" Nov 01 21:16:12.805371 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.805345 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1aca0497-55c4-4de2-8a73-5a72ba9054b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aca0497-55c4-4de2-8a73-5a72ba9054b2-plugins-config-volume\") pod \"1aca0497-55c4-4de2-8a73-5a72ba9054b2\" (UID: \"1aca0497-55c4-4de2-8a73-5a72ba9054b2\") " Nov 01 21:16:12.805530 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.805446 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2801b6dc-8797-4099-81fa-5fed21c2de2e-kube-api-access-zt64r\" (UniqueName: \"kubernetes.io/projected/2801b6dc-8797-4099-81fa-5fed21c2de2e-kube-api-access-zt64r\") pod \"2801b6dc-8797-4099-81fa-5fed21c2de2e\" (UID: \"2801b6dc-8797-4099-81fa-5fed21c2de2e\") " Nov 01 21:16:12.805530 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.805504 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2801b6dc-8797-4099-81fa-5fed21c2de2e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2801b6dc-8797-4099-81fa-5fed21c2de2e-plugins-config-volume\") pod \"2801b6dc-8797-4099-81fa-5fed21c2de2e\" (UID: \"2801b6dc-8797-4099-81fa-5fed21c2de2e\") " Nov 01 21:16:12.805637 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.805571 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aca0497-55c4-4de2-8a73-5a72ba9054b2-kube-api-access-kp5n2\" (UniqueName: \"kubernetes.io/projected/1aca0497-55c4-4de2-8a73-5a72ba9054b2-kube-api-access-kp5n2\") pod \"1aca0497-55c4-4de2-8a73-5a72ba9054b2\" (UID: \"1aca0497-55c4-4de2-8a73-5a72ba9054b2\") " Nov 01 21:16:12.805727 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.805701 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aca0497-55c4-4de2-8a73-5a72ba9054b2-plugins-config-volume" pod "1aca0497-55c4-4de2-8a73-5a72ba9054b2" (UID: "1aca0497-55c4-4de2-8a73-5a72ba9054b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:16:12.806022 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.805978 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2801b6dc-8797-4099-81fa-5fed21c2de2e-plugins-config-volume" pod "2801b6dc-8797-4099-81fa-5fed21c2de2e" (UID: "2801b6dc-8797-4099-81fa-5fed21c2de2e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:16:12.806157 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.806088 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2801b6dc-8797-4099-81fa-5fed21c2de2e-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:12.806157 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.806107 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aca0497-55c4-4de2-8a73-5a72ba9054b2-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:12.807744 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.807721 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aca0497-55c4-4de2-8a73-5a72ba9054b2-kube-api-access-kp5n2" pod "1aca0497-55c4-4de2-8a73-5a72ba9054b2" (UID: "1aca0497-55c4-4de2-8a73-5a72ba9054b2"). InnerVolumeSpecName "kube-api-access-kp5n2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:16:12.807848 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.807812 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2801b6dc-8797-4099-81fa-5fed21c2de2e-kube-api-access-zt64r" pod "2801b6dc-8797-4099-81fa-5fed21c2de2e" (UID: "2801b6dc-8797-4099-81fa-5fed21c2de2e"). InnerVolumeSpecName "kube-api-access-zt64r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:16:12.817351 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.817328 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bf3fa2-5f4bb58487-jrpbx" Nov 01 21:16:12.906829 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.906795 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64efda00-ea04-476c-b80e-9deba0bc9827-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64efda00-ea04-476c-b80e-9deba0bc9827-maas-api-xks-ca\") pod \"64efda00-ea04-476c-b80e-9deba0bc9827\" (UID: \"64efda00-ea04-476c-b80e-9deba0bc9827\") " Nov 01 21:16:12.907062 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.906933 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64efda00-ea04-476c-b80e-9deba0bc9827-kube-api-access-xnvxp\" (UniqueName: \"kubernetes.io/projected/64efda00-ea04-476c-b80e-9deba0bc9827-kube-api-access-xnvxp\") pod \"64efda00-ea04-476c-b80e-9deba0bc9827\" (UID: \"64efda00-ea04-476c-b80e-9deba0bc9827\") " Nov 01 21:16:12.907062 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.906955 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64efda00-ea04-476c-b80e-9deba0bc9827-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64efda00-ea04-476c-b80e-9deba0bc9827-maas-api-openshift-ca\") pod \"64efda00-ea04-476c-b80e-9deba0bc9827\" (UID: \"64efda00-ea04-476c-b80e-9deba0bc9827\") " Nov 01 21:16:12.907062 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.906979 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84fa582f-9eaf-4d54-9684-87e33f3f8b81-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84fa582f-9eaf-4d54-9684-87e33f3f8b81-plugins-config-volume\") pod \"84fa582f-9eaf-4d54-9684-87e33f3f8b81\" (UID: \"84fa582f-9eaf-4d54-9684-87e33f3f8b81\") " Nov 01 21:16:12.907062 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.907022 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84fa582f-9eaf-4d54-9684-87e33f3f8b81-kube-api-access-ltsrb\" (UniqueName: \"kubernetes.io/projected/84fa582f-9eaf-4d54-9684-87e33f3f8b81-kube-api-access-ltsrb\") pod \"84fa582f-9eaf-4d54-9684-87e33f3f8b81\" (UID: \"84fa582f-9eaf-4d54-9684-87e33f3f8b81\") " Nov 01 21:16:12.907406 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.907381 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp5n2\" (UniqueName: \"kubernetes.io/projected/1aca0497-55c4-4de2-8a73-5a72ba9054b2-kube-api-access-kp5n2\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:12.907489 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.907404 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64efda00-ea04-476c-b80e-9deba0bc9827-maas-api-openshift-ca" pod "64efda00-ea04-476c-b80e-9deba0bc9827" (UID: "64efda00-ea04-476c-b80e-9deba0bc9827"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:16:12.907489 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.907410 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt64r\" (UniqueName: \"kubernetes.io/projected/2801b6dc-8797-4099-81fa-5fed21c2de2e-kube-api-access-zt64r\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:12.907489 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.907428 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84fa582f-9eaf-4d54-9684-87e33f3f8b81-plugins-config-volume" pod "84fa582f-9eaf-4d54-9684-87e33f3f8b81" (UID: "84fa582f-9eaf-4d54-9684-87e33f3f8b81"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:16:12.909231 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.909199 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84fa582f-9eaf-4d54-9684-87e33f3f8b81-kube-api-access-ltsrb" pod "84fa582f-9eaf-4d54-9684-87e33f3f8b81" (UID: "84fa582f-9eaf-4d54-9684-87e33f3f8b81"). InnerVolumeSpecName "kube-api-access-ltsrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:16:12.909231 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.909203 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64efda00-ea04-476c-b80e-9deba0bc9827-maas-api-xks-ca" pod "64efda00-ea04-476c-b80e-9deba0bc9827" (UID: "64efda00-ea04-476c-b80e-9deba0bc9827"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:16:12.909231 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:12.909227 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64efda00-ea04-476c-b80e-9deba0bc9827-kube-api-access-xnvxp" pod "64efda00-ea04-476c-b80e-9deba0bc9827" (UID: "64efda00-ea04-476c-b80e-9deba0bc9827"). InnerVolumeSpecName "kube-api-access-xnvxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:16:13.008288 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:13.008252 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnvxp\" (UniqueName: \"kubernetes.io/projected/64efda00-ea04-476c-b80e-9deba0bc9827-kube-api-access-xnvxp\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:13.008288 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:13.008283 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64efda00-ea04-476c-b80e-9deba0bc9827-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:13.008288 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:13.008292 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84fa582f-9eaf-4d54-9684-87e33f3f8b81-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:13.008550 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:13.008302 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltsrb\" (UniqueName: \"kubernetes.io/projected/84fa582f-9eaf-4d54-9684-87e33f3f8b81-kube-api-access-ltsrb\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:13.008550 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:13.008313 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64efda00-ea04-476c-b80e-9deba0bc9827-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:13.030479 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:13.030448 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822"] Nov 01 21:16:13.049472 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:13.049437 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-df700a-openshift-default-6cf7fb69dc-dg822"] Nov 01 21:16:13.371046 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:13.370948 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09664937-66a9-4626-83a6-3a4667d4dc68" path="/var/lib/kubelet/pods/09664937-66a9-4626-83a6-3a4667d4dc68/volumes" Nov 01 21:16:13.371460 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:13.371445 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4048751a-034d-4ab2-937f-6161505e4166" path="/var/lib/kubelet/pods/4048751a-034d-4ab2-937f-6161505e4166/volumes" Nov 01 21:16:13.371800 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:13.371788 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63adc0cc-05ea-4b6a-9f41-cde67a045b02" path="/var/lib/kubelet/pods/63adc0cc-05ea-4b6a-9f41-cde67a045b02/volumes" Nov 01 21:16:13.622177 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:13.622087 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e99db3-4z8lg" Nov 01 21:16:13.622177 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:13.622091 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e99db3-4z8lg" event={"ID":"64efda00-ea04-476c-b80e-9deba0bc9827","Type":"ContainerDied","Data":"e5a94e8866e5ea37736d5e2931051a8b7ac6654337c68e5a8131f029a1b0d615"} Nov 01 21:16:13.622698 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:13.622212 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5a94e8866e5ea37736d5e2931051a8b7ac6654337c68e5a8131f029a1b0d615" Nov 01 21:16:13.624793 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:13.624762 4484 generic.go:356] "Generic (PLEG): container finished" podID="84fa582f-9eaf-4d54-9684-87e33f3f8b81" containerID="c3d0df010b25403eda246c44b289954a35c4bd0e39d76fd94b11d915951d7b91" exitCode=0 Nov 01 21:16:13.624981 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:13.624812 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bf3fa2-5f4bb58487-jrpbx" Nov 01 21:16:13.624981 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:13.624848 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bf3fa2-5f4bb58487-jrpbx" event={"ID":"84fa582f-9eaf-4d54-9684-87e33f3f8b81","Type":"ContainerDied","Data":"c3d0df010b25403eda246c44b289954a35c4bd0e39d76fd94b11d915951d7b91"} Nov 01 21:16:13.624981 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:13.624908 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bf3fa2-5f4bb58487-jrpbx" event={"ID":"84fa582f-9eaf-4d54-9684-87e33f3f8b81","Type":"ContainerDied","Data":"e8dc7c77b30cfebc282b5e3c06af00d8abbe30d6b846134946d4f34850b59bab"} Nov 01 21:16:13.624981 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:13.624940 4484 scope.go:122] "RemoveContainer" containerID="c3d0df010b25403eda246c44b289954a35c4bd0e39d76fd94b11d915951d7b91" Nov 01 21:16:13.626431 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:13.626416 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-208f2b-76b7b8c9f-fxd8f" Nov 01 21:16:13.626539 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:13.626475 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-208f2b-76b7b8c9f-fxd8f" event={"ID":"2801b6dc-8797-4099-81fa-5fed21c2de2e","Type":"ContainerDied","Data":"0792c2a279cf4edcb1cbee17984c1f47fe62f625552f315e41f4924bfd7a3920"} Nov 01 21:16:13.628474 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:13.628457 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bf3fa2-79cb8b8f-w5wfb" Nov 01 21:16:13.628595 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:13.628476 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bf3fa2-79cb8b8f-w5wfb" event={"ID":"1aca0497-55c4-4de2-8a73-5a72ba9054b2","Type":"ContainerDied","Data":"793dec16ac7d3d6b1c29326ecbb430847ef82326c13cb6b051c5b8e0ebd74b3b"} Nov 01 21:16:13.640249 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:13.640219 4484 scope.go:122] "RemoveContainer" containerID="c3d0df010b25403eda246c44b289954a35c4bd0e39d76fd94b11d915951d7b91" Nov 01 21:16:13.640589 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:16:13.640567 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3d0df010b25403eda246c44b289954a35c4bd0e39d76fd94b11d915951d7b91\": container with ID starting with c3d0df010b25403eda246c44b289954a35c4bd0e39d76fd94b11d915951d7b91 not found: ID does not exist" containerID="c3d0df010b25403eda246c44b289954a35c4bd0e39d76fd94b11d915951d7b91" Nov 01 21:16:13.640660 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:13.640600 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3d0df010b25403eda246c44b289954a35c4bd0e39d76fd94b11d915951d7b91"} err="failed to get container status \"c3d0df010b25403eda246c44b289954a35c4bd0e39d76fd94b11d915951d7b91\": rpc error: code = NotFound desc = could not find container \"c3d0df010b25403eda246c44b289954a35c4bd0e39d76fd94b11d915951d7b91\": container with ID starting with c3d0df010b25403eda246c44b289954a35c4bd0e39d76fd94b11d915951d7b91 not found: ID does not exist" Nov 01 21:16:13.640660 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:13.640621 4484 scope.go:122] "RemoveContainer" containerID="61f951c2913180db98752f89fbbb4d4af848cce81f6e65e421fe55d201799d2b" Nov 01 21:16:13.651043 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:13.650991 4484 scope.go:122] "RemoveContainer" containerID="878105ee2f079c980b4969f87d59112adea8e0795fde80d710b749e15ad6af99" Nov 01 21:16:13.791544 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:13.791500 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e67bc43-bs7hs" Nov 01 21:16:13.815370 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:13.815330 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39899bb0-5323-441a-8267-e70ae6fe4e5e-kube-api-access-dqr52\" (UniqueName: \"kubernetes.io/projected/39899bb0-5323-441a-8267-e70ae6fe4e5e-kube-api-access-dqr52\") pod \"39899bb0-5323-441a-8267-e70ae6fe4e5e\" (UID: \"39899bb0-5323-441a-8267-e70ae6fe4e5e\") " Nov 01 21:16:13.815530 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:13.815496 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39899bb0-5323-441a-8267-e70ae6fe4e5e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39899bb0-5323-441a-8267-e70ae6fe4e5e-maas-api-xks-ca\") pod \"39899bb0-5323-441a-8267-e70ae6fe4e5e\" (UID: \"39899bb0-5323-441a-8267-e70ae6fe4e5e\") " Nov 01 21:16:13.815590 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:13.815575 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39899bb0-5323-441a-8267-e70ae6fe4e5e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39899bb0-5323-441a-8267-e70ae6fe4e5e-maas-api-openshift-ca\") pod \"39899bb0-5323-441a-8267-e70ae6fe4e5e\" (UID: \"39899bb0-5323-441a-8267-e70ae6fe4e5e\") " Nov 01 21:16:13.816027 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:13.815964 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39899bb0-5323-441a-8267-e70ae6fe4e5e-maas-api-openshift-ca" pod "39899bb0-5323-441a-8267-e70ae6fe4e5e" (UID: "39899bb0-5323-441a-8267-e70ae6fe4e5e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:16:13.817538 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:13.817510 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39899bb0-5323-441a-8267-e70ae6fe4e5e-maas-api-xks-ca" pod "39899bb0-5323-441a-8267-e70ae6fe4e5e" (UID: "39899bb0-5323-441a-8267-e70ae6fe4e5e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:16:13.817538 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:13.817528 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39899bb0-5323-441a-8267-e70ae6fe4e5e-kube-api-access-dqr52" pod "39899bb0-5323-441a-8267-e70ae6fe4e5e" (UID: "39899bb0-5323-441a-8267-e70ae6fe4e5e"). InnerVolumeSpecName "kube-api-access-dqr52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:16:13.846253 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:13.846216 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bf3fa2-5f4bb58487-jrpbx"] Nov 01 21:16:13.851559 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:13.851528 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bf3fa2-5f4bb58487-jrpbx"] Nov 01 21:16:13.916605 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:13.916524 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39899bb0-5323-441a-8267-e70ae6fe4e5e-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:13.916605 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:13.916555 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqr52\" (UniqueName: \"kubernetes.io/projected/39899bb0-5323-441a-8267-e70ae6fe4e5e-kube-api-access-dqr52\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:13.916605 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:13.916565 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39899bb0-5323-441a-8267-e70ae6fe4e5e-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:13.964021 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:13.963963 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-bf3fa2-79cb8b8f-w5wfb"] Nov 01 21:16:13.966296 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:13.966262 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-bf3fa2-79cb8b8f-w5wfb"] Nov 01 21:16:14.077108 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:14.077075 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-208f2b-76b7b8c9f-fxd8f"] Nov 01 21:16:14.081073 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:14.081043 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-208f2b-76b7b8c9f-fxd8f"] Nov 01 21:16:14.638786 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:14.638749 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e67bc43-bs7hs" Nov 01 21:16:14.639303 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:14.638750 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e67bc43-bs7hs" event={"ID":"39899bb0-5323-441a-8267-e70ae6fe4e5e","Type":"ContainerDied","Data":"e5dd60efd1e58e6840371a9e73f97cbdd122f0ab1e4b89c59f5eb59dbd99958d"} Nov 01 21:16:14.639303 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:14.638889 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5dd60efd1e58e6840371a9e73f97cbdd122f0ab1e4b89c59f5eb59dbd99958d" Nov 01 21:16:15.255839 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:15.255805 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:15.370112 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:15.370077 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1aca0497-55c4-4de2-8a73-5a72ba9054b2" path="/var/lib/kubelet/pods/1aca0497-55c4-4de2-8a73-5a72ba9054b2/volumes" Nov 01 21:16:15.370499 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:15.370485 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2801b6dc-8797-4099-81fa-5fed21c2de2e" path="/var/lib/kubelet/pods/2801b6dc-8797-4099-81fa-5fed21c2de2e/volumes" Nov 01 21:16:15.370842 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:15.370829 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84fa582f-9eaf-4d54-9684-87e33f3f8b81" path="/var/lib/kubelet/pods/84fa582f-9eaf-4d54-9684-87e33f3f8b81/volumes" Nov 01 21:16:15.424276 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:15.424241 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:15.767596 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:16:15.767560 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:16:16.099859 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:16.099771 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:16.260966 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:16.260926 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:16.657065 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:16.657021 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:17.566540 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:17.566454 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-1a1e9e-66d4f8c5f-cr9fp" podUID="8a9e967b-5887-4361-95ad-9a4122698833" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 21:16:17.566540 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:17.566534 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1a1e9e-8fc6dcb7-bs6sm" podUID="903377a5-cfac-4d8a-adf0-aa729a3cfdde" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 21:16:18.317341 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:18.317302 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:18.479027 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:18.478979 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:18.644323 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:18.644198 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:18.767214 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:18.767182 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk"] Nov 01 21:16:18.767417 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:18.767395 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk" podUID="b77b5e66-a42b-4800-a34f-d415b42913d1" containerName="istio-proxy" containerID="cri-o://f15450a4c0bce764238a93f2cc364b4cc4b3d91b94f98e1de4b43ecafac0a2f3" gracePeriod=30 Nov 01 21:16:18.810443 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:18.810409 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:18.857956 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:18.857920 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1a1e9e-p4bw8"] Nov 01 21:16:18.860823 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:18.860792 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1a1e9e-p4bw8"] Nov 01 21:16:19.202864 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:19.202821 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:19.370068 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:19.370035 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:19.372317 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:19.370517 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="395ed984-c93b-415a-8f5b-72c79c4e4f1a" path="/var/lib/kubelet/pods/395ed984-c93b-415a-8f5b-72c79c4e4f1a/volumes" Nov 01 21:16:19.658501 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:19.658466 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-944263-cbd5k"] Nov 01 21:16:19.658925 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:19.658912 4484 cpu_manager.go:414] "containerMap: removing container" podUID="84fa582f-9eaf-4d54-9684-87e33f3f8b81" containerName="payload-pre-processing" Nov 01 21:16:19.658982 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:19.658926 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84fa582f-9eaf-4d54-9684-87e33f3f8b81" containerName="payload-pre-processing" Nov 01 21:16:19.658982 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:19.658936 4484 cpu_manager.go:414] "containerMap: removing container" podUID="63adc0cc-05ea-4b6a-9f41-cde67a045b02" containerName="payload-processing" Nov 01 21:16:19.658982 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:19.658941 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63adc0cc-05ea-4b6a-9f41-cde67a045b02" containerName="payload-processing" Nov 01 21:16:19.658982 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:19.658948 4484 cpu_manager.go:414] "containerMap: removing container" podUID="1aca0497-55c4-4de2-8a73-5a72ba9054b2" containerName="payload-processing" Nov 01 21:16:19.658982 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:19.658953 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aca0497-55c4-4de2-8a73-5a72ba9054b2" containerName="payload-processing" Nov 01 21:16:19.658982 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:19.658965 4484 cpu_manager.go:414] "containerMap: removing container" podUID="2801b6dc-8797-4099-81fa-5fed21c2de2e" containerName="payload-pre-processing" Nov 01 21:16:19.658982 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:19.658970 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2801b6dc-8797-4099-81fa-5fed21c2de2e" containerName="payload-pre-processing" Nov 01 21:16:19.658982 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:19.658981 4484 cpu_manager.go:414] "containerMap: removing container" podUID="4048751a-034d-4ab2-937f-6161505e4166" containerName="istio-proxy" Nov 01 21:16:19.659278 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:19.658986 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4048751a-034d-4ab2-937f-6161505e4166" containerName="istio-proxy" Nov 01 21:16:19.659278 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:19.659017 4484 cpu_manager.go:414] "containerMap: removing container" podUID="48639132-0d2c-4597-b056-68e1766437dd" containerName="revoke-keys" Nov 01 21:16:19.659278 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:19.659022 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48639132-0d2c-4597-b056-68e1766437dd" containerName="revoke-keys" Nov 01 21:16:19.659278 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:19.659033 4484 cpu_manager.go:414] "containerMap: removing container" podUID="64efda00-ea04-476c-b80e-9deba0bc9827" containerName="revoke-keys" Nov 01 21:16:19.659278 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:19.659038 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64efda00-ea04-476c-b80e-9deba0bc9827" containerName="revoke-keys" Nov 01 21:16:19.659278 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:19.659048 4484 cpu_manager.go:414] "containerMap: removing container" podUID="39899bb0-5323-441a-8267-e70ae6fe4e5e" containerName="revoke-keys" Nov 01 21:16:19.659278 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:19.659053 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39899bb0-5323-441a-8267-e70ae6fe4e5e" containerName="revoke-keys" Nov 01 21:16:19.659278 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:19.659115 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="84fa582f-9eaf-4d54-9684-87e33f3f8b81" containerName="payload-pre-processing" Nov 01 21:16:19.659278 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:19.659123 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="63adc0cc-05ea-4b6a-9f41-cde67a045b02" containerName="payload-processing" Nov 01 21:16:19.659278 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:19.659129 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aca0497-55c4-4de2-8a73-5a72ba9054b2" containerName="payload-processing" Nov 01 21:16:19.659278 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:19.659136 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="2801b6dc-8797-4099-81fa-5fed21c2de2e" containerName="payload-pre-processing" Nov 01 21:16:19.659278 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:19.659144 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="4048751a-034d-4ab2-937f-6161505e4166" containerName="istio-proxy" Nov 01 21:16:19.659278 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:19.659151 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="48639132-0d2c-4597-b056-68e1766437dd" containerName="revoke-keys" Nov 01 21:16:19.659278 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:19.659157 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="64efda00-ea04-476c-b80e-9deba0bc9827" containerName="revoke-keys" Nov 01 21:16:19.659278 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:19.659163 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="4048751a-034d-4ab2-937f-6161505e4166" containerName="istio-proxy" Nov 01 21:16:19.659278 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:19.659170 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="39899bb0-5323-441a-8267-e70ae6fe4e5e" containerName="revoke-keys" Nov 01 21:16:19.662579 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:19.662557 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-944263-cbd5k" Nov 01 21:16:19.716731 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:19.716700 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:16:19.716942 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:19.716785 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:16:19.717146 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:19.717125 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:16:19.719527 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:19.719500 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-944263-cbd5k"] Nov 01 21:16:19.767077 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:19.767042 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:16:19.785506 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:19.785465 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a67a9b61-9e37-4294-bcc6-fbdd25ab4346-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-944263-cbd5k\" (UID: \"a67a9b61-9e37-4294-bcc6-fbdd25ab4346\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-944263-cbd5k" Nov 01 21:16:19.785506 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:19.785513 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89lpj\" (UniqueName: \"kubernetes.io/projected/a67a9b61-9e37-4294-bcc6-fbdd25ab4346-kube-api-access-89lpj\") pod \"maas-api-revoke-keys-e2e-worker-w3-944263-cbd5k\" (UID: \"a67a9b61-9e37-4294-bcc6-fbdd25ab4346\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-944263-cbd5k" Nov 01 21:16:19.785764 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:19.785657 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a67a9b61-9e37-4294-bcc6-fbdd25ab4346-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-944263-cbd5k\" (UID: \"a67a9b61-9e37-4294-bcc6-fbdd25ab4346\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-944263-cbd5k" Nov 01 21:16:19.887011 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:19.886955 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a67a9b61-9e37-4294-bcc6-fbdd25ab4346-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-944263-cbd5k\" (UID: \"a67a9b61-9e37-4294-bcc6-fbdd25ab4346\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-944263-cbd5k" Nov 01 21:16:19.887234 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:19.887044 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89lpj\" (UniqueName: \"kubernetes.io/projected/a67a9b61-9e37-4294-bcc6-fbdd25ab4346-kube-api-access-89lpj\") pod \"maas-api-revoke-keys-e2e-worker-w3-944263-cbd5k\" (UID: \"a67a9b61-9e37-4294-bcc6-fbdd25ab4346\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-944263-cbd5k" Nov 01 21:16:19.887234 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:19.887120 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a67a9b61-9e37-4294-bcc6-fbdd25ab4346-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-944263-cbd5k\" (UID: \"a67a9b61-9e37-4294-bcc6-fbdd25ab4346\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-944263-cbd5k" Nov 01 21:16:19.887788 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:19.887763 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a67a9b61-9e37-4294-bcc6-fbdd25ab4346-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-944263-cbd5k\" (UID: \"a67a9b61-9e37-4294-bcc6-fbdd25ab4346\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-944263-cbd5k" Nov 01 21:16:19.889408 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:19.889383 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a67a9b61-9e37-4294-bcc6-fbdd25ab4346-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-944263-cbd5k\" (UID: \"a67a9b61-9e37-4294-bcc6-fbdd25ab4346\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-944263-cbd5k" Nov 01 21:16:19.945311 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:19.945238 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89lpj\" (UniqueName: \"kubernetes.io/projected/a67a9b61-9e37-4294-bcc6-fbdd25ab4346-kube-api-access-89lpj\") pod \"maas-api-revoke-keys-e2e-worker-w3-944263-cbd5k\" (UID: \"a67a9b61-9e37-4294-bcc6-fbdd25ab4346\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-944263-cbd5k" Nov 01 21:16:19.978369 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:19.978332 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-944263-cbd5k" Nov 01 21:16:20.352483 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:20.352455 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-944263-cbd5k"] Nov 01 21:16:20.353723 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:16:20.353693 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda67a9b61_9e37_4294_bcc6_fbdd25ab4346.slice/crio-2531c2fd98f73f046e7e845de946737b33bed4c7273cadaa8ff2e5e080d97270 WatchSource:0}: Error finding container 2531c2fd98f73f046e7e845de946737b33bed4c7273cadaa8ff2e5e080d97270: Status 404 returned error can't find the container with id 2531c2fd98f73f046e7e845de946737b33bed4c7273cadaa8ff2e5e080d97270 Nov 01 21:16:20.684073 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:20.684029 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-944263-cbd5k" event={"ID":"a67a9b61-9e37-4294-bcc6-fbdd25ab4346","Type":"ContainerStarted","Data":"bf7cd31a6c28f30e1df47bb433e7bb3cf4cbd7bc43b2818898c15cbc1fcf0915"} Nov 01 21:16:20.684496 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:20.684091 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-944263-cbd5k" event={"ID":"a67a9b61-9e37-4294-bcc6-fbdd25ab4346","Type":"ContainerStarted","Data":"2531c2fd98f73f046e7e845de946737b33bed4c7273cadaa8ff2e5e080d97270"} Nov 01 21:16:20.800244 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:20.800189 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-944263-cbd5k" podStartSLOduration=1.8001766240000001 podStartE2EDuration="1.800176624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:16:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:16:20.797946558 +0000 UTC m=+2731.575586686" watchObservedRunningTime="2026-11-01 21:16:20.800176624 +0000 UTC m=+2731.577816758" Nov 01 21:16:21.594918 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:21.594883 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-db0157-654558c4b7-vsbx9"] Nov 01 21:16:21.691717 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:21.691681 4484 generic.go:356] "Generic (PLEG): container finished" podID="a67a9b61-9e37-4294-bcc6-fbdd25ab4346" containerID="bf7cd31a6c28f30e1df47bb433e7bb3cf4cbd7bc43b2818898c15cbc1fcf0915" exitCode=0 Nov 01 21:16:21.692135 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:21.691719 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-944263-cbd5k" event={"ID":"a67a9b61-9e37-4294-bcc6-fbdd25ab4346","Type":"ContainerDied","Data":"bf7cd31a6c28f30e1df47bb433e7bb3cf4cbd7bc43b2818898c15cbc1fcf0915"} Nov 01 21:16:21.703265 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:21.703228 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-db0157-6689cd59bc-nd7kc"] Nov 01 21:16:21.984302 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:21.984212 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:22.152048 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:22.152013 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:22.914436 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:22.914410 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-944263-cbd5k" Nov 01 21:16:22.918804 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:22.918774 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-1a1e9e-66d4f8c5f-cr9fp" Nov 01 21:16:23.018798 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:23.018769 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a67a9b61-9e37-4294-bcc6-fbdd25ab4346-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a67a9b61-9e37-4294-bcc6-fbdd25ab4346-maas-api-openshift-ca\") pod \"a67a9b61-9e37-4294-bcc6-fbdd25ab4346\" (UID: \"a67a9b61-9e37-4294-bcc6-fbdd25ab4346\") " Nov 01 21:16:23.018958 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:23.018851 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a67a9b61-9e37-4294-bcc6-fbdd25ab4346-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a67a9b61-9e37-4294-bcc6-fbdd25ab4346-maas-api-xks-ca\") pod \"a67a9b61-9e37-4294-bcc6-fbdd25ab4346\" (UID: \"a67a9b61-9e37-4294-bcc6-fbdd25ab4346\") " Nov 01 21:16:23.018958 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:23.018921 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a9e967b-5887-4361-95ad-9a4122698833-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a9e967b-5887-4361-95ad-9a4122698833-plugins-config-volume\") pod \"8a9e967b-5887-4361-95ad-9a4122698833\" (UID: \"8a9e967b-5887-4361-95ad-9a4122698833\") " Nov 01 21:16:23.019116 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:23.019015 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a9e967b-5887-4361-95ad-9a4122698833-kube-api-access-rn4s2\" (UniqueName: \"kubernetes.io/projected/8a9e967b-5887-4361-95ad-9a4122698833-kube-api-access-rn4s2\") pod \"8a9e967b-5887-4361-95ad-9a4122698833\" (UID: \"8a9e967b-5887-4361-95ad-9a4122698833\") " Nov 01 21:16:23.019116 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:23.019085 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a67a9b61-9e37-4294-bcc6-fbdd25ab4346-kube-api-access-89lpj\" (UniqueName: \"kubernetes.io/projected/a67a9b61-9e37-4294-bcc6-fbdd25ab4346-kube-api-access-89lpj\") pod \"a67a9b61-9e37-4294-bcc6-fbdd25ab4346\" (UID: \"a67a9b61-9e37-4294-bcc6-fbdd25ab4346\") " Nov 01 21:16:23.019241 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:23.019116 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a67a9b61-9e37-4294-bcc6-fbdd25ab4346-maas-api-openshift-ca" pod "a67a9b61-9e37-4294-bcc6-fbdd25ab4346" (UID: "a67a9b61-9e37-4294-bcc6-fbdd25ab4346"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:16:23.019351 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:23.019332 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a9e967b-5887-4361-95ad-9a4122698833-plugins-config-volume" pod "8a9e967b-5887-4361-95ad-9a4122698833" (UID: "8a9e967b-5887-4361-95ad-9a4122698833"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:16:23.019452 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:23.019433 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a9e967b-5887-4361-95ad-9a4122698833-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:23.019512 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:23.019455 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a67a9b61-9e37-4294-bcc6-fbdd25ab4346-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:23.021112 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:23.021087 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a67a9b61-9e37-4294-bcc6-fbdd25ab4346-maas-api-xks-ca" pod "a67a9b61-9e37-4294-bcc6-fbdd25ab4346" (UID: "a67a9b61-9e37-4294-bcc6-fbdd25ab4346"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:16:23.021234 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:23.021160 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a9e967b-5887-4361-95ad-9a4122698833-kube-api-access-rn4s2" pod "8a9e967b-5887-4361-95ad-9a4122698833" (UID: "8a9e967b-5887-4361-95ad-9a4122698833"). InnerVolumeSpecName "kube-api-access-rn4s2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:16:23.021446 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:23.021425 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a67a9b61-9e37-4294-bcc6-fbdd25ab4346-kube-api-access-89lpj" pod "a67a9b61-9e37-4294-bcc6-fbdd25ab4346" (UID: "a67a9b61-9e37-4294-bcc6-fbdd25ab4346"). InnerVolumeSpecName "kube-api-access-89lpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:16:23.026216 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:23.026194 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1a1e9e-8fc6dcb7-bs6sm" Nov 01 21:16:23.120412 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:23.120378 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/903377a5-cfac-4d8a-adf0-aa729a3cfdde-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/903377a5-cfac-4d8a-adf0-aa729a3cfdde-plugins-config-volume\") pod \"903377a5-cfac-4d8a-adf0-aa729a3cfdde\" (UID: \"903377a5-cfac-4d8a-adf0-aa729a3cfdde\") " Nov 01 21:16:23.120592 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:23.120509 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/903377a5-cfac-4d8a-adf0-aa729a3cfdde-kube-api-access-lsq2l\" (UniqueName: \"kubernetes.io/projected/903377a5-cfac-4d8a-adf0-aa729a3cfdde-kube-api-access-lsq2l\") pod \"903377a5-cfac-4d8a-adf0-aa729a3cfdde\" (UID: \"903377a5-cfac-4d8a-adf0-aa729a3cfdde\") " Nov 01 21:16:23.120760 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:23.120736 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/903377a5-cfac-4d8a-adf0-aa729a3cfdde-plugins-config-volume" pod "903377a5-cfac-4d8a-adf0-aa729a3cfdde" (UID: "903377a5-cfac-4d8a-adf0-aa729a3cfdde"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:16:23.120876 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:23.120860 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/903377a5-cfac-4d8a-adf0-aa729a3cfdde-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:23.120917 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:23.120881 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a67a9b61-9e37-4294-bcc6-fbdd25ab4346-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:23.120917 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:23.120891 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rn4s2\" (UniqueName: \"kubernetes.io/projected/8a9e967b-5887-4361-95ad-9a4122698833-kube-api-access-rn4s2\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:23.120917 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:23.120899 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89lpj\" (UniqueName: \"kubernetes.io/projected/a67a9b61-9e37-4294-bcc6-fbdd25ab4346-kube-api-access-89lpj\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:23.122563 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:23.122541 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/903377a5-cfac-4d8a-adf0-aa729a3cfdde-kube-api-access-lsq2l" pod "903377a5-cfac-4d8a-adf0-aa729a3cfdde" (UID: "903377a5-cfac-4d8a-adf0-aa729a3cfdde"). InnerVolumeSpecName "kube-api-access-lsq2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:16:23.222402 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:23.222317 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsq2l\" (UniqueName: \"kubernetes.io/projected/903377a5-cfac-4d8a-adf0-aa729a3cfdde-kube-api-access-lsq2l\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:23.705750 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:23.705718 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-944263-cbd5k" Nov 01 21:16:23.705949 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:23.705750 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-944263-cbd5k" event={"ID":"a67a9b61-9e37-4294-bcc6-fbdd25ab4346","Type":"ContainerDied","Data":"2531c2fd98f73f046e7e845de946737b33bed4c7273cadaa8ff2e5e080d97270"} Nov 01 21:16:23.705949 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:23.705801 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2531c2fd98f73f046e7e845de946737b33bed4c7273cadaa8ff2e5e080d97270" Nov 01 21:16:23.707482 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:23.707453 4484 generic.go:356] "Generic (PLEG): container finished" podID="903377a5-cfac-4d8a-adf0-aa729a3cfdde" containerID="c18048df1d8d86a50759e6376198bc7db34272c3c122c1e3070a9671b7b67572" exitCode=0 Nov 01 21:16:23.707634 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:23.707506 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1a1e9e-8fc6dcb7-bs6sm" Nov 01 21:16:23.707634 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:23.707549 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1a1e9e-8fc6dcb7-bs6sm" event={"ID":"903377a5-cfac-4d8a-adf0-aa729a3cfdde","Type":"ContainerDied","Data":"c18048df1d8d86a50759e6376198bc7db34272c3c122c1e3070a9671b7b67572"} Nov 01 21:16:23.707634 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:23.707596 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1a1e9e-8fc6dcb7-bs6sm" event={"ID":"903377a5-cfac-4d8a-adf0-aa729a3cfdde","Type":"ContainerDied","Data":"477a68da7bfef68ff662b749645af0bc0bb32c37b47175944ab698cba8952e8a"} Nov 01 21:16:23.707634 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:23.707624 4484 scope.go:122] "RemoveContainer" containerID="c18048df1d8d86a50759e6376198bc7db34272c3c122c1e3070a9671b7b67572" Nov 01 21:16:23.710627 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:23.710598 4484 generic.go:356] "Generic (PLEG): container finished" podID="8a9e967b-5887-4361-95ad-9a4122698833" containerID="964c4d7650f59a65aa7a22dc715d5f3763ebf12890a3c110838a4f3f43ca558c" exitCode=0 Nov 01 21:16:23.710761 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:23.710642 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-1a1e9e-66d4f8c5f-cr9fp" Nov 01 21:16:23.710761 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:23.710652 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-1a1e9e-66d4f8c5f-cr9fp" event={"ID":"8a9e967b-5887-4361-95ad-9a4122698833","Type":"ContainerDied","Data":"964c4d7650f59a65aa7a22dc715d5f3763ebf12890a3c110838a4f3f43ca558c"} Nov 01 21:16:23.710761 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:23.710684 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-1a1e9e-66d4f8c5f-cr9fp" event={"ID":"8a9e967b-5887-4361-95ad-9a4122698833","Type":"ContainerDied","Data":"ce683eca5367ccd60c9a8e9325bdbb3118685b5031ab0b2c4107d455fd3eb7c1"} Nov 01 21:16:23.719751 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:23.719725 4484 scope.go:122] "RemoveContainer" containerID="c18048df1d8d86a50759e6376198bc7db34272c3c122c1e3070a9671b7b67572" Nov 01 21:16:23.720087 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:16:23.720065 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c18048df1d8d86a50759e6376198bc7db34272c3c122c1e3070a9671b7b67572\": container with ID starting with c18048df1d8d86a50759e6376198bc7db34272c3c122c1e3070a9671b7b67572 not found: ID does not exist" containerID="c18048df1d8d86a50759e6376198bc7db34272c3c122c1e3070a9671b7b67572" Nov 01 21:16:23.720172 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:23.720097 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c18048df1d8d86a50759e6376198bc7db34272c3c122c1e3070a9671b7b67572"} err="failed to get container status \"c18048df1d8d86a50759e6376198bc7db34272c3c122c1e3070a9671b7b67572\": rpc error: code = NotFound desc = could not find container \"c18048df1d8d86a50759e6376198bc7db34272c3c122c1e3070a9671b7b67572\": container with ID starting with c18048df1d8d86a50759e6376198bc7db34272c3c122c1e3070a9671b7b67572 not found: ID does not exist" Nov 01 21:16:23.720172 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:23.720118 4484 scope.go:122] "RemoveContainer" containerID="964c4d7650f59a65aa7a22dc715d5f3763ebf12890a3c110838a4f3f43ca558c" Nov 01 21:16:23.729940 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:23.729915 4484 scope.go:122] "RemoveContainer" containerID="964c4d7650f59a65aa7a22dc715d5f3763ebf12890a3c110838a4f3f43ca558c" Nov 01 21:16:23.730290 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:16:23.730267 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"964c4d7650f59a65aa7a22dc715d5f3763ebf12890a3c110838a4f3f43ca558c\": container with ID starting with 964c4d7650f59a65aa7a22dc715d5f3763ebf12890a3c110838a4f3f43ca558c not found: ID does not exist" containerID="964c4d7650f59a65aa7a22dc715d5f3763ebf12890a3c110838a4f3f43ca558c" Nov 01 21:16:23.730349 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:23.730300 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"964c4d7650f59a65aa7a22dc715d5f3763ebf12890a3c110838a4f3f43ca558c"} err="failed to get container status \"964c4d7650f59a65aa7a22dc715d5f3763ebf12890a3c110838a4f3f43ca558c\": rpc error: code = NotFound desc = could not find container \"964c4d7650f59a65aa7a22dc715d5f3763ebf12890a3c110838a4f3f43ca558c\": container with ID starting with 964c4d7650f59a65aa7a22dc715d5f3763ebf12890a3c110838a4f3f43ca558c not found: ID does not exist" Nov 01 21:16:23.732030 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:23.732007 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:23.926981 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:23.926945 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-1a1e9e-66d4f8c5f-cr9fp"] Nov 01 21:16:23.930678 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:23.930649 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-1a1e9e-66d4f8c5f-cr9fp"] Nov 01 21:16:24.028769 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:24.028741 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk" Nov 01 21:16:24.043765 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:24.043737 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-1a1e9e-8fc6dcb7-bs6sm"] Nov 01 21:16:24.045794 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:24.045774 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-1a1e9e-8fc6dcb7-bs6sm"] Nov 01 21:16:24.129426 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:24.129391 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b77b5e66-a42b-4800-a34f-d415b42913d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b77b5e66-a42b-4800-a34f-d415b42913d1-istio-data\") pod \"b77b5e66-a42b-4800-a34f-d415b42913d1\" (UID: \"b77b5e66-a42b-4800-a34f-d415b42913d1\") " Nov 01 21:16:24.129636 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:24.129440 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b77b5e66-a42b-4800-a34f-d415b42913d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b77b5e66-a42b-4800-a34f-d415b42913d1-istio-envoy\") pod \"b77b5e66-a42b-4800-a34f-d415b42913d1\" (UID: \"b77b5e66-a42b-4800-a34f-d415b42913d1\") " Nov 01 21:16:24.129636 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:24.129468 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b77b5e66-a42b-4800-a34f-d415b42913d1-istio-token\" (UniqueName: \"kubernetes.io/projected/b77b5e66-a42b-4800-a34f-d415b42913d1-istio-token\") pod \"b77b5e66-a42b-4800-a34f-d415b42913d1\" (UID: \"b77b5e66-a42b-4800-a34f-d415b42913d1\") " Nov 01 21:16:24.129636 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:24.129547 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b77b5e66-a42b-4800-a34f-d415b42913d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b77b5e66-a42b-4800-a34f-d415b42913d1-istiod-ca-cert\") pod \"b77b5e66-a42b-4800-a34f-d415b42913d1\" (UID: \"b77b5e66-a42b-4800-a34f-d415b42913d1\") " Nov 01 21:16:24.129636 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:24.129574 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b77b5e66-a42b-4800-a34f-d415b42913d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b77b5e66-a42b-4800-a34f-d415b42913d1-credential-socket\") pod \"b77b5e66-a42b-4800-a34f-d415b42913d1\" (UID: \"b77b5e66-a42b-4800-a34f-d415b42913d1\") " Nov 01 21:16:24.129636 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:24.129597 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b77b5e66-a42b-4800-a34f-d415b42913d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b77b5e66-a42b-4800-a34f-d415b42913d1-istio-podinfo\") pod \"b77b5e66-a42b-4800-a34f-d415b42913d1\" (UID: \"b77b5e66-a42b-4800-a34f-d415b42913d1\") " Nov 01 21:16:24.129636 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:24.129617 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b77b5e66-a42b-4800-a34f-d415b42913d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b77b5e66-a42b-4800-a34f-d415b42913d1-workload-certs\") pod \"b77b5e66-a42b-4800-a34f-d415b42913d1\" (UID: \"b77b5e66-a42b-4800-a34f-d415b42913d1\") " Nov 01 21:16:24.130481 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:24.129647 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b77b5e66-a42b-4800-a34f-d415b42913d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b77b5e66-a42b-4800-a34f-d415b42913d1-workload-socket\") pod \"b77b5e66-a42b-4800-a34f-d415b42913d1\" (UID: \"b77b5e66-a42b-4800-a34f-d415b42913d1\") " Nov 01 21:16:24.130481 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:24.129693 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b77b5e66-a42b-4800-a34f-d415b42913d1-kube-api-access-kf6b5\" (UniqueName: \"kubernetes.io/projected/b77b5e66-a42b-4800-a34f-d415b42913d1-kube-api-access-kf6b5\") pod \"b77b5e66-a42b-4800-a34f-d415b42913d1\" (UID: \"b77b5e66-a42b-4800-a34f-d415b42913d1\") " Nov 01 21:16:24.130481 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:24.129703 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b77b5e66-a42b-4800-a34f-d415b42913d1-istio-data" pod "b77b5e66-a42b-4800-a34f-d415b42913d1" (UID: "b77b5e66-a42b-4800-a34f-d415b42913d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:16:24.130481 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:24.129943 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b77b5e66-a42b-4800-a34f-d415b42913d1-credential-socket" pod "b77b5e66-a42b-4800-a34f-d415b42913d1" (UID: "b77b5e66-a42b-4800-a34f-d415b42913d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:16:24.130481 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:24.129966 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b77b5e66-a42b-4800-a34f-d415b42913d1-workload-certs" pod "b77b5e66-a42b-4800-a34f-d415b42913d1" (UID: "b77b5e66-a42b-4800-a34f-d415b42913d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:16:24.130481 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:24.129974 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b77b5e66-a42b-4800-a34f-d415b42913d1-workload-socket" pod "b77b5e66-a42b-4800-a34f-d415b42913d1" (UID: "b77b5e66-a42b-4800-a34f-d415b42913d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:16:24.130481 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:24.130042 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b77b5e66-a42b-4800-a34f-d415b42913d1-istiod-ca-cert" pod "b77b5e66-a42b-4800-a34f-d415b42913d1" (UID: "b77b5e66-a42b-4800-a34f-d415b42913d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:16:24.130481 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:24.130118 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b77b5e66-a42b-4800-a34f-d415b42913d1-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:24.130481 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:24.130134 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b77b5e66-a42b-4800-a34f-d415b42913d1-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:24.130481 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:24.130145 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b77b5e66-a42b-4800-a34f-d415b42913d1-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:24.130481 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:24.130158 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b77b5e66-a42b-4800-a34f-d415b42913d1-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:24.130481 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:24.130173 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b77b5e66-a42b-4800-a34f-d415b42913d1-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:24.131790 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:24.131763 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b77b5e66-a42b-4800-a34f-d415b42913d1-istio-token" pod "b77b5e66-a42b-4800-a34f-d415b42913d1" (UID: "b77b5e66-a42b-4800-a34f-d415b42913d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:16:24.131906 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:24.131847 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b77b5e66-a42b-4800-a34f-d415b42913d1-istio-podinfo" pod "b77b5e66-a42b-4800-a34f-d415b42913d1" (UID: "b77b5e66-a42b-4800-a34f-d415b42913d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:16:24.131906 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:24.131874 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b77b5e66-a42b-4800-a34f-d415b42913d1-kube-api-access-kf6b5" pod "b77b5e66-a42b-4800-a34f-d415b42913d1" (UID: "b77b5e66-a42b-4800-a34f-d415b42913d1"). InnerVolumeSpecName "kube-api-access-kf6b5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:16:24.131989 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:24.131900 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b77b5e66-a42b-4800-a34f-d415b42913d1-istio-envoy" pod "b77b5e66-a42b-4800-a34f-d415b42913d1" (UID: "b77b5e66-a42b-4800-a34f-d415b42913d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:16:24.145513 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:24.145493 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq"] Nov 01 21:16:24.230989 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:24.230899 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b77b5e66-a42b-4800-a34f-d415b42913d1-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:24.230989 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:24.230931 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf6b5\" (UniqueName: \"kubernetes.io/projected/b77b5e66-a42b-4800-a34f-d415b42913d1-kube-api-access-kf6b5\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:24.230989 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:24.230945 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b77b5e66-a42b-4800-a34f-d415b42913d1-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:24.230989 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:24.230957 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b77b5e66-a42b-4800-a34f-d415b42913d1-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:24.345458 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:24.345425 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:24.718505 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:24.718468 4484 generic.go:356] "Generic (PLEG): container finished" podID="b77b5e66-a42b-4800-a34f-d415b42913d1" containerID="f15450a4c0bce764238a93f2cc364b4cc4b3d91b94f98e1de4b43ecafac0a2f3" exitCode=0 Nov 01 21:16:24.718732 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:24.718521 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk" Nov 01 21:16:24.718732 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:24.718552 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk" event={"ID":"b77b5e66-a42b-4800-a34f-d415b42913d1","Type":"ContainerDied","Data":"f15450a4c0bce764238a93f2cc364b4cc4b3d91b94f98e1de4b43ecafac0a2f3"} Nov 01 21:16:24.718732 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:24.718611 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk" event={"ID":"b77b5e66-a42b-4800-a34f-d415b42913d1","Type":"ContainerDied","Data":"bfb7d3e450cffee87dc6f9068aac1007f012e8b40f25cd79f86ee53750eb0d82"} Nov 01 21:16:24.718732 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:24.718645 4484 scope.go:122] "RemoveContainer" containerID="f15450a4c0bce764238a93f2cc364b4cc4b3d91b94f98e1de4b43ecafac0a2f3" Nov 01 21:16:24.729567 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:24.729541 4484 scope.go:122] "RemoveContainer" containerID="4afcb3d14e0d6c7c7e19fce05ade68290b13463bc828c8fb42f69d54a03804dd" Nov 01 21:16:24.738892 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:24.738870 4484 scope.go:122] "RemoveContainer" containerID="f15450a4c0bce764238a93f2cc364b4cc4b3d91b94f98e1de4b43ecafac0a2f3" Nov 01 21:16:24.739232 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:16:24.739211 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f15450a4c0bce764238a93f2cc364b4cc4b3d91b94f98e1de4b43ecafac0a2f3\": container with ID starting with f15450a4c0bce764238a93f2cc364b4cc4b3d91b94f98e1de4b43ecafac0a2f3 not found: ID does not exist" containerID="f15450a4c0bce764238a93f2cc364b4cc4b3d91b94f98e1de4b43ecafac0a2f3" Nov 01 21:16:24.739313 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:24.739241 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f15450a4c0bce764238a93f2cc364b4cc4b3d91b94f98e1de4b43ecafac0a2f3"} err="failed to get container status \"f15450a4c0bce764238a93f2cc364b4cc4b3d91b94f98e1de4b43ecafac0a2f3\": rpc error: code = NotFound desc = could not find container \"f15450a4c0bce764238a93f2cc364b4cc4b3d91b94f98e1de4b43ecafac0a2f3\": container with ID starting with f15450a4c0bce764238a93f2cc364b4cc4b3d91b94f98e1de4b43ecafac0a2f3 not found: ID does not exist" Nov 01 21:16:24.739313 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:24.739261 4484 scope.go:122] "RemoveContainer" containerID="4afcb3d14e0d6c7c7e19fce05ade68290b13463bc828c8fb42f69d54a03804dd" Nov 01 21:16:24.739575 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:16:24.739549 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4afcb3d14e0d6c7c7e19fce05ade68290b13463bc828c8fb42f69d54a03804dd\": container with ID starting with 4afcb3d14e0d6c7c7e19fce05ade68290b13463bc828c8fb42f69d54a03804dd not found: ID does not exist" containerID="4afcb3d14e0d6c7c7e19fce05ade68290b13463bc828c8fb42f69d54a03804dd" Nov 01 21:16:24.739636 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:24.739585 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4afcb3d14e0d6c7c7e19fce05ade68290b13463bc828c8fb42f69d54a03804dd"} err="failed to get container status \"4afcb3d14e0d6c7c7e19fce05ade68290b13463bc828c8fb42f69d54a03804dd\": rpc error: code = NotFound desc = could not find container \"4afcb3d14e0d6c7c7e19fce05ade68290b13463bc828c8fb42f69d54a03804dd\": container with ID starting with 4afcb3d14e0d6c7c7e19fce05ade68290b13463bc828c8fb42f69d54a03804dd not found: ID does not exist" Nov 01 21:16:24.886595 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:24.886558 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk"] Nov 01 21:16:24.937389 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:24.937350 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bf3fa2-openshift-default-69d7c5c975-84bmk"] Nov 01 21:16:25.099313 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:25.099276 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:25.270089 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:25.270053 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:25.371350 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:25.371277 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a9e967b-5887-4361-95ad-9a4122698833" path="/var/lib/kubelet/pods/8a9e967b-5887-4361-95ad-9a4122698833/volumes" Nov 01 21:16:25.371701 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:25.371688 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="903377a5-cfac-4d8a-adf0-aa729a3cfdde" path="/var/lib/kubelet/pods/903377a5-cfac-4d8a-adf0-aa729a3cfdde/volumes" Nov 01 21:16:25.372133 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:25.372121 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b77b5e66-a42b-4800-a34f-d415b42913d1" path="/var/lib/kubelet/pods/b77b5e66-a42b-4800-a34f-d415b42913d1/volumes" Nov 01 21:16:25.428222 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:25.428185 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:26.595237 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:26.595189 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-db0157-654558c4b7-vsbx9" podUID="0b416214-5015-4e36-89e4-e9dab20238fd" containerName="payload-processing" containerID="cri-o://8e5e0b63de1ddb3e6a9856b1de46170f993606739250ef98c40b1cb9dd7416ed" gracePeriod=25 Nov 01 21:16:26.704207 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:26.704146 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-db0157-6689cd59bc-nd7kc" podUID="8e365260-257f-45ef-9306-6e899ae672e9" containerName="payload-pre-processing" containerID="cri-o://5dd647bb2931451cc37d3a893c1c95c6e3ba16c0629cacda915aee8f8ac785b7" gracePeriod=25 Nov 01 21:16:27.304705 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:27.304661 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" podUID="70b4dacf-b3ac-46ec-99e8-a44f05587e7c" containerName="main" containerID="cri-o://391b7775010b6326b8a4242c3a87f859eb8589e190e9470c6efc490e6d0491fc" gracePeriod=45 Nov 01 21:16:27.348383 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:27.348346 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-db0157-6689cd59bc-nd7kc" podUID="8e365260-257f-45ef-9306-6e899ae672e9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 21:16:27.348556 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:27.348343 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-db0157-654558c4b7-vsbx9" podUID="0b416214-5015-4e36-89e4-e9dab20238fd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 21:16:27.561125 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:27.561067 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" Nov 01 21:16:27.668093 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:27.668061 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70b4dacf-b3ac-46ec-99e8-a44f05587e7c-home\" (UniqueName: \"kubernetes.io/empty-dir/70b4dacf-b3ac-46ec-99e8-a44f05587e7c-home\") pod \"70b4dacf-b3ac-46ec-99e8-a44f05587e7c\" (UID: \"70b4dacf-b3ac-46ec-99e8-a44f05587e7c\") " Nov 01 21:16:27.668532 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:27.668108 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70b4dacf-b3ac-46ec-99e8-a44f05587e7c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70b4dacf-b3ac-46ec-99e8-a44f05587e7c-tmp-dir\") pod \"70b4dacf-b3ac-46ec-99e8-a44f05587e7c\" (UID: \"70b4dacf-b3ac-46ec-99e8-a44f05587e7c\") " Nov 01 21:16:27.668532 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:27.668171 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70b4dacf-b3ac-46ec-99e8-a44f05587e7c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/70b4dacf-b3ac-46ec-99e8-a44f05587e7c-dshm\") pod \"70b4dacf-b3ac-46ec-99e8-a44f05587e7c\" (UID: \"70b4dacf-b3ac-46ec-99e8-a44f05587e7c\") " Nov 01 21:16:27.668532 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:27.668224 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70b4dacf-b3ac-46ec-99e8-a44f05587e7c-kube-api-access-sldq4\" (UniqueName: \"kubernetes.io/projected/70b4dacf-b3ac-46ec-99e8-a44f05587e7c-kube-api-access-sldq4\") pod \"70b4dacf-b3ac-46ec-99e8-a44f05587e7c\" (UID: \"70b4dacf-b3ac-46ec-99e8-a44f05587e7c\") " Nov 01 21:16:27.668532 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:27.668255 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70b4dacf-b3ac-46ec-99e8-a44f05587e7c-tls-certs\" (UniqueName: \"kubernetes.io/secret/70b4dacf-b3ac-46ec-99e8-a44f05587e7c-tls-certs\") pod \"70b4dacf-b3ac-46ec-99e8-a44f05587e7c\" (UID: \"70b4dacf-b3ac-46ec-99e8-a44f05587e7c\") " Nov 01 21:16:27.668532 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:27.668298 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70b4dacf-b3ac-46ec-99e8-a44f05587e7c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70b4dacf-b3ac-46ec-99e8-a44f05587e7c-model-cache\") pod \"70b4dacf-b3ac-46ec-99e8-a44f05587e7c\" (UID: \"70b4dacf-b3ac-46ec-99e8-a44f05587e7c\") " Nov 01 21:16:27.668532 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:27.668355 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70b4dacf-b3ac-46ec-99e8-a44f05587e7c-home" pod "70b4dacf-b3ac-46ec-99e8-a44f05587e7c" (UID: "70b4dacf-b3ac-46ec-99e8-a44f05587e7c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:16:27.668532 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:27.668430 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70b4dacf-b3ac-46ec-99e8-a44f05587e7c-tmp-dir" pod "70b4dacf-b3ac-46ec-99e8-a44f05587e7c" (UID: "70b4dacf-b3ac-46ec-99e8-a44f05587e7c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:16:27.668859 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:27.668678 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70b4dacf-b3ac-46ec-99e8-a44f05587e7c-model-cache" pod "70b4dacf-b3ac-46ec-99e8-a44f05587e7c" (UID: "70b4dacf-b3ac-46ec-99e8-a44f05587e7c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:16:27.668859 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:27.668696 4484 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70b4dacf-b3ac-46ec-99e8-a44f05587e7c-home\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:27.668859 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:27.668736 4484 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70b4dacf-b3ac-46ec-99e8-a44f05587e7c-tmp-dir\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:27.670562 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:27.670532 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70b4dacf-b3ac-46ec-99e8-a44f05587e7c-tls-certs" pod "70b4dacf-b3ac-46ec-99e8-a44f05587e7c" (UID: "70b4dacf-b3ac-46ec-99e8-a44f05587e7c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:16:27.670824 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:27.670807 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70b4dacf-b3ac-46ec-99e8-a44f05587e7c-kube-api-access-sldq4" pod "70b4dacf-b3ac-46ec-99e8-a44f05587e7c" (UID: "70b4dacf-b3ac-46ec-99e8-a44f05587e7c"). InnerVolumeSpecName "kube-api-access-sldq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:16:27.670894 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:27.670807 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70b4dacf-b3ac-46ec-99e8-a44f05587e7c-dshm" pod "70b4dacf-b3ac-46ec-99e8-a44f05587e7c" (UID: "70b4dacf-b3ac-46ec-99e8-a44f05587e7c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:16:27.742299 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:27.742047 4484 generic.go:356] "Generic (PLEG): container finished" podID="70b4dacf-b3ac-46ec-99e8-a44f05587e7c" containerID="391b7775010b6326b8a4242c3a87f859eb8589e190e9470c6efc490e6d0491fc" exitCode=0 Nov 01 21:16:27.742299 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:27.742109 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" Nov 01 21:16:27.742299 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:27.742120 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" event={"ID":"70b4dacf-b3ac-46ec-99e8-a44f05587e7c","Type":"ContainerDied","Data":"391b7775010b6326b8a4242c3a87f859eb8589e190e9470c6efc490e6d0491fc"} Nov 01 21:16:27.742299 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:27.742179 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4" event={"ID":"70b4dacf-b3ac-46ec-99e8-a44f05587e7c","Type":"ContainerDied","Data":"8e31c566aed249e91d44cac7035f4d0a3658ca586e3e71c8a94aba1e67d2f03e"} Nov 01 21:16:27.742299 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:27.742203 4484 scope.go:122] "RemoveContainer" containerID="391b7775010b6326b8a4242c3a87f859eb8589e190e9470c6efc490e6d0491fc" Nov 01 21:16:27.753783 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:27.753755 4484 scope.go:122] "RemoveContainer" containerID="391b7775010b6326b8a4242c3a87f859eb8589e190e9470c6efc490e6d0491fc" Nov 01 21:16:27.754168 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:16:27.754137 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"391b7775010b6326b8a4242c3a87f859eb8589e190e9470c6efc490e6d0491fc\": container with ID starting with 391b7775010b6326b8a4242c3a87f859eb8589e190e9470c6efc490e6d0491fc not found: ID does not exist" containerID="391b7775010b6326b8a4242c3a87f859eb8589e190e9470c6efc490e6d0491fc" Nov 01 21:16:27.754313 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:27.754176 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"391b7775010b6326b8a4242c3a87f859eb8589e190e9470c6efc490e6d0491fc"} err="failed to get container status \"391b7775010b6326b8a4242c3a87f859eb8589e190e9470c6efc490e6d0491fc\": rpc error: code = NotFound desc = could not find container \"391b7775010b6326b8a4242c3a87f859eb8589e190e9470c6efc490e6d0491fc\": container with ID starting with 391b7775010b6326b8a4242c3a87f859eb8589e190e9470c6efc490e6d0491fc not found: ID does not exist" Nov 01 21:16:27.769666 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:27.769642 4484 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70b4dacf-b3ac-46ec-99e8-a44f05587e7c-dshm\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:27.769666 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:27.769667 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sldq4\" (UniqueName: \"kubernetes.io/projected/70b4dacf-b3ac-46ec-99e8-a44f05587e7c-kube-api-access-sldq4\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:27.769827 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:27.769678 4484 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70b4dacf-b3ac-46ec-99e8-a44f05587e7c-tls-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:27.769827 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:27.769688 4484 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70b4dacf-b3ac-46ec-99e8-a44f05587e7c-model-cache\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:27.908164 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:27.908076 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4"] Nov 01 21:16:27.915418 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:27.915386 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-db0157/distinct-w1-db0157-kserve-646d4c46b9-mqss4"] Nov 01 21:16:27.942897 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:27.942862 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:29.370305 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:29.370275 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70b4dacf-b3ac-46ec-99e8-a44f05587e7c" path="/var/lib/kubelet/pods/70b4dacf-b3ac-46ec-99e8-a44f05587e7c/volumes" Nov 01 21:16:29.505143 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:29.505106 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:29.723848 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:29.723767 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:29.884653 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:29.884620 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:29.978780 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:29.978695 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-208f2b-jdcwz"] Nov 01 21:16:30.029419 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:30.029384 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-208f2b-jdcwz"] Nov 01 21:16:30.383418 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:30.383385 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:31.370602 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:31.370525 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fa4342c-78ad-493c-8709-f7e5b2ceaa3b" path="/var/lib/kubelet/pods/6fa4342c-78ad-493c-8709-f7e5b2ceaa3b/volumes" Nov 01 21:16:31.694693 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:31.694606 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-944263-5cf85467cc-sl4cb"] Nov 01 21:16:31.808172 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:31.808140 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-944263-78999d46dc-m8ml8"] Nov 01 21:16:32.080129 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:32.080095 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:32.230737 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:32.230691 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:34.102775 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:34.102731 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6911ec62-s7w9h"] Nov 01 21:16:34.103212 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:34.103181 4484 cpu_manager.go:414] "containerMap: removing container" podUID="903377a5-cfac-4d8a-adf0-aa729a3cfdde" containerName="payload-pre-processing" Nov 01 21:16:34.103212 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:34.103193 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="903377a5-cfac-4d8a-adf0-aa729a3cfdde" containerName="payload-pre-processing" Nov 01 21:16:34.103212 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:34.103211 4484 cpu_manager.go:414] "containerMap: removing container" podUID="b77b5e66-a42b-4800-a34f-d415b42913d1" containerName="istio-proxy" Nov 01 21:16:34.103323 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:34.103217 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b77b5e66-a42b-4800-a34f-d415b42913d1" containerName="istio-proxy" Nov 01 21:16:34.103323 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:34.103225 4484 cpu_manager.go:414] "containerMap: removing container" podUID="70b4dacf-b3ac-46ec-99e8-a44f05587e7c" containerName="main" Nov 01 21:16:34.103323 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:34.103229 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70b4dacf-b3ac-46ec-99e8-a44f05587e7c" containerName="main" Nov 01 21:16:34.103323 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:34.103238 4484 cpu_manager.go:414] "containerMap: removing container" podUID="4048751a-034d-4ab2-937f-6161505e4166" containerName="istio-proxy" Nov 01 21:16:34.103323 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:34.103243 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4048751a-034d-4ab2-937f-6161505e4166" containerName="istio-proxy" Nov 01 21:16:34.103323 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:34.103254 4484 cpu_manager.go:414] "containerMap: removing container" podUID="b77b5e66-a42b-4800-a34f-d415b42913d1" containerName="istio-proxy" Nov 01 21:16:34.103323 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:34.103259 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b77b5e66-a42b-4800-a34f-d415b42913d1" containerName="istio-proxy" Nov 01 21:16:34.103323 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:34.103269 4484 cpu_manager.go:414] "containerMap: removing container" podUID="8a9e967b-5887-4361-95ad-9a4122698833" containerName="payload-processing" Nov 01 21:16:34.103323 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:34.103273 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a9e967b-5887-4361-95ad-9a4122698833" containerName="payload-processing" Nov 01 21:16:34.103323 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:34.103286 4484 cpu_manager.go:414] "containerMap: removing container" podUID="a67a9b61-9e37-4294-bcc6-fbdd25ab4346" containerName="revoke-keys" Nov 01 21:16:34.103323 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:34.103290 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a67a9b61-9e37-4294-bcc6-fbdd25ab4346" containerName="revoke-keys" Nov 01 21:16:34.103653 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:34.103346 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="a67a9b61-9e37-4294-bcc6-fbdd25ab4346" containerName="revoke-keys" Nov 01 21:16:34.103653 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:34.103356 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="903377a5-cfac-4d8a-adf0-aa729a3cfdde" containerName="payload-pre-processing" Nov 01 21:16:34.103653 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:34.103364 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="b77b5e66-a42b-4800-a34f-d415b42913d1" containerName="istio-proxy" Nov 01 21:16:34.103653 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:34.103369 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="70b4dacf-b3ac-46ec-99e8-a44f05587e7c" containerName="main" Nov 01 21:16:34.103653 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:34.103377 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="b77b5e66-a42b-4800-a34f-d415b42913d1" containerName="istio-proxy" Nov 01 21:16:34.103653 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:34.103382 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a9e967b-5887-4361-95ad-9a4122698833" containerName="payload-processing" Nov 01 21:16:34.107634 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:34.107617 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6911ec62-s7w9h" Nov 01 21:16:34.163680 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:34.163650 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6911ec62-s7w9h"] Nov 01 21:16:34.163843 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:34.163775 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:16:34.163843 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:34.163799 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:16:34.163987 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:34.163867 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:16:34.212176 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:34.212145 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:16:34.241349 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:34.241318 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8fbc4e0-a109-4059-87d6-d785b84eabe0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6911ec62-s7w9h\" (UID: \"f8fbc4e0-a109-4059-87d6-d785b84eabe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6911ec62-s7w9h" Nov 01 21:16:34.241563 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:34.241378 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlr5t\" (UniqueName: \"kubernetes.io/projected/f8fbc4e0-a109-4059-87d6-d785b84eabe0-kube-api-access-hlr5t\") pod \"maas-api-revoke-sub-6911ec62-s7w9h\" (UID: \"f8fbc4e0-a109-4059-87d6-d785b84eabe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6911ec62-s7w9h" Nov 01 21:16:34.241563 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:34.241482 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8fbc4e0-a109-4059-87d6-d785b84eabe0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6911ec62-s7w9h\" (UID: \"f8fbc4e0-a109-4059-87d6-d785b84eabe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6911ec62-s7w9h" Nov 01 21:16:34.343007 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:34.342952 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8fbc4e0-a109-4059-87d6-d785b84eabe0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6911ec62-s7w9h\" (UID: \"f8fbc4e0-a109-4059-87d6-d785b84eabe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6911ec62-s7w9h" Nov 01 21:16:34.343227 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:34.343049 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlr5t\" (UniqueName: \"kubernetes.io/projected/f8fbc4e0-a109-4059-87d6-d785b84eabe0-kube-api-access-hlr5t\") pod \"maas-api-revoke-sub-6911ec62-s7w9h\" (UID: \"f8fbc4e0-a109-4059-87d6-d785b84eabe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6911ec62-s7w9h" Nov 01 21:16:34.343227 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:34.343068 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8fbc4e0-a109-4059-87d6-d785b84eabe0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6911ec62-s7w9h\" (UID: \"f8fbc4e0-a109-4059-87d6-d785b84eabe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6911ec62-s7w9h" Nov 01 21:16:34.343720 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:34.343694 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8fbc4e0-a109-4059-87d6-d785b84eabe0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6911ec62-s7w9h\" (UID: \"f8fbc4e0-a109-4059-87d6-d785b84eabe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6911ec62-s7w9h" Nov 01 21:16:34.345423 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:34.345402 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8fbc4e0-a109-4059-87d6-d785b84eabe0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6911ec62-s7w9h\" (UID: \"f8fbc4e0-a109-4059-87d6-d785b84eabe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6911ec62-s7w9h" Nov 01 21:16:34.401430 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:34.401355 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlr5t\" (UniqueName: \"kubernetes.io/projected/f8fbc4e0-a109-4059-87d6-d785b84eabe0-kube-api-access-hlr5t\") pod \"maas-api-revoke-sub-6911ec62-s7w9h\" (UID: \"f8fbc4e0-a109-4059-87d6-d785b84eabe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6911ec62-s7w9h" Nov 01 21:16:34.421192 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:34.421158 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6911ec62-s7w9h" Nov 01 21:16:34.797784 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:34.797760 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6911ec62-s7w9h"] Nov 01 21:16:34.799539 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:16:34.799503 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8fbc4e0_a109_4059_87d6_d785b84eabe0.slice/crio-30e805311e271520fd56e628408d854f4a1b4da7f748bc4137b035f43d80d066 WatchSource:0}: Error finding container 30e805311e271520fd56e628408d854f4a1b4da7f748bc4137b035f43d80d066: Status 404 returned error can't find the container with id 30e805311e271520fd56e628408d854f4a1b4da7f748bc4137b035f43d80d066 Nov 01 21:16:34.936128 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:34.936098 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:35.794336 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:35.794300 4484 generic.go:356] "Generic (PLEG): container finished" podID="f8fbc4e0-a109-4059-87d6-d785b84eabe0" containerID="5bf567914a60ad7a46214015df4d59bbb5c6a09f776bb228f164f68cbe9072be" exitCode=0 Nov 01 21:16:35.794778 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:35.794343 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6911ec62-s7w9h" event={"ID":"f8fbc4e0-a109-4059-87d6-d785b84eabe0","Type":"ContainerDied","Data":"5bf567914a60ad7a46214015df4d59bbb5c6a09f776bb228f164f68cbe9072be"} Nov 01 21:16:35.794778 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:35.794379 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6911ec62-s7w9h" event={"ID":"f8fbc4e0-a109-4059-87d6-d785b84eabe0","Type":"ContainerStarted","Data":"30e805311e271520fd56e628408d854f4a1b4da7f748bc4137b035f43d80d066"} Nov 01 21:16:36.115603 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:36.115568 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:36.278380 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:36.278344 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:36.403384 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:36.403303 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw"] Nov 01 21:16:36.403554 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:36.403530 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw" podUID="e9c302ea-168a-4fa6-885f-2a3263835c79" containerName="istio-proxy" containerID="cri-o://54b57e02f36025126bec7241c8f6c9529b485e6d3f662fdff7d589afcf3ab366" gracePeriod=30 Nov 01 21:16:36.470352 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:36.470311 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:36.695783 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:36.695685 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-944263-5cf85467cc-sl4cb" podUID="aca41804-736c-425b-9449-f4a331b56769" containerName="payload-processing" containerID="cri-o://aea2e69ddf4ba383c068edce30acde5f210f74ba9000de88bcee44033712f556" gracePeriod=25 Nov 01 21:16:36.809238 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:36.809197 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-944263-78999d46dc-m8ml8" podUID="86181c50-8350-4bb1-9d9d-c3e46768606f" containerName="payload-pre-processing" containerID="cri-o://0db602d024ea9cd86437fb8be96b50f6028443b8e1f3564a11e36e28d9c13e1e" gracePeriod=25 Nov 01 21:16:36.944575 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:36.944548 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6911ec62-s7w9h" Nov 01 21:16:36.978735 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:36.978645 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8fbc4e0-a109-4059-87d6-d785b84eabe0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8fbc4e0-a109-4059-87d6-d785b84eabe0-maas-api-xks-ca\") pod \"f8fbc4e0-a109-4059-87d6-d785b84eabe0\" (UID: \"f8fbc4e0-a109-4059-87d6-d785b84eabe0\") " Nov 01 21:16:36.978735 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:36.978691 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8fbc4e0-a109-4059-87d6-d785b84eabe0-kube-api-access-hlr5t\" (UniqueName: \"kubernetes.io/projected/f8fbc4e0-a109-4059-87d6-d785b84eabe0-kube-api-access-hlr5t\") pod \"f8fbc4e0-a109-4059-87d6-d785b84eabe0\" (UID: \"f8fbc4e0-a109-4059-87d6-d785b84eabe0\") " Nov 01 21:16:36.978980 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:36.978749 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8fbc4e0-a109-4059-87d6-d785b84eabe0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8fbc4e0-a109-4059-87d6-d785b84eabe0-maas-api-openshift-ca\") pod \"f8fbc4e0-a109-4059-87d6-d785b84eabe0\" (UID: \"f8fbc4e0-a109-4059-87d6-d785b84eabe0\") " Nov 01 21:16:36.979204 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:36.979178 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8fbc4e0-a109-4059-87d6-d785b84eabe0-maas-api-openshift-ca" pod "f8fbc4e0-a109-4059-87d6-d785b84eabe0" (UID: "f8fbc4e0-a109-4059-87d6-d785b84eabe0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:16:36.980849 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:36.980825 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8fbc4e0-a109-4059-87d6-d785b84eabe0-maas-api-xks-ca" pod "f8fbc4e0-a109-4059-87d6-d785b84eabe0" (UID: "f8fbc4e0-a109-4059-87d6-d785b84eabe0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:16:36.980940 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:36.980889 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8fbc4e0-a109-4059-87d6-d785b84eabe0-kube-api-access-hlr5t" pod "f8fbc4e0-a109-4059-87d6-d785b84eabe0" (UID: "f8fbc4e0-a109-4059-87d6-d785b84eabe0"). InnerVolumeSpecName "kube-api-access-hlr5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:16:37.080269 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:37.080228 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8fbc4e0-a109-4059-87d6-d785b84eabe0-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:37.080429 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:37.080269 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlr5t\" (UniqueName: \"kubernetes.io/projected/f8fbc4e0-a109-4059-87d6-d785b84eabe0-kube-api-access-hlr5t\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:37.080429 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:37.080290 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8fbc4e0-a109-4059-87d6-d785b84eabe0-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:37.347829 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:37.347785 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-db0157-6689cd59bc-nd7kc" podUID="8e365260-257f-45ef-9306-6e899ae672e9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 21:16:37.348053 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:37.347785 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-db0157-654558c4b7-vsbx9" podUID="0b416214-5015-4e36-89e4-e9dab20238fd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 21:16:37.808346 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:37.808319 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6911ec62-s7w9h" Nov 01 21:16:37.808548 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:37.808348 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6911ec62-s7w9h" event={"ID":"f8fbc4e0-a109-4059-87d6-d785b84eabe0","Type":"ContainerDied","Data":"30e805311e271520fd56e628408d854f4a1b4da7f748bc4137b035f43d80d066"} Nov 01 21:16:37.808548 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:37.808390 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30e805311e271520fd56e628408d854f4a1b4da7f748bc4137b035f43d80d066" Nov 01 21:16:39.110309 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:39.110275 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65195188-2f5s5"] Nov 01 21:16:39.110714 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:39.110703 4484 cpu_manager.go:414] "containerMap: removing container" podUID="f8fbc4e0-a109-4059-87d6-d785b84eabe0" containerName="revoke-keys" Nov 01 21:16:39.110761 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:39.110717 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8fbc4e0-a109-4059-87d6-d785b84eabe0" containerName="revoke-keys" Nov 01 21:16:39.110811 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:39.110802 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8fbc4e0-a109-4059-87d6-d785b84eabe0" containerName="revoke-keys" Nov 01 21:16:39.113662 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:39.113645 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65195188-2f5s5" Nov 01 21:16:39.166279 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:39.166237 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" podUID="a65d9fd8-6193-4d03-9a3c-123f48b5dcdd" containerName="main" containerID="cri-o://7e1a9a32479449f680c8f44d0b622af78c1b572fbb4eca5a085f71e505ddcaa2" gracePeriod=45 Nov 01 21:16:39.198355 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:39.198330 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0f691d8-cc30-4367-901f-cc880d12d118-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65195188-2f5s5\" (UID: \"b0f691d8-cc30-4367-901f-cc880d12d118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65195188-2f5s5" Nov 01 21:16:39.198570 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:39.198547 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxcxt\" (UniqueName: \"kubernetes.io/projected/b0f691d8-cc30-4367-901f-cc880d12d118-kube-api-access-xxcxt\") pod \"maas-api-revoke-sub-65195188-2f5s5\" (UID: \"b0f691d8-cc30-4367-901f-cc880d12d118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65195188-2f5s5" Nov 01 21:16:39.198648 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:39.198592 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0f691d8-cc30-4367-901f-cc880d12d118-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65195188-2f5s5\" (UID: \"b0f691d8-cc30-4367-901f-cc880d12d118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65195188-2f5s5" Nov 01 21:16:39.219100 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:39.219071 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:16:39.219275 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:39.219109 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:16:39.219275 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:39.219168 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:16:39.219275 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:39.219168 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:16:39.221398 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:39.221379 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65195188-2f5s5"] Nov 01 21:16:39.299320 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:39.299291 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxcxt\" (UniqueName: \"kubernetes.io/projected/b0f691d8-cc30-4367-901f-cc880d12d118-kube-api-access-xxcxt\") pod \"maas-api-revoke-sub-65195188-2f5s5\" (UID: \"b0f691d8-cc30-4367-901f-cc880d12d118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65195188-2f5s5" Nov 01 21:16:39.299463 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:39.299328 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0f691d8-cc30-4367-901f-cc880d12d118-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65195188-2f5s5\" (UID: \"b0f691d8-cc30-4367-901f-cc880d12d118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65195188-2f5s5" Nov 01 21:16:39.299463 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:39.299447 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0f691d8-cc30-4367-901f-cc880d12d118-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65195188-2f5s5\" (UID: \"b0f691d8-cc30-4367-901f-cc880d12d118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65195188-2f5s5" Nov 01 21:16:39.300125 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:39.300099 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0f691d8-cc30-4367-901f-cc880d12d118-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65195188-2f5s5\" (UID: \"b0f691d8-cc30-4367-901f-cc880d12d118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65195188-2f5s5" Nov 01 21:16:39.301632 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:39.301608 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0f691d8-cc30-4367-901f-cc880d12d118-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65195188-2f5s5\" (UID: \"b0f691d8-cc30-4367-901f-cc880d12d118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65195188-2f5s5" Nov 01 21:16:39.365201 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:39.365116 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxcxt\" (UniqueName: \"kubernetes.io/projected/b0f691d8-cc30-4367-901f-cc880d12d118-kube-api-access-xxcxt\") pod \"maas-api-revoke-sub-65195188-2f5s5\" (UID: \"b0f691d8-cc30-4367-901f-cc880d12d118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65195188-2f5s5" Nov 01 21:16:39.424290 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:39.424265 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" Nov 01 21:16:39.426082 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:39.426065 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65195188-2f5s5" Nov 01 21:16:39.501400 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:39.501371 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a65d9fd8-6193-4d03-9a3c-123f48b5dcdd-tls-certs\" (UniqueName: \"kubernetes.io/secret/a65d9fd8-6193-4d03-9a3c-123f48b5dcdd-tls-certs\") pod \"a65d9fd8-6193-4d03-9a3c-123f48b5dcdd\" (UID: \"a65d9fd8-6193-4d03-9a3c-123f48b5dcdd\") " Nov 01 21:16:39.501587 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:39.501414 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a65d9fd8-6193-4d03-9a3c-123f48b5dcdd-home\" (UniqueName: \"kubernetes.io/empty-dir/a65d9fd8-6193-4d03-9a3c-123f48b5dcdd-home\") pod \"a65d9fd8-6193-4d03-9a3c-123f48b5dcdd\" (UID: \"a65d9fd8-6193-4d03-9a3c-123f48b5dcdd\") " Nov 01 21:16:39.501587 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:39.501480 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a65d9fd8-6193-4d03-9a3c-123f48b5dcdd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a65d9fd8-6193-4d03-9a3c-123f48b5dcdd-dshm\") pod \"a65d9fd8-6193-4d03-9a3c-123f48b5dcdd\" (UID: \"a65d9fd8-6193-4d03-9a3c-123f48b5dcdd\") " Nov 01 21:16:39.501587 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:39.501500 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a65d9fd8-6193-4d03-9a3c-123f48b5dcdd-kube-api-access-h78c8\" (UniqueName: \"kubernetes.io/projected/a65d9fd8-6193-4d03-9a3c-123f48b5dcdd-kube-api-access-h78c8\") pod \"a65d9fd8-6193-4d03-9a3c-123f48b5dcdd\" (UID: \"a65d9fd8-6193-4d03-9a3c-123f48b5dcdd\") " Nov 01 21:16:39.501587 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:39.501548 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a65d9fd8-6193-4d03-9a3c-123f48b5dcdd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a65d9fd8-6193-4d03-9a3c-123f48b5dcdd-tmp-dir\") pod \"a65d9fd8-6193-4d03-9a3c-123f48b5dcdd\" (UID: \"a65d9fd8-6193-4d03-9a3c-123f48b5dcdd\") " Nov 01 21:16:39.501829 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:39.501611 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a65d9fd8-6193-4d03-9a3c-123f48b5dcdd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a65d9fd8-6193-4d03-9a3c-123f48b5dcdd-model-cache\") pod \"a65d9fd8-6193-4d03-9a3c-123f48b5dcdd\" (UID: \"a65d9fd8-6193-4d03-9a3c-123f48b5dcdd\") " Nov 01 21:16:39.501829 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:39.501714 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a65d9fd8-6193-4d03-9a3c-123f48b5dcdd-home" pod "a65d9fd8-6193-4d03-9a3c-123f48b5dcdd" (UID: "a65d9fd8-6193-4d03-9a3c-123f48b5dcdd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:16:39.501975 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:39.501921 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a65d9fd8-6193-4d03-9a3c-123f48b5dcdd-model-cache" pod "a65d9fd8-6193-4d03-9a3c-123f48b5dcdd" (UID: "a65d9fd8-6193-4d03-9a3c-123f48b5dcdd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:16:39.501975 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:39.501961 4484 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a65d9fd8-6193-4d03-9a3c-123f48b5dcdd-home\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:39.502109 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:39.502029 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a65d9fd8-6193-4d03-9a3c-123f48b5dcdd-tmp-dir" pod "a65d9fd8-6193-4d03-9a3c-123f48b5dcdd" (UID: "a65d9fd8-6193-4d03-9a3c-123f48b5dcdd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:16:39.504180 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:39.504152 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a65d9fd8-6193-4d03-9a3c-123f48b5dcdd-tls-certs" pod "a65d9fd8-6193-4d03-9a3c-123f48b5dcdd" (UID: "a65d9fd8-6193-4d03-9a3c-123f48b5dcdd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:16:39.504316 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:39.504177 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a65d9fd8-6193-4d03-9a3c-123f48b5dcdd-dshm" pod "a65d9fd8-6193-4d03-9a3c-123f48b5dcdd" (UID: "a65d9fd8-6193-4d03-9a3c-123f48b5dcdd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:16:39.504316 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:39.504275 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a65d9fd8-6193-4d03-9a3c-123f48b5dcdd-kube-api-access-h78c8" pod "a65d9fd8-6193-4d03-9a3c-123f48b5dcdd" (UID: "a65d9fd8-6193-4d03-9a3c-123f48b5dcdd"). InnerVolumeSpecName "kube-api-access-h78c8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:16:39.603475 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:39.603440 4484 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a65d9fd8-6193-4d03-9a3c-123f48b5dcdd-dshm\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:39.603475 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:39.603472 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h78c8\" (UniqueName: \"kubernetes.io/projected/a65d9fd8-6193-4d03-9a3c-123f48b5dcdd-kube-api-access-h78c8\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:39.603475 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:39.603482 4484 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a65d9fd8-6193-4d03-9a3c-123f48b5dcdd-tmp-dir\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:39.603724 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:39.603491 4484 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a65d9fd8-6193-4d03-9a3c-123f48b5dcdd-model-cache\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:39.603724 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:39.603502 4484 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a65d9fd8-6193-4d03-9a3c-123f48b5dcdd-tls-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:39.801566 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:16:39.801527 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0f691d8_cc30_4367_901f_cc880d12d118.slice/crio-9eb7d9631651b43a9bcbab609ab76e23b38b47edab7d8963b631c69872004841 WatchSource:0}: Error finding container 9eb7d9631651b43a9bcbab609ab76e23b38b47edab7d8963b631c69872004841: Status 404 returned error can't find the container with id 9eb7d9631651b43a9bcbab609ab76e23b38b47edab7d8963b631c69872004841 Nov 01 21:16:39.802363 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:39.802334 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65195188-2f5s5"] Nov 01 21:16:39.821592 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:39.821560 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65195188-2f5s5" event={"ID":"b0f691d8-cc30-4367-901f-cc880d12d118","Type":"ContainerStarted","Data":"9eb7d9631651b43a9bcbab609ab76e23b38b47edab7d8963b631c69872004841"} Nov 01 21:16:39.823087 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:39.823053 4484 generic.go:356] "Generic (PLEG): container finished" podID="a65d9fd8-6193-4d03-9a3c-123f48b5dcdd" containerID="7e1a9a32479449f680c8f44d0b622af78c1b572fbb4eca5a085f71e505ddcaa2" exitCode=0 Nov 01 21:16:39.823222 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:39.823095 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" Nov 01 21:16:39.823222 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:39.823127 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" event={"ID":"a65d9fd8-6193-4d03-9a3c-123f48b5dcdd","Type":"ContainerDied","Data":"7e1a9a32479449f680c8f44d0b622af78c1b572fbb4eca5a085f71e505ddcaa2"} Nov 01 21:16:39.823222 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:39.823166 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq" event={"ID":"a65d9fd8-6193-4d03-9a3c-123f48b5dcdd","Type":"ContainerDied","Data":"34f31e577b423012adf294027ca20e9e8ae9fe3dfb034e66ec4011132da3917e"} Nov 01 21:16:39.823222 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:39.823196 4484 scope.go:122] "RemoveContainer" containerID="7e1a9a32479449f680c8f44d0b622af78c1b572fbb4eca5a085f71e505ddcaa2" Nov 01 21:16:39.839342 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:39.839325 4484 scope.go:122] "RemoveContainer" containerID="7e1a9a32479449f680c8f44d0b622af78c1b572fbb4eca5a085f71e505ddcaa2" Nov 01 21:16:39.839653 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:16:39.839634 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e1a9a32479449f680c8f44d0b622af78c1b572fbb4eca5a085f71e505ddcaa2\": container with ID starting with 7e1a9a32479449f680c8f44d0b622af78c1b572fbb4eca5a085f71e505ddcaa2 not found: ID does not exist" containerID="7e1a9a32479449f680c8f44d0b622af78c1b572fbb4eca5a085f71e505ddcaa2" Nov 01 21:16:39.839721 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:39.839659 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e1a9a32479449f680c8f44d0b622af78c1b572fbb4eca5a085f71e505ddcaa2"} err="failed to get container status \"7e1a9a32479449f680c8f44d0b622af78c1b572fbb4eca5a085f71e505ddcaa2\": rpc error: code = NotFound desc = could not find container \"7e1a9a32479449f680c8f44d0b622af78c1b572fbb4eca5a085f71e505ddcaa2\": container with ID starting with 7e1a9a32479449f680c8f44d0b622af78c1b572fbb4eca5a085f71e505ddcaa2 not found: ID does not exist" Nov 01 21:16:39.990912 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:39.990879 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq"] Nov 01 21:16:39.992844 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:39.992820 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-944263/distinct-2-w3-944263-kserve-6cd7fbc98f-g2dxq"] Nov 01 21:16:40.831597 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:40.831561 4484 generic.go:356] "Generic (PLEG): container finished" podID="b0f691d8-cc30-4367-901f-cc880d12d118" containerID="e370babedebd99baec75b329b2ab52a05c74d448551236c89cc38928b63d2840" exitCode=0 Nov 01 21:16:40.832053 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:40.831646 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65195188-2f5s5" event={"ID":"b0f691d8-cc30-4367-901f-cc880d12d118","Type":"ContainerDied","Data":"e370babedebd99baec75b329b2ab52a05c74d448551236c89cc38928b63d2840"} Nov 01 21:16:41.371345 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:41.371313 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a65d9fd8-6193-4d03-9a3c-123f48b5dcdd" path="/var/lib/kubelet/pods/a65d9fd8-6193-4d03-9a3c-123f48b5dcdd/volumes" Nov 01 21:16:41.664398 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:41.664374 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw" Nov 01 21:16:41.721359 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:41.721326 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9c302ea-168a-4fa6-885f-2a3263835c79-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9c302ea-168a-4fa6-885f-2a3263835c79-istio-data\") pod \"e9c302ea-168a-4fa6-885f-2a3263835c79\" (UID: \"e9c302ea-168a-4fa6-885f-2a3263835c79\") " Nov 01 21:16:41.721543 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:41.721389 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9c302ea-168a-4fa6-885f-2a3263835c79-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9c302ea-168a-4fa6-885f-2a3263835c79-credential-socket\") pod \"e9c302ea-168a-4fa6-885f-2a3263835c79\" (UID: \"e9c302ea-168a-4fa6-885f-2a3263835c79\") " Nov 01 21:16:41.721543 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:41.721428 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9c302ea-168a-4fa6-885f-2a3263835c79-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9c302ea-168a-4fa6-885f-2a3263835c79-istio-envoy\") pod \"e9c302ea-168a-4fa6-885f-2a3263835c79\" (UID: \"e9c302ea-168a-4fa6-885f-2a3263835c79\") " Nov 01 21:16:41.721543 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:41.721445 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9c302ea-168a-4fa6-885f-2a3263835c79-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9c302ea-168a-4fa6-885f-2a3263835c79-workload-socket\") pod \"e9c302ea-168a-4fa6-885f-2a3263835c79\" (UID: \"e9c302ea-168a-4fa6-885f-2a3263835c79\") " Nov 01 21:16:41.721543 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:41.721488 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9c302ea-168a-4fa6-885f-2a3263835c79-kube-api-access-6nkkg\" (UniqueName: \"kubernetes.io/projected/e9c302ea-168a-4fa6-885f-2a3263835c79-kube-api-access-6nkkg\") pod \"e9c302ea-168a-4fa6-885f-2a3263835c79\" (UID: \"e9c302ea-168a-4fa6-885f-2a3263835c79\") " Nov 01 21:16:41.721543 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:41.721530 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9c302ea-168a-4fa6-885f-2a3263835c79-istio-token\" (UniqueName: \"kubernetes.io/projected/e9c302ea-168a-4fa6-885f-2a3263835c79-istio-token\") pod \"e9c302ea-168a-4fa6-885f-2a3263835c79\" (UID: \"e9c302ea-168a-4fa6-885f-2a3263835c79\") " Nov 01 21:16:41.721841 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:41.721577 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e9c302ea-168a-4fa6-885f-2a3263835c79-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9c302ea-168a-4fa6-885f-2a3263835c79-istio-podinfo\") pod \"e9c302ea-168a-4fa6-885f-2a3263835c79\" (UID: \"e9c302ea-168a-4fa6-885f-2a3263835c79\") " Nov 01 21:16:41.721841 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:41.721619 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9c302ea-168a-4fa6-885f-2a3263835c79-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9c302ea-168a-4fa6-885f-2a3263835c79-istiod-ca-cert\") pod \"e9c302ea-168a-4fa6-885f-2a3263835c79\" (UID: \"e9c302ea-168a-4fa6-885f-2a3263835c79\") " Nov 01 21:16:41.721841 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:41.721643 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9c302ea-168a-4fa6-885f-2a3263835c79-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9c302ea-168a-4fa6-885f-2a3263835c79-workload-certs\") pod \"e9c302ea-168a-4fa6-885f-2a3263835c79\" (UID: \"e9c302ea-168a-4fa6-885f-2a3263835c79\") " Nov 01 21:16:41.721841 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:41.721734 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9c302ea-168a-4fa6-885f-2a3263835c79-istio-data" pod "e9c302ea-168a-4fa6-885f-2a3263835c79" (UID: "e9c302ea-168a-4fa6-885f-2a3263835c79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:16:41.721841 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:41.721762 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9c302ea-168a-4fa6-885f-2a3263835c79-credential-socket" pod "e9c302ea-168a-4fa6-885f-2a3263835c79" (UID: "e9c302ea-168a-4fa6-885f-2a3263835c79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:16:41.722116 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:41.721960 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9c302ea-168a-4fa6-885f-2a3263835c79-workload-certs" pod "e9c302ea-168a-4fa6-885f-2a3263835c79" (UID: "e9c302ea-168a-4fa6-885f-2a3263835c79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:16:41.722116 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:41.721977 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9c302ea-168a-4fa6-885f-2a3263835c79-workload-socket" pod "e9c302ea-168a-4fa6-885f-2a3263835c79" (UID: "e9c302ea-168a-4fa6-885f-2a3263835c79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:16:41.722116 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:41.722049 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9c302ea-168a-4fa6-885f-2a3263835c79-istiod-ca-cert" pod "e9c302ea-168a-4fa6-885f-2a3263835c79" (UID: "e9c302ea-168a-4fa6-885f-2a3263835c79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:16:41.722116 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:41.722055 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9c302ea-168a-4fa6-885f-2a3263835c79-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:41.722116 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:41.722096 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9c302ea-168a-4fa6-885f-2a3263835c79-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:41.723954 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:41.723931 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9c302ea-168a-4fa6-885f-2a3263835c79-istio-token" pod "e9c302ea-168a-4fa6-885f-2a3263835c79" (UID: "e9c302ea-168a-4fa6-885f-2a3263835c79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:16:41.723954 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:41.723943 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9c302ea-168a-4fa6-885f-2a3263835c79-kube-api-access-6nkkg" pod "e9c302ea-168a-4fa6-885f-2a3263835c79" (UID: "e9c302ea-168a-4fa6-885f-2a3263835c79"). InnerVolumeSpecName "kube-api-access-6nkkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:16:41.724283 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:41.724264 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9c302ea-168a-4fa6-885f-2a3263835c79-istio-envoy" pod "e9c302ea-168a-4fa6-885f-2a3263835c79" (UID: "e9c302ea-168a-4fa6-885f-2a3263835c79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:16:41.724283 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:41.724270 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9c302ea-168a-4fa6-885f-2a3263835c79-istio-podinfo" pod "e9c302ea-168a-4fa6-885f-2a3263835c79" (UID: "e9c302ea-168a-4fa6-885f-2a3263835c79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:16:41.823530 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:41.823493 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9c302ea-168a-4fa6-885f-2a3263835c79-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:41.823530 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:41.823525 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9c302ea-168a-4fa6-885f-2a3263835c79-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:41.823530 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:41.823537 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nkkg\" (UniqueName: \"kubernetes.io/projected/e9c302ea-168a-4fa6-885f-2a3263835c79-kube-api-access-6nkkg\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:41.823799 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:41.823548 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9c302ea-168a-4fa6-885f-2a3263835c79-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:41.823799 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:41.823559 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9c302ea-168a-4fa6-885f-2a3263835c79-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:41.823799 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:41.823571 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9c302ea-168a-4fa6-885f-2a3263835c79-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:41.823799 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:41.823583 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9c302ea-168a-4fa6-885f-2a3263835c79-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:41.841748 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:41.841699 4484 generic.go:356] "Generic (PLEG): container finished" podID="e9c302ea-168a-4fa6-885f-2a3263835c79" containerID="54b57e02f36025126bec7241c8f6c9529b485e6d3f662fdff7d589afcf3ab366" exitCode=0 Nov 01 21:16:41.841748 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:41.841744 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw" Nov 01 21:16:41.842299 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:41.841769 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw" event={"ID":"e9c302ea-168a-4fa6-885f-2a3263835c79","Type":"ContainerDied","Data":"54b57e02f36025126bec7241c8f6c9529b485e6d3f662fdff7d589afcf3ab366"} Nov 01 21:16:41.842299 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:41.841865 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw" event={"ID":"e9c302ea-168a-4fa6-885f-2a3263835c79","Type":"ContainerDied","Data":"b4ed7865476bbf19cbad5b1be56863705b025aa812967d7a304be1c1e92714cb"} Nov 01 21:16:41.842299 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:41.841902 4484 scope.go:122] "RemoveContainer" containerID="54b57e02f36025126bec7241c8f6c9529b485e6d3f662fdff7d589afcf3ab366" Nov 01 21:16:41.853413 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:41.853386 4484 scope.go:122] "RemoveContainer" containerID="297e2449b94efbf6299679b7996c8245b7492d28246c60a767b6a54e6716a790" Nov 01 21:16:41.868699 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:41.868672 4484 scope.go:122] "RemoveContainer" containerID="54b57e02f36025126bec7241c8f6c9529b485e6d3f662fdff7d589afcf3ab366" Nov 01 21:16:41.869079 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:16:41.869056 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54b57e02f36025126bec7241c8f6c9529b485e6d3f662fdff7d589afcf3ab366\": container with ID starting with 54b57e02f36025126bec7241c8f6c9529b485e6d3f662fdff7d589afcf3ab366 not found: ID does not exist" containerID="54b57e02f36025126bec7241c8f6c9529b485e6d3f662fdff7d589afcf3ab366" Nov 01 21:16:41.869189 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:41.869090 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54b57e02f36025126bec7241c8f6c9529b485e6d3f662fdff7d589afcf3ab366"} err="failed to get container status \"54b57e02f36025126bec7241c8f6c9529b485e6d3f662fdff7d589afcf3ab366\": rpc error: code = NotFound desc = could not find container \"54b57e02f36025126bec7241c8f6c9529b485e6d3f662fdff7d589afcf3ab366\": container with ID starting with 54b57e02f36025126bec7241c8f6c9529b485e6d3f662fdff7d589afcf3ab366 not found: ID does not exist" Nov 01 21:16:41.869189 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:41.869116 4484 scope.go:122] "RemoveContainer" containerID="297e2449b94efbf6299679b7996c8245b7492d28246c60a767b6a54e6716a790" Nov 01 21:16:41.869445 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:16:41.869422 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"297e2449b94efbf6299679b7996c8245b7492d28246c60a767b6a54e6716a790\": container with ID starting with 297e2449b94efbf6299679b7996c8245b7492d28246c60a767b6a54e6716a790 not found: ID does not exist" containerID="297e2449b94efbf6299679b7996c8245b7492d28246c60a767b6a54e6716a790" Nov 01 21:16:41.869513 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:41.869454 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"297e2449b94efbf6299679b7996c8245b7492d28246c60a767b6a54e6716a790"} err="failed to get container status \"297e2449b94efbf6299679b7996c8245b7492d28246c60a767b6a54e6716a790\": rpc error: code = NotFound desc = could not find container \"297e2449b94efbf6299679b7996c8245b7492d28246c60a767b6a54e6716a790\": container with ID starting with 297e2449b94efbf6299679b7996c8245b7492d28246c60a767b6a54e6716a790 not found: ID does not exist" Nov 01 21:16:41.971782 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:41.971755 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65195188-2f5s5" Nov 01 21:16:41.996833 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:41.996804 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw"] Nov 01 21:16:42.002656 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:42.002629 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-208f2b-openshift-default-5cc586d6f-blcnw"] Nov 01 21:16:42.025899 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:42.025870 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0f691d8-cc30-4367-901f-cc880d12d118-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0f691d8-cc30-4367-901f-cc880d12d118-maas-api-openshift-ca\") pod \"b0f691d8-cc30-4367-901f-cc880d12d118\" (UID: \"b0f691d8-cc30-4367-901f-cc880d12d118\") " Nov 01 21:16:42.025899 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:42.025900 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0f691d8-cc30-4367-901f-cc880d12d118-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0f691d8-cc30-4367-901f-cc880d12d118-maas-api-xks-ca\") pod \"b0f691d8-cc30-4367-901f-cc880d12d118\" (UID: \"b0f691d8-cc30-4367-901f-cc880d12d118\") " Nov 01 21:16:42.026126 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:42.025974 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0f691d8-cc30-4367-901f-cc880d12d118-kube-api-access-xxcxt\" (UniqueName: \"kubernetes.io/projected/b0f691d8-cc30-4367-901f-cc880d12d118-kube-api-access-xxcxt\") pod \"b0f691d8-cc30-4367-901f-cc880d12d118\" (UID: \"b0f691d8-cc30-4367-901f-cc880d12d118\") " Nov 01 21:16:42.026280 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:42.026255 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0f691d8-cc30-4367-901f-cc880d12d118-maas-api-openshift-ca" pod "b0f691d8-cc30-4367-901f-cc880d12d118" (UID: "b0f691d8-cc30-4367-901f-cc880d12d118"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:16:42.028008 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:42.027974 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0f691d8-cc30-4367-901f-cc880d12d118-maas-api-xks-ca" pod "b0f691d8-cc30-4367-901f-cc880d12d118" (UID: "b0f691d8-cc30-4367-901f-cc880d12d118"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:16:42.028108 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:42.028090 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0f691d8-cc30-4367-901f-cc880d12d118-kube-api-access-xxcxt" pod "b0f691d8-cc30-4367-901f-cc880d12d118" (UID: "b0f691d8-cc30-4367-901f-cc880d12d118"). InnerVolumeSpecName "kube-api-access-xxcxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:16:42.126949 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:42.126919 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0f691d8-cc30-4367-901f-cc880d12d118-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:42.126949 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:42.126948 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0f691d8-cc30-4367-901f-cc880d12d118-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:42.127164 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:42.126957 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxcxt\" (UniqueName: \"kubernetes.io/projected/b0f691d8-cc30-4367-901f-cc880d12d118-kube-api-access-xxcxt\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:42.221345 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:42.221255 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-944263-5cf85467cc-sl4cb" podUID="aca41804-736c-425b-9449-f4a331b56769" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 21:16:42.221489 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:42.221265 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-944263-78999d46dc-m8ml8" podUID="86181c50-8350-4bb1-9d9d-c3e46768606f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 21:16:42.850088 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:42.850055 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65195188-2f5s5" Nov 01 21:16:42.850088 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:42.850067 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65195188-2f5s5" event={"ID":"b0f691d8-cc30-4367-901f-cc880d12d118","Type":"ContainerDied","Data":"9eb7d9631651b43a9bcbab609ab76e23b38b47edab7d8963b631c69872004841"} Nov 01 21:16:42.850598 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:42.850126 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9eb7d9631651b43a9bcbab609ab76e23b38b47edab7d8963b631c69872004841" Nov 01 21:16:43.371100 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:43.371068 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9c302ea-168a-4fa6-885f-2a3263835c79" path="/var/lib/kubelet/pods/e9c302ea-168a-4fa6-885f-2a3263835c79/volumes" Nov 01 21:16:45.766570 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:16:45.766536 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:16:46.846936 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:46.846904 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-db0157-654558c4b7-vsbx9" Nov 01 21:16:46.868098 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:46.868017 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b416214-5015-4e36-89e4-e9dab20238fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b416214-5015-4e36-89e4-e9dab20238fd-plugins-config-volume\") pod \"0b416214-5015-4e36-89e4-e9dab20238fd\" (UID: \"0b416214-5015-4e36-89e4-e9dab20238fd\") " Nov 01 21:16:46.868263 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:46.868129 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b416214-5015-4e36-89e4-e9dab20238fd-kube-api-access-27k6n\" (UniqueName: \"kubernetes.io/projected/0b416214-5015-4e36-89e4-e9dab20238fd-kube-api-access-27k6n\") pod \"0b416214-5015-4e36-89e4-e9dab20238fd\" (UID: \"0b416214-5015-4e36-89e4-e9dab20238fd\") " Nov 01 21:16:46.868430 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:46.868391 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b416214-5015-4e36-89e4-e9dab20238fd-plugins-config-volume" pod "0b416214-5015-4e36-89e4-e9dab20238fd" (UID: "0b416214-5015-4e36-89e4-e9dab20238fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:16:46.870247 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:46.870215 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b416214-5015-4e36-89e4-e9dab20238fd-kube-api-access-27k6n" pod "0b416214-5015-4e36-89e4-e9dab20238fd" (UID: "0b416214-5015-4e36-89e4-e9dab20238fd"). InnerVolumeSpecName "kube-api-access-27k6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:16:46.877104 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:46.877029 4484 generic.go:356] "Generic (PLEG): container finished" podID="8e365260-257f-45ef-9306-6e899ae672e9" containerID="5dd647bb2931451cc37d3a893c1c95c6e3ba16c0629cacda915aee8f8ac785b7" exitCode=0 Nov 01 21:16:46.877104 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:46.877073 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-db0157-6689cd59bc-nd7kc" event={"ID":"8e365260-257f-45ef-9306-6e899ae672e9","Type":"ContainerDied","Data":"5dd647bb2931451cc37d3a893c1c95c6e3ba16c0629cacda915aee8f8ac785b7"} Nov 01 21:16:46.879232 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:46.879206 4484 generic.go:356] "Generic (PLEG): container finished" podID="0b416214-5015-4e36-89e4-e9dab20238fd" containerID="8e5e0b63de1ddb3e6a9856b1de46170f993606739250ef98c40b1cb9dd7416ed" exitCode=0 Nov 01 21:16:46.879363 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:46.879260 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-db0157-654558c4b7-vsbx9" Nov 01 21:16:46.879495 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:46.879463 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-db0157-654558c4b7-vsbx9" event={"ID":"0b416214-5015-4e36-89e4-e9dab20238fd","Type":"ContainerDied","Data":"8e5e0b63de1ddb3e6a9856b1de46170f993606739250ef98c40b1cb9dd7416ed"} Nov 01 21:16:46.879611 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:46.879529 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-db0157-654558c4b7-vsbx9" event={"ID":"0b416214-5015-4e36-89e4-e9dab20238fd","Type":"ContainerDied","Data":"75551f2d1f425a0a1d5fad7cce6aad651922ee69cd1075d21ae25bda3f7e7018"} Nov 01 21:16:46.879611 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:46.879565 4484 scope.go:122] "RemoveContainer" containerID="8e5e0b63de1ddb3e6a9856b1de46170f993606739250ef98c40b1cb9dd7416ed" Nov 01 21:16:46.890815 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:46.890793 4484 scope.go:122] "RemoveContainer" containerID="8e5e0b63de1ddb3e6a9856b1de46170f993606739250ef98c40b1cb9dd7416ed" Nov 01 21:16:46.891163 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:16:46.891142 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e5e0b63de1ddb3e6a9856b1de46170f993606739250ef98c40b1cb9dd7416ed\": container with ID starting with 8e5e0b63de1ddb3e6a9856b1de46170f993606739250ef98c40b1cb9dd7416ed not found: ID does not exist" containerID="8e5e0b63de1ddb3e6a9856b1de46170f993606739250ef98c40b1cb9dd7416ed" Nov 01 21:16:46.891263 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:46.891175 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e5e0b63de1ddb3e6a9856b1de46170f993606739250ef98c40b1cb9dd7416ed"} err="failed to get container status \"8e5e0b63de1ddb3e6a9856b1de46170f993606739250ef98c40b1cb9dd7416ed\": rpc error: code = NotFound desc = could not find container \"8e5e0b63de1ddb3e6a9856b1de46170f993606739250ef98c40b1cb9dd7416ed\": container with ID starting with 8e5e0b63de1ddb3e6a9856b1de46170f993606739250ef98c40b1cb9dd7416ed not found: ID does not exist" Nov 01 21:16:46.950518 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:46.950494 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-db0157-6689cd59bc-nd7kc" Nov 01 21:16:46.969470 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:46.969443 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27k6n\" (UniqueName: \"kubernetes.io/projected/0b416214-5015-4e36-89e4-e9dab20238fd-kube-api-access-27k6n\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:46.969470 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:46.969468 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b416214-5015-4e36-89e4-e9dab20238fd-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:47.070581 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:47.070449 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e365260-257f-45ef-9306-6e899ae672e9-kube-api-access-9csqq\" (UniqueName: \"kubernetes.io/projected/8e365260-257f-45ef-9306-6e899ae672e9-kube-api-access-9csqq\") pod \"8e365260-257f-45ef-9306-6e899ae672e9\" (UID: \"8e365260-257f-45ef-9306-6e899ae672e9\") " Nov 01 21:16:47.070838 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:47.070658 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e365260-257f-45ef-9306-6e899ae672e9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e365260-257f-45ef-9306-6e899ae672e9-plugins-config-volume\") pod \"8e365260-257f-45ef-9306-6e899ae672e9\" (UID: \"8e365260-257f-45ef-9306-6e899ae672e9\") " Nov 01 21:16:47.071493 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:47.071467 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e365260-257f-45ef-9306-6e899ae672e9-plugins-config-volume" pod "8e365260-257f-45ef-9306-6e899ae672e9" (UID: "8e365260-257f-45ef-9306-6e899ae672e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:16:47.072831 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:47.072805 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e365260-257f-45ef-9306-6e899ae672e9-kube-api-access-9csqq" pod "8e365260-257f-45ef-9306-6e899ae672e9" (UID: "8e365260-257f-45ef-9306-6e899ae672e9"). InnerVolumeSpecName "kube-api-access-9csqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:16:47.073800 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:47.073777 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-db0157-654558c4b7-vsbx9"] Nov 01 21:16:47.079609 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:47.079582 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-db0157-654558c4b7-vsbx9"] Nov 01 21:16:47.172326 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:47.172233 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e365260-257f-45ef-9306-6e899ae672e9-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:47.172326 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:47.172264 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9csqq\" (UniqueName: \"kubernetes.io/projected/8e365260-257f-45ef-9306-6e899ae672e9-kube-api-access-9csqq\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:47.370266 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:47.370230 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b416214-5015-4e36-89e4-e9dab20238fd" path="/var/lib/kubelet/pods/0b416214-5015-4e36-89e4-e9dab20238fd/volumes" Nov 01 21:16:47.886897 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:47.886815 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-db0157-6689cd59bc-nd7kc" Nov 01 21:16:47.886897 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:47.886823 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-db0157-6689cd59bc-nd7kc" event={"ID":"8e365260-257f-45ef-9306-6e899ae672e9","Type":"ContainerDied","Data":"22b7b0ba36c3448ccb4beb2d69663149cf03925bc5df0131a3da526db0d09cc7"} Nov 01 21:16:47.886897 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:47.886886 4484 scope.go:122] "RemoveContainer" containerID="5dd647bb2931451cc37d3a893c1c95c6e3ba16c0629cacda915aee8f8ac785b7" Nov 01 21:16:48.051840 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:48.051807 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-db0157-6689cd59bc-nd7kc"] Nov 01 21:16:48.108584 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:48.103877 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-db0157-6689cd59bc-nd7kc"] Nov 01 21:16:48.587168 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:48.587124 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:48.782166 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:48.782135 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:49.148556 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:49.148519 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:49.369819 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:49.369783 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e365260-257f-45ef-9306-6e899ae672e9" path="/var/lib/kubelet/pods/8e365260-257f-45ef-9306-6e899ae672e9/volumes" Nov 01 21:16:50.095165 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:50.095133 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:50.315189 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:50.315156 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:50.681712 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:50.681656 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:50.968664 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:50.968575 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:51.164282 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:51.164247 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-db0157-7d8ls"] Nov 01 21:16:51.214963 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:51.214930 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-db0157-7d8ls"] Nov 01 21:16:51.371126 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:51.371093 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48639132-0d2c-4597-b056-68e1766437dd" path="/var/lib/kubelet/pods/48639132-0d2c-4597-b056-68e1766437dd/volumes" Nov 01 21:16:51.587603 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:51.587575 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:51.890925 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:51.890890 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:52.221274 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:52.221189 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-944263-78999d46dc-m8ml8" podUID="86181c50-8350-4bb1-9d9d-c3e46768606f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 21:16:52.221274 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:52.221212 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-944263-5cf85467cc-sl4cb" podUID="aca41804-736c-425b-9449-f4a331b56769" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 21:16:52.224686 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:52.224663 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:52.476956 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:52.476874 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-944263-cbd5k"] Nov 01 21:16:52.527715 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:52.527650 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-944263-cbd5k"] Nov 01 21:16:52.833175 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:52.833136 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:53.370809 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:53.370775 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a67a9b61-9e37-4294-bcc6-fbdd25ab4346" path="/var/lib/kubelet/pods/a67a9b61-9e37-4294-bcc6-fbdd25ab4346/volumes" Nov 01 21:16:55.587478 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:55.587442 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:55.804441 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:55.804396 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:56.949468 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:56.949439 4484 generic.go:356] "Generic (PLEG): container finished" podID="aca41804-736c-425b-9449-f4a331b56769" containerID="aea2e69ddf4ba383c068edce30acde5f210f74ba9000de88bcee44033712f556" exitCode=0 Nov 01 21:16:56.949936 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:56.949519 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-944263-5cf85467cc-sl4cb" event={"ID":"aca41804-736c-425b-9449-f4a331b56769","Type":"ContainerDied","Data":"aea2e69ddf4ba383c068edce30acde5f210f74ba9000de88bcee44033712f556"} Nov 01 21:16:56.949936 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:56.949581 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-944263-5cf85467cc-sl4cb" event={"ID":"aca41804-736c-425b-9449-f4a331b56769","Type":"ContainerDied","Data":"51623c5e1229419d55d29b80893bec22d377e2bd93c4806d1bb324961039dd54"} Nov 01 21:16:56.949936 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:56.949607 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51623c5e1229419d55d29b80893bec22d377e2bd93c4806d1bb324961039dd54" Nov 01 21:16:56.951344 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:56.951292 4484 generic.go:356] "Generic (PLEG): container finished" podID="86181c50-8350-4bb1-9d9d-c3e46768606f" containerID="0db602d024ea9cd86437fb8be96b50f6028443b8e1f3564a11e36e28d9c13e1e" exitCode=0 Nov 01 21:16:56.951455 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:56.951359 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-944263-78999d46dc-m8ml8" event={"ID":"86181c50-8350-4bb1-9d9d-c3e46768606f","Type":"ContainerDied","Data":"0db602d024ea9cd86437fb8be96b50f6028443b8e1f3564a11e36e28d9c13e1e"} Nov 01 21:16:56.960056 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:56.960030 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-944263-5cf85467cc-sl4cb" Nov 01 21:16:57.061705 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:57.061679 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-944263-78999d46dc-m8ml8" Nov 01 21:16:57.065689 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:57.065669 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aca41804-736c-425b-9449-f4a331b56769-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aca41804-736c-425b-9449-f4a331b56769-plugins-config-volume\") pod \"aca41804-736c-425b-9449-f4a331b56769\" (UID: \"aca41804-736c-425b-9449-f4a331b56769\") " Nov 01 21:16:57.065802 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:57.065706 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aca41804-736c-425b-9449-f4a331b56769-kube-api-access-crdnn\" (UniqueName: \"kubernetes.io/projected/aca41804-736c-425b-9449-f4a331b56769-kube-api-access-crdnn\") pod \"aca41804-736c-425b-9449-f4a331b56769\" (UID: \"aca41804-736c-425b-9449-f4a331b56769\") " Nov 01 21:16:57.066021 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:57.065986 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aca41804-736c-425b-9449-f4a331b56769-plugins-config-volume" pod "aca41804-736c-425b-9449-f4a331b56769" (UID: "aca41804-736c-425b-9449-f4a331b56769"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:16:57.067753 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:57.067735 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aca41804-736c-425b-9449-f4a331b56769-kube-api-access-crdnn" pod "aca41804-736c-425b-9449-f4a331b56769" (UID: "aca41804-736c-425b-9449-f4a331b56769"). InnerVolumeSpecName "kube-api-access-crdnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:16:57.166701 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:57.166668 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86181c50-8350-4bb1-9d9d-c3e46768606f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86181c50-8350-4bb1-9d9d-c3e46768606f-plugins-config-volume\") pod \"86181c50-8350-4bb1-9d9d-c3e46768606f\" (UID: \"86181c50-8350-4bb1-9d9d-c3e46768606f\") " Nov 01 21:16:57.166919 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:57.166731 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86181c50-8350-4bb1-9d9d-c3e46768606f-kube-api-access-4q8p2\" (UniqueName: \"kubernetes.io/projected/86181c50-8350-4bb1-9d9d-c3e46768606f-kube-api-access-4q8p2\") pod \"86181c50-8350-4bb1-9d9d-c3e46768606f\" (UID: \"86181c50-8350-4bb1-9d9d-c3e46768606f\") " Nov 01 21:16:57.167041 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:57.167025 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aca41804-736c-425b-9449-f4a331b56769-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:57.167129 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:57.167047 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crdnn\" (UniqueName: \"kubernetes.io/projected/aca41804-736c-425b-9449-f4a331b56769-kube-api-access-crdnn\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:57.167129 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:57.167071 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86181c50-8350-4bb1-9d9d-c3e46768606f-plugins-config-volume" pod "86181c50-8350-4bb1-9d9d-c3e46768606f" (UID: "86181c50-8350-4bb1-9d9d-c3e46768606f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:16:57.168918 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:57.168889 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86181c50-8350-4bb1-9d9d-c3e46768606f-kube-api-access-4q8p2" pod "86181c50-8350-4bb1-9d9d-c3e46768606f" (UID: "86181c50-8350-4bb1-9d9d-c3e46768606f"). InnerVolumeSpecName "kube-api-access-4q8p2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:16:57.267637 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:57.267599 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86181c50-8350-4bb1-9d9d-c3e46768606f-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:57.267637 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:57.267631 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4q8p2\" (UniqueName: \"kubernetes.io/projected/86181c50-8350-4bb1-9d9d-c3e46768606f-kube-api-access-4q8p2\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:57.530303 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:57.530219 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:57.687189 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:57.687150 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:57.888317 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:57.888118 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:57.957945 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:57.957914 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-944263-78999d46dc-m8ml8" Nov 01 21:16:57.958441 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:57.957954 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-944263-78999d46dc-m8ml8" event={"ID":"86181c50-8350-4bb1-9d9d-c3e46768606f","Type":"ContainerDied","Data":"0b31fe530dfeed00ce796185db8a8e82b414081063fda3c96be81c409fc95ebd"} Nov 01 21:16:57.958441 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:57.958020 4484 scope.go:122] "RemoveContainer" containerID="0db602d024ea9cd86437fb8be96b50f6028443b8e1f3564a11e36e28d9c13e1e" Nov 01 21:16:57.958441 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:57.958236 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-944263-5cf85467cc-sl4cb" Nov 01 21:16:58.123461 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:58.123429 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-944263-5cf85467cc-sl4cb"] Nov 01 21:16:58.174628 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:58.174535 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-944263-5cf85467cc-sl4cb"] Nov 01 21:16:58.236491 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:58.236452 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-944263-78999d46dc-m8ml8"] Nov 01 21:16:58.240223 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:58.240198 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-944263-78999d46dc-m8ml8"] Nov 01 21:16:59.105662 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:59.105622 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:59.272730 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:59.272696 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:59.371633 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:59.371558 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86181c50-8350-4bb1-9d9d-c3e46768606f" path="/var/lib/kubelet/pods/86181c50-8350-4bb1-9d9d-c3e46768606f/volumes" Nov 01 21:16:59.372058 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:59.372038 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aca41804-736c-425b-9449-f4a331b56769" path="/var/lib/kubelet/pods/aca41804-736c-425b-9449-f4a331b56769/volumes" Nov 01 21:16:59.430780 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:16:59.430735 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:17:00.422760 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:00.422702 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:17:00.589904 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:00.589858 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:17:01.319383 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:01.319347 4484 scope.go:122] "RemoveContainer" containerID="ee0ed5bd5c45470bbf9bcf9498fd27943dc4ad13b355e1618edf0295167a51e6" Nov 01 21:17:01.328420 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:01.328401 4484 scope.go:122] "RemoveContainer" containerID="e733ba9ef7e45667878e32f24afcc597b6b95710e2e7c89005ab21e89f843531" Nov 01 21:17:01.337088 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:01.337057 4484 scope.go:122] "RemoveContainer" containerID="aea2e69ddf4ba383c068edce30acde5f210f74ba9000de88bcee44033712f556" Nov 01 21:17:01.631462 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:01.631371 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:17:15.766138 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:17:15.766098 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:17:15.934570 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:15.934537 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:17:27.283709 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:27.283675 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6bjrx"] Nov 01 21:17:27.284216 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:27.284107 4484 cpu_manager.go:414] "containerMap: removing container" podUID="86181c50-8350-4bb1-9d9d-c3e46768606f" containerName="payload-pre-processing" Nov 01 21:17:27.284216 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:27.284119 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86181c50-8350-4bb1-9d9d-c3e46768606f" containerName="payload-pre-processing" Nov 01 21:17:27.284216 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:27.284131 4484 cpu_manager.go:414] "containerMap: removing container" podUID="e9c302ea-168a-4fa6-885f-2a3263835c79" containerName="istio-proxy" Nov 01 21:17:27.284216 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:27.284136 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9c302ea-168a-4fa6-885f-2a3263835c79" containerName="istio-proxy" Nov 01 21:17:27.284216 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:27.284146 4484 cpu_manager.go:414] "containerMap: removing container" podUID="8e365260-257f-45ef-9306-6e899ae672e9" containerName="payload-pre-processing" Nov 01 21:17:27.284216 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:27.284150 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e365260-257f-45ef-9306-6e899ae672e9" containerName="payload-pre-processing" Nov 01 21:17:27.284216 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:27.284156 4484 cpu_manager.go:414] "containerMap: removing container" podUID="b0f691d8-cc30-4367-901f-cc880d12d118" containerName="revoke-keys" Nov 01 21:17:27.284216 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:27.284161 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0f691d8-cc30-4367-901f-cc880d12d118" containerName="revoke-keys" Nov 01 21:17:27.284216 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:27.284175 4484 cpu_manager.go:414] "containerMap: removing container" podUID="e9c302ea-168a-4fa6-885f-2a3263835c79" containerName="istio-proxy" Nov 01 21:17:27.284216 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:27.284179 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9c302ea-168a-4fa6-885f-2a3263835c79" containerName="istio-proxy" Nov 01 21:17:27.284216 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:27.284190 4484 cpu_manager.go:414] "containerMap: removing container" podUID="0b416214-5015-4e36-89e4-e9dab20238fd" containerName="payload-processing" Nov 01 21:17:27.284216 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:27.284195 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b416214-5015-4e36-89e4-e9dab20238fd" containerName="payload-processing" Nov 01 21:17:27.284216 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:27.284202 4484 cpu_manager.go:414] "containerMap: removing container" podUID="a65d9fd8-6193-4d03-9a3c-123f48b5dcdd" containerName="main" Nov 01 21:17:27.284216 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:27.284206 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a65d9fd8-6193-4d03-9a3c-123f48b5dcdd" containerName="main" Nov 01 21:17:27.284216 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:27.284224 4484 cpu_manager.go:414] "containerMap: removing container" podUID="aca41804-736c-425b-9449-f4a331b56769" containerName="payload-processing" Nov 01 21:17:27.284716 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:27.284229 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aca41804-736c-425b-9449-f4a331b56769" containerName="payload-processing" Nov 01 21:17:27.284716 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:27.284289 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="86181c50-8350-4bb1-9d9d-c3e46768606f" containerName="payload-pre-processing" Nov 01 21:17:27.284716 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:27.284299 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9c302ea-168a-4fa6-885f-2a3263835c79" containerName="istio-proxy" Nov 01 21:17:27.284716 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:27.284306 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e365260-257f-45ef-9306-6e899ae672e9" containerName="payload-pre-processing" Nov 01 21:17:27.284716 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:27.284312 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0f691d8-cc30-4367-901f-cc880d12d118" containerName="revoke-keys" Nov 01 21:17:27.284716 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:27.284318 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9c302ea-168a-4fa6-885f-2a3263835c79" containerName="istio-proxy" Nov 01 21:17:27.284716 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:27.284323 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b416214-5015-4e36-89e4-e9dab20238fd" containerName="payload-processing" Nov 01 21:17:27.284716 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:27.284329 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="a65d9fd8-6193-4d03-9a3c-123f48b5dcdd" containerName="main" Nov 01 21:17:27.284716 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:27.284337 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="aca41804-736c-425b-9449-f4a331b56769" containerName="payload-processing" Nov 01 21:17:27.287181 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:27.287163 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6bjrx" Nov 01 21:17:27.345291 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:27.345251 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6bjrx"] Nov 01 21:17:27.357121 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:27.357069 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvmjp\" (UniqueName: \"kubernetes.io/projected/4fff41a8-bef5-464b-89b6-2bc7247d528c-kube-api-access-kvmjp\") pod \"maas-api-revoke-sub-85237e45-6bjrx\" (UID: \"4fff41a8-bef5-464b-89b6-2bc7247d528c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6bjrx" Nov 01 21:17:27.357315 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:27.357256 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fff41a8-bef5-464b-89b6-2bc7247d528c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-6bjrx\" (UID: \"4fff41a8-bef5-464b-89b6-2bc7247d528c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6bjrx" Nov 01 21:17:27.357382 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:27.357361 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fff41a8-bef5-464b-89b6-2bc7247d528c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-6bjrx\" (UID: \"4fff41a8-bef5-464b-89b6-2bc7247d528c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6bjrx" Nov 01 21:17:27.394051 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:27.393989 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:17:27.394051 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:27.394034 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:17:27.394286 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:27.394069 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:17:27.394286 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:27.394069 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:17:27.458537 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:27.458502 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fff41a8-bef5-464b-89b6-2bc7247d528c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-6bjrx\" (UID: \"4fff41a8-bef5-464b-89b6-2bc7247d528c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6bjrx" Nov 01 21:17:27.458761 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:27.458600 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fff41a8-bef5-464b-89b6-2bc7247d528c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-6bjrx\" (UID: \"4fff41a8-bef5-464b-89b6-2bc7247d528c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6bjrx" Nov 01 21:17:27.458761 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:27.458726 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvmjp\" (UniqueName: \"kubernetes.io/projected/4fff41a8-bef5-464b-89b6-2bc7247d528c-kube-api-access-kvmjp\") pod \"maas-api-revoke-sub-85237e45-6bjrx\" (UID: \"4fff41a8-bef5-464b-89b6-2bc7247d528c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6bjrx" Nov 01 21:17:27.459333 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:27.459310 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fff41a8-bef5-464b-89b6-2bc7247d528c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-6bjrx\" (UID: \"4fff41a8-bef5-464b-89b6-2bc7247d528c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6bjrx" Nov 01 21:17:27.461043 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:27.461025 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fff41a8-bef5-464b-89b6-2bc7247d528c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-6bjrx\" (UID: \"4fff41a8-bef5-464b-89b6-2bc7247d528c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6bjrx" Nov 01 21:17:27.518510 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:27.518471 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvmjp\" (UniqueName: \"kubernetes.io/projected/4fff41a8-bef5-464b-89b6-2bc7247d528c-kube-api-access-kvmjp\") pod \"maas-api-revoke-sub-85237e45-6bjrx\" (UID: \"4fff41a8-bef5-464b-89b6-2bc7247d528c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6bjrx" Nov 01 21:17:27.599823 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:27.599730 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6bjrx" Nov 01 21:17:27.972616 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:27.972587 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6bjrx"] Nov 01 21:17:27.975066 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:17:27.975035 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fff41a8_bef5_464b_89b6_2bc7247d528c.slice/crio-b4ed1733dc6f74f03d4aa0613a99fcf3b9eff5852c7bf6387f9325c0bf74efd5 WatchSource:0}: Error finding container b4ed1733dc6f74f03d4aa0613a99fcf3b9eff5852c7bf6387f9325c0bf74efd5: Status 404 returned error can't find the container with id b4ed1733dc6f74f03d4aa0613a99fcf3b9eff5852c7bf6387f9325c0bf74efd5 Nov 01 21:17:28.128921 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:28.128878 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6bjrx" event={"ID":"4fff41a8-bef5-464b-89b6-2bc7247d528c","Type":"ContainerStarted","Data":"b4ed1733dc6f74f03d4aa0613a99fcf3b9eff5852c7bf6387f9325c0bf74efd5"} Nov 01 21:17:29.135881 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:29.135846 4484 generic.go:356] "Generic (PLEG): container finished" podID="4fff41a8-bef5-464b-89b6-2bc7247d528c" containerID="7da3462e5c91133c44b0c89bbadb01097da4d0acdd5edce5bf3a4b959daaaa15" exitCode=0 Nov 01 21:17:29.136324 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:29.135936 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6bjrx" event={"ID":"4fff41a8-bef5-464b-89b6-2bc7247d528c","Type":"ContainerDied","Data":"7da3462e5c91133c44b0c89bbadb01097da4d0acdd5edce5bf3a4b959daaaa15"} Nov 01 21:17:30.276381 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:30.276356 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6bjrx" Nov 01 21:17:30.385398 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:30.385366 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fff41a8-bef5-464b-89b6-2bc7247d528c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fff41a8-bef5-464b-89b6-2bc7247d528c-maas-api-openshift-ca\") pod \"4fff41a8-bef5-464b-89b6-2bc7247d528c\" (UID: \"4fff41a8-bef5-464b-89b6-2bc7247d528c\") " Nov 01 21:17:30.385576 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:30.385407 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fff41a8-bef5-464b-89b6-2bc7247d528c-kube-api-access-kvmjp\" (UniqueName: \"kubernetes.io/projected/4fff41a8-bef5-464b-89b6-2bc7247d528c-kube-api-access-kvmjp\") pod \"4fff41a8-bef5-464b-89b6-2bc7247d528c\" (UID: \"4fff41a8-bef5-464b-89b6-2bc7247d528c\") " Nov 01 21:17:30.385576 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:30.385462 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fff41a8-bef5-464b-89b6-2bc7247d528c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fff41a8-bef5-464b-89b6-2bc7247d528c-maas-api-xks-ca\") pod \"4fff41a8-bef5-464b-89b6-2bc7247d528c\" (UID: \"4fff41a8-bef5-464b-89b6-2bc7247d528c\") " Nov 01 21:17:30.385735 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:30.385710 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fff41a8-bef5-464b-89b6-2bc7247d528c-maas-api-openshift-ca" pod "4fff41a8-bef5-464b-89b6-2bc7247d528c" (UID: "4fff41a8-bef5-464b-89b6-2bc7247d528c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:17:30.387578 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:30.387544 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fff41a8-bef5-464b-89b6-2bc7247d528c-maas-api-xks-ca" pod "4fff41a8-bef5-464b-89b6-2bc7247d528c" (UID: "4fff41a8-bef5-464b-89b6-2bc7247d528c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:17:30.387697 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:30.387614 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fff41a8-bef5-464b-89b6-2bc7247d528c-kube-api-access-kvmjp" pod "4fff41a8-bef5-464b-89b6-2bc7247d528c" (UID: "4fff41a8-bef5-464b-89b6-2bc7247d528c"). InnerVolumeSpecName "kube-api-access-kvmjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:17:30.486520 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:30.486478 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fff41a8-bef5-464b-89b6-2bc7247d528c-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:17:30.486520 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:30.486507 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvmjp\" (UniqueName: \"kubernetes.io/projected/4fff41a8-bef5-464b-89b6-2bc7247d528c-kube-api-access-kvmjp\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:17:30.486520 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:30.486521 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fff41a8-bef5-464b-89b6-2bc7247d528c-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:17:31.147824 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:31.147795 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6bjrx" Nov 01 21:17:31.147824 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:31.147816 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6bjrx" event={"ID":"4fff41a8-bef5-464b-89b6-2bc7247d528c","Type":"ContainerDied","Data":"b4ed1733dc6f74f03d4aa0613a99fcf3b9eff5852c7bf6387f9325c0bf74efd5"} Nov 01 21:17:31.148070 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:31.147887 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4ed1733dc6f74f03d4aa0613a99fcf3b9eff5852c7bf6387f9325c0bf74efd5" Nov 01 21:17:33.106928 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.106886 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-hdvw9"] Nov 01 21:17:33.107358 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.107294 4484 cpu_manager.go:414] "containerMap: removing container" podUID="4fff41a8-bef5-464b-89b6-2bc7247d528c" containerName="revoke-keys" Nov 01 21:17:33.107358 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.107305 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fff41a8-bef5-464b-89b6-2bc7247d528c" containerName="revoke-keys" Nov 01 21:17:33.107433 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.107377 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fff41a8-bef5-464b-89b6-2bc7247d528c" containerName="revoke-keys" Nov 01 21:17:33.110637 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.110615 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-hdvw9" Nov 01 21:17:33.158082 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.158043 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6zwld"] Nov 01 21:17:33.162184 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.162160 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2n5h6"] Nov 01 21:17:33.162327 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.162277 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6zwld" Nov 01 21:17:33.165632 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.165613 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2n5h6" Nov 01 21:17:33.206765 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.206734 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-cqvhp\"" Nov 01 21:17:33.211024 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.210973 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-hdvw9"] Nov 01 21:17:33.211486 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.211464 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec0b1a26-5b6c-405c-a042-7141708b6b22-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-hdvw9\" (UID: \"ec0b1a26-5b6c-405c-a042-7141708b6b22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-hdvw9" Nov 01 21:17:33.211547 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.211506 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw4zr\" (UniqueName: \"kubernetes.io/projected/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-kube-api-access-rw4zr\") pod \"maas-default-gateway-openshift-default-58c6585447-6zwld\" (UID: \"8e6a4f7c-75e1-4506-afb5-4ab00107a7f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6zwld" Nov 01 21:17:33.211547 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.211541 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5mg4\" (UniqueName: \"kubernetes.io/projected/35afbdf8-b193-42b5-9c8f-85471b4288b7-kube-api-access-l5mg4\") pod \"maas-default-gateway-openshift-default-58c6585447-2n5h6\" (UID: \"35afbdf8-b193-42b5-9c8f-85471b4288b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2n5h6" Nov 01 21:17:33.211670 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.211629 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec0b1a26-5b6c-405c-a042-7141708b6b22-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-hdvw9\" (UID: \"ec0b1a26-5b6c-405c-a042-7141708b6b22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-hdvw9" Nov 01 21:17:33.211713 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.211687 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35afbdf8-b193-42b5-9c8f-85471b4288b7-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-2n5h6\" (UID: \"35afbdf8-b193-42b5-9c8f-85471b4288b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2n5h6" Nov 01 21:17:33.211818 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.211800 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35afbdf8-b193-42b5-9c8f-85471b4288b7-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-2n5h6\" (UID: \"35afbdf8-b193-42b5-9c8f-85471b4288b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2n5h6" Nov 01 21:17:33.211859 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.211838 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec0b1a26-5b6c-405c-a042-7141708b6b22-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-hdvw9\" (UID: \"ec0b1a26-5b6c-405c-a042-7141708b6b22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-hdvw9" Nov 01 21:17:33.211898 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.211866 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec0b1a26-5b6c-405c-a042-7141708b6b22-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-hdvw9\" (UID: \"ec0b1a26-5b6c-405c-a042-7141708b6b22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-hdvw9" Nov 01 21:17:33.211951 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.211900 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-6zwld\" (UID: \"8e6a4f7c-75e1-4506-afb5-4ab00107a7f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6zwld" Nov 01 21:17:33.211951 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.211926 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec0b1a26-5b6c-405c-a042-7141708b6b22-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-hdvw9\" (UID: \"ec0b1a26-5b6c-405c-a042-7141708b6b22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-hdvw9" Nov 01 21:17:33.212049 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.211978 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec0b1a26-5b6c-405c-a042-7141708b6b22-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-hdvw9\" (UID: \"ec0b1a26-5b6c-405c-a042-7141708b6b22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-hdvw9" Nov 01 21:17:33.212049 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.212041 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-6zwld\" (UID: \"8e6a4f7c-75e1-4506-afb5-4ab00107a7f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6zwld" Nov 01 21:17:33.212122 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.212095 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35afbdf8-b193-42b5-9c8f-85471b4288b7-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-2n5h6\" (UID: \"35afbdf8-b193-42b5-9c8f-85471b4288b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2n5h6" Nov 01 21:17:33.212155 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.212129 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35afbdf8-b193-42b5-9c8f-85471b4288b7-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-2n5h6\" (UID: \"35afbdf8-b193-42b5-9c8f-85471b4288b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2n5h6" Nov 01 21:17:33.212190 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.212154 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35afbdf8-b193-42b5-9c8f-85471b4288b7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-2n5h6\" (UID: \"35afbdf8-b193-42b5-9c8f-85471b4288b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2n5h6" Nov 01 21:17:33.212190 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.212174 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-6zwld\" (UID: \"8e6a4f7c-75e1-4506-afb5-4ab00107a7f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6zwld" Nov 01 21:17:33.212281 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.212200 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec0b1a26-5b6c-405c-a042-7141708b6b22-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-hdvw9\" (UID: \"ec0b1a26-5b6c-405c-a042-7141708b6b22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-hdvw9" Nov 01 21:17:33.212369 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.212351 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-6zwld\" (UID: \"8e6a4f7c-75e1-4506-afb5-4ab00107a7f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6zwld" Nov 01 21:17:33.212435 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.212382 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35afbdf8-b193-42b5-9c8f-85471b4288b7-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-2n5h6\" (UID: \"35afbdf8-b193-42b5-9c8f-85471b4288b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2n5h6" Nov 01 21:17:33.212435 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.212415 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-6zwld\" (UID: \"8e6a4f7c-75e1-4506-afb5-4ab00107a7f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6zwld" Nov 01 21:17:33.212530 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.212462 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-6zwld\" (UID: \"8e6a4f7c-75e1-4506-afb5-4ab00107a7f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6zwld" Nov 01 21:17:33.212530 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.212487 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec0b1a26-5b6c-405c-a042-7141708b6b22-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-hdvw9\" (UID: \"ec0b1a26-5b6c-405c-a042-7141708b6b22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-hdvw9" Nov 01 21:17:33.212530 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.212525 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-6zwld\" (UID: \"8e6a4f7c-75e1-4506-afb5-4ab00107a7f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6zwld" Nov 01 21:17:33.212641 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.212564 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-6zwld\" (UID: \"8e6a4f7c-75e1-4506-afb5-4ab00107a7f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6zwld" Nov 01 21:17:33.212641 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.212613 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbsm6\" (UniqueName: \"kubernetes.io/projected/ec0b1a26-5b6c-405c-a042-7141708b6b22-kube-api-access-hbsm6\") pod \"maas-default-gateway-openshift-default-58c6585447-hdvw9\" (UID: \"ec0b1a26-5b6c-405c-a042-7141708b6b22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-hdvw9" Nov 01 21:17:33.212641 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.212631 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35afbdf8-b193-42b5-9c8f-85471b4288b7-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-2n5h6\" (UID: \"35afbdf8-b193-42b5-9c8f-85471b4288b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2n5h6" Nov 01 21:17:33.212767 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.212654 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35afbdf8-b193-42b5-9c8f-85471b4288b7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-2n5h6\" (UID: \"35afbdf8-b193-42b5-9c8f-85471b4288b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2n5h6" Nov 01 21:17:33.261149 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.261111 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6zwld"] Nov 01 21:17:33.313221 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.313182 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-6zwld\" (UID: \"8e6a4f7c-75e1-4506-afb5-4ab00107a7f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6zwld" Nov 01 21:17:33.313221 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.313217 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35afbdf8-b193-42b5-9c8f-85471b4288b7-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-2n5h6\" (UID: \"35afbdf8-b193-42b5-9c8f-85471b4288b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2n5h6" Nov 01 21:17:33.313540 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.313237 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-6zwld\" (UID: \"8e6a4f7c-75e1-4506-afb5-4ab00107a7f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6zwld" Nov 01 21:17:33.313540 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.313257 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-6zwld\" (UID: \"8e6a4f7c-75e1-4506-afb5-4ab00107a7f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6zwld" Nov 01 21:17:33.313540 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.313391 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec0b1a26-5b6c-405c-a042-7141708b6b22-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-hdvw9\" (UID: \"ec0b1a26-5b6c-405c-a042-7141708b6b22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-hdvw9" Nov 01 21:17:33.313540 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.313437 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-6zwld\" (UID: \"8e6a4f7c-75e1-4506-afb5-4ab00107a7f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6zwld" Nov 01 21:17:33.313540 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.313481 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-6zwld\" (UID: \"8e6a4f7c-75e1-4506-afb5-4ab00107a7f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6zwld" Nov 01 21:17:33.313540 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.313521 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbsm6\" (UniqueName: \"kubernetes.io/projected/ec0b1a26-5b6c-405c-a042-7141708b6b22-kube-api-access-hbsm6\") pod \"maas-default-gateway-openshift-default-58c6585447-hdvw9\" (UID: \"ec0b1a26-5b6c-405c-a042-7141708b6b22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-hdvw9" Nov 01 21:17:33.313896 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.313553 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35afbdf8-b193-42b5-9c8f-85471b4288b7-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-2n5h6\" (UID: \"35afbdf8-b193-42b5-9c8f-85471b4288b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2n5h6" Nov 01 21:17:33.313896 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.313596 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35afbdf8-b193-42b5-9c8f-85471b4288b7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-2n5h6\" (UID: \"35afbdf8-b193-42b5-9c8f-85471b4288b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2n5h6" Nov 01 21:17:33.313896 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.313626 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-6zwld\" (UID: \"8e6a4f7c-75e1-4506-afb5-4ab00107a7f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6zwld" Nov 01 21:17:33.313896 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.313627 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec0b1a26-5b6c-405c-a042-7141708b6b22-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-hdvw9\" (UID: \"ec0b1a26-5b6c-405c-a042-7141708b6b22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-hdvw9" Nov 01 21:17:33.313896 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.313697 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw4zr\" (UniqueName: \"kubernetes.io/projected/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-kube-api-access-rw4zr\") pod \"maas-default-gateway-openshift-default-58c6585447-6zwld\" (UID: \"8e6a4f7c-75e1-4506-afb5-4ab00107a7f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6zwld" Nov 01 21:17:33.313896 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.313740 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5mg4\" (UniqueName: \"kubernetes.io/projected/35afbdf8-b193-42b5-9c8f-85471b4288b7-kube-api-access-l5mg4\") pod \"maas-default-gateway-openshift-default-58c6585447-2n5h6\" (UID: \"35afbdf8-b193-42b5-9c8f-85471b4288b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2n5h6" Nov 01 21:17:33.313896 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.313771 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec0b1a26-5b6c-405c-a042-7141708b6b22-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-hdvw9\" (UID: \"ec0b1a26-5b6c-405c-a042-7141708b6b22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-hdvw9" Nov 01 21:17:33.313896 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.313796 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35afbdf8-b193-42b5-9c8f-85471b4288b7-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-2n5h6\" (UID: \"35afbdf8-b193-42b5-9c8f-85471b4288b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2n5h6" Nov 01 21:17:33.313896 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.313804 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-6zwld\" (UID: \"8e6a4f7c-75e1-4506-afb5-4ab00107a7f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6zwld" Nov 01 21:17:33.313896 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.313833 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35afbdf8-b193-42b5-9c8f-85471b4288b7-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-2n5h6\" (UID: \"35afbdf8-b193-42b5-9c8f-85471b4288b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2n5h6" Nov 01 21:17:33.313896 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.313868 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec0b1a26-5b6c-405c-a042-7141708b6b22-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-hdvw9\" (UID: \"ec0b1a26-5b6c-405c-a042-7141708b6b22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-hdvw9" Nov 01 21:17:33.313896 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.313899 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec0b1a26-5b6c-405c-a042-7141708b6b22-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-hdvw9\" (UID: \"ec0b1a26-5b6c-405c-a042-7141708b6b22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-hdvw9" Nov 01 21:17:33.314615 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.313924 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec0b1a26-5b6c-405c-a042-7141708b6b22-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-hdvw9\" (UID: \"ec0b1a26-5b6c-405c-a042-7141708b6b22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-hdvw9" Nov 01 21:17:33.314615 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.313940 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-6zwld\" (UID: \"8e6a4f7c-75e1-4506-afb5-4ab00107a7f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6zwld" Nov 01 21:17:33.314615 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.313967 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec0b1a26-5b6c-405c-a042-7141708b6b22-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-hdvw9\" (UID: \"ec0b1a26-5b6c-405c-a042-7141708b6b22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-hdvw9" Nov 01 21:17:33.314615 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.314032 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec0b1a26-5b6c-405c-a042-7141708b6b22-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-hdvw9\" (UID: \"ec0b1a26-5b6c-405c-a042-7141708b6b22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-hdvw9" Nov 01 21:17:33.314615 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.314065 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-6zwld\" (UID: \"8e6a4f7c-75e1-4506-afb5-4ab00107a7f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6zwld" Nov 01 21:17:33.314615 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.314093 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35afbdf8-b193-42b5-9c8f-85471b4288b7-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-2n5h6\" (UID: \"35afbdf8-b193-42b5-9c8f-85471b4288b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2n5h6" Nov 01 21:17:33.314615 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.314121 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35afbdf8-b193-42b5-9c8f-85471b4288b7-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-2n5h6\" (UID: \"35afbdf8-b193-42b5-9c8f-85471b4288b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2n5h6" Nov 01 21:17:33.314615 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.314151 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35afbdf8-b193-42b5-9c8f-85471b4288b7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-2n5h6\" (UID: \"35afbdf8-b193-42b5-9c8f-85471b4288b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2n5h6" Nov 01 21:17:33.314615 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.314182 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-6zwld\" (UID: \"8e6a4f7c-75e1-4506-afb5-4ab00107a7f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6zwld" Nov 01 21:17:33.314615 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.314200 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-6zwld\" (UID: \"8e6a4f7c-75e1-4506-afb5-4ab00107a7f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6zwld" Nov 01 21:17:33.314615 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.314215 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec0b1a26-5b6c-405c-a042-7141708b6b22-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-hdvw9\" (UID: \"ec0b1a26-5b6c-405c-a042-7141708b6b22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-hdvw9" Nov 01 21:17:33.314615 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.314379 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35afbdf8-b193-42b5-9c8f-85471b4288b7-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-2n5h6\" (UID: \"35afbdf8-b193-42b5-9c8f-85471b4288b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2n5h6" Nov 01 21:17:33.314615 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.314477 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec0b1a26-5b6c-405c-a042-7141708b6b22-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-hdvw9\" (UID: \"ec0b1a26-5b6c-405c-a042-7141708b6b22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-hdvw9" Nov 01 21:17:33.315400 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.314645 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35afbdf8-b193-42b5-9c8f-85471b4288b7-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-2n5h6\" (UID: \"35afbdf8-b193-42b5-9c8f-85471b4288b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2n5h6" Nov 01 21:17:33.315400 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.314919 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec0b1a26-5b6c-405c-a042-7141708b6b22-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-hdvw9\" (UID: \"ec0b1a26-5b6c-405c-a042-7141708b6b22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-hdvw9" Nov 01 21:17:33.315400 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.314937 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-6zwld\" (UID: \"8e6a4f7c-75e1-4506-afb5-4ab00107a7f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6zwld" Nov 01 21:17:33.315400 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.315040 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35afbdf8-b193-42b5-9c8f-85471b4288b7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-2n5h6\" (UID: \"35afbdf8-b193-42b5-9c8f-85471b4288b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2n5h6" Nov 01 21:17:33.315400 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.315198 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-6zwld\" (UID: \"8e6a4f7c-75e1-4506-afb5-4ab00107a7f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6zwld" Nov 01 21:17:33.315400 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.315214 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec0b1a26-5b6c-405c-a042-7141708b6b22-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-hdvw9\" (UID: \"ec0b1a26-5b6c-405c-a042-7141708b6b22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-hdvw9" Nov 01 21:17:33.315400 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.315239 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec0b1a26-5b6c-405c-a042-7141708b6b22-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-hdvw9\" (UID: \"ec0b1a26-5b6c-405c-a042-7141708b6b22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-hdvw9" Nov 01 21:17:33.315400 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.315304 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35afbdf8-b193-42b5-9c8f-85471b4288b7-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-2n5h6\" (UID: \"35afbdf8-b193-42b5-9c8f-85471b4288b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2n5h6" Nov 01 21:17:33.315797 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.315462 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35afbdf8-b193-42b5-9c8f-85471b4288b7-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-2n5h6\" (UID: \"35afbdf8-b193-42b5-9c8f-85471b4288b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2n5h6" Nov 01 21:17:33.316190 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.316161 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35afbdf8-b193-42b5-9c8f-85471b4288b7-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-2n5h6\" (UID: \"35afbdf8-b193-42b5-9c8f-85471b4288b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2n5h6" Nov 01 21:17:33.316337 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.316286 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec0b1a26-5b6c-405c-a042-7141708b6b22-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-hdvw9\" (UID: \"ec0b1a26-5b6c-405c-a042-7141708b6b22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-hdvw9" Nov 01 21:17:33.316417 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.316396 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-6zwld\" (UID: \"8e6a4f7c-75e1-4506-afb5-4ab00107a7f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6zwld" Nov 01 21:17:33.317652 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.317630 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec0b1a26-5b6c-405c-a042-7141708b6b22-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-hdvw9\" (UID: \"ec0b1a26-5b6c-405c-a042-7141708b6b22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-hdvw9" Nov 01 21:17:33.317726 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.317681 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-6zwld\" (UID: \"8e6a4f7c-75e1-4506-afb5-4ab00107a7f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6zwld" Nov 01 21:17:33.317841 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.317825 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35afbdf8-b193-42b5-9c8f-85471b4288b7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-2n5h6\" (UID: \"35afbdf8-b193-42b5-9c8f-85471b4288b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2n5h6" Nov 01 21:17:33.325197 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.325171 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2n5h6"] Nov 01 21:17:33.372586 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.372511 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5mg4\" (UniqueName: \"kubernetes.io/projected/35afbdf8-b193-42b5-9c8f-85471b4288b7-kube-api-access-l5mg4\") pod \"maas-default-gateway-openshift-default-58c6585447-2n5h6\" (UID: \"35afbdf8-b193-42b5-9c8f-85471b4288b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2n5h6" Nov 01 21:17:33.372765 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.372694 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-6zwld\" (UID: \"8e6a4f7c-75e1-4506-afb5-4ab00107a7f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6zwld" Nov 01 21:17:33.372765 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.372703 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec0b1a26-5b6c-405c-a042-7141708b6b22-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-hdvw9\" (UID: \"ec0b1a26-5b6c-405c-a042-7141708b6b22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-hdvw9" Nov 01 21:17:33.373438 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.373409 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35afbdf8-b193-42b5-9c8f-85471b4288b7-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-2n5h6\" (UID: \"35afbdf8-b193-42b5-9c8f-85471b4288b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2n5h6" Nov 01 21:17:33.373568 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.373461 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw4zr\" (UniqueName: \"kubernetes.io/projected/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-kube-api-access-rw4zr\") pod \"maas-default-gateway-openshift-default-58c6585447-6zwld\" (UID: \"8e6a4f7c-75e1-4506-afb5-4ab00107a7f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6zwld" Nov 01 21:17:33.373636 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.373619 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbsm6\" (UniqueName: \"kubernetes.io/projected/ec0b1a26-5b6c-405c-a042-7141708b6b22-kube-api-access-hbsm6\") pod \"maas-default-gateway-openshift-default-58c6585447-hdvw9\" (UID: \"ec0b1a26-5b6c-405c-a042-7141708b6b22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-hdvw9" Nov 01 21:17:33.423308 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.423272 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-hdvw9" Nov 01 21:17:33.475902 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.475869 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6zwld" Nov 01 21:17:33.483844 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.482273 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2n5h6" Nov 01 21:17:33.853628 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.853591 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-hdvw9"] Nov 01 21:17:33.895880 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:17:33.895848 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec0b1a26_5b6c_405c_a042_7141708b6b22.slice/crio-b13dcbb27389f75e2f8b657eaecb2cf1c118355ff5227c694712bc50efe2fa3f WatchSource:0}: Error finding container b13dcbb27389f75e2f8b657eaecb2cf1c118355ff5227c694712bc50efe2fa3f: Status 404 returned error can't find the container with id b13dcbb27389f75e2f8b657eaecb2cf1c118355ff5227c694712bc50efe2fa3f Nov 01 21:17:33.898078 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.898049 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:17:33.898145 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.898116 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:17:33.898189 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.898148 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:17:33.906683 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.906658 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2n5h6"] Nov 01 21:17:33.911554 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:17:33.911507 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35afbdf8_b193_42b5_9c8f_85471b4288b7.slice/crio-4366df2b04cd091dbd5fbe8b51560659deb6ae9dd4c8968e06cfdc9d81a50609 WatchSource:0}: Error finding container 4366df2b04cd091dbd5fbe8b51560659deb6ae9dd4c8968e06cfdc9d81a50609: Status 404 returned error can't find the container with id 4366df2b04cd091dbd5fbe8b51560659deb6ae9dd4c8968e06cfdc9d81a50609 Nov 01 21:17:33.913339 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.913306 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:17:33.913440 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.913379 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:17:33.913440 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.913422 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:17:33.956666 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.956641 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6zwld"] Nov 01 21:17:33.958199 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:17:33.958159 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e6a4f7c_75e1_4506_afb5_4ab00107a7f0.slice/crio-135a8b639689b8e947b71717cfb656d1383accc585faa706a4392f455f049ac6 WatchSource:0}: Error finding container 135a8b639689b8e947b71717cfb656d1383accc585faa706a4392f455f049ac6: Status 404 returned error can't find the container with id 135a8b639689b8e947b71717cfb656d1383accc585faa706a4392f455f049ac6 Nov 01 21:17:33.960509 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.960411 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:17:33.960509 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.960473 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:17:33.960509 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:33.960505 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:17:34.167437 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:34.167344 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6zwld" event={"ID":"8e6a4f7c-75e1-4506-afb5-4ab00107a7f0","Type":"ContainerStarted","Data":"c9b777e4fde78925d8f9a532335030b0dfbaf6d198729fbb6efd6f4736b40122"} Nov 01 21:17:34.167437 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:34.167409 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6zwld" event={"ID":"8e6a4f7c-75e1-4506-afb5-4ab00107a7f0","Type":"ContainerStarted","Data":"135a8b639689b8e947b71717cfb656d1383accc585faa706a4392f455f049ac6"} Nov 01 21:17:34.168985 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:34.168955 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-hdvw9" event={"ID":"ec0b1a26-5b6c-405c-a042-7141708b6b22","Type":"ContainerStarted","Data":"192367791b1cd7b5b98efbf801cfc63ec5ec9742a3f76f1c3dc7753f305d3db9"} Nov 01 21:17:34.169148 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:34.169025 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-hdvw9" event={"ID":"ec0b1a26-5b6c-405c-a042-7141708b6b22","Type":"ContainerStarted","Data":"b13dcbb27389f75e2f8b657eaecb2cf1c118355ff5227c694712bc50efe2fa3f"} Nov 01 21:17:34.170670 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:34.170645 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2n5h6" event={"ID":"35afbdf8-b193-42b5-9c8f-85471b4288b7","Type":"ContainerStarted","Data":"83d61343b766754a0443bda62ee8204519448aa5b8ee4a423219845b01ae1cd9"} Nov 01 21:17:34.170805 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:34.170689 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2n5h6" event={"ID":"35afbdf8-b193-42b5-9c8f-85471b4288b7","Type":"ContainerStarted","Data":"4366df2b04cd091dbd5fbe8b51560659deb6ae9dd4c8968e06cfdc9d81a50609"} Nov 01 21:17:34.294264 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:34.294180 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6zwld" podStartSLOduration=1.294156377 podStartE2EDuration="1.294156377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:17:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:17:34.284055414 +0000 UTC m=+2805.061695550" watchObservedRunningTime="2026-11-01 21:17:34.294156377 +0000 UTC m=+2805.071796513" Nov 01 21:17:34.403836 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:34.403775 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2n5h6" podStartSLOduration=1.403753803 podStartE2EDuration="1.403753803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:17:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:17:34.401416563 +0000 UTC m=+2805.179056700" watchObservedRunningTime="2026-11-01 21:17:34.403753803 +0000 UTC m=+2805.181393939" Nov 01 21:17:34.440077 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:34.434334 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-hdvw9" Nov 01 21:17:34.447035 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:34.446206 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-hdvw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.32:15021/healthz/ready\": dial tcp 10.134.1.32:15021: connect: connection refused" start-of-body= Nov 01 21:17:34.447035 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:34.446301 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-hdvw9" podUID="ec0b1a26-5b6c-405c-a042-7141708b6b22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.32:15021/healthz/ready\": dial tcp 10.134.1.32:15021: connect: connection refused" Nov 01 21:17:34.476631 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:34.476590 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6zwld" Nov 01 21:17:34.483222 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:34.483194 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2n5h6" Nov 01 21:17:34.484377 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:34.484357 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-2n5h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" start-of-body= Nov 01 21:17:34.484430 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:34.484409 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2n5h6" podUID="35afbdf8-b193-42b5-9c8f-85471b4288b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" Nov 01 21:17:34.529323 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:34.529249 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-hdvw9" podStartSLOduration=1.529227028 podStartE2EDuration="1.529227028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:17:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:17:34.526967655 +0000 UTC m=+2805.304607791" watchObservedRunningTime="2026-11-01 21:17:34.529227028 +0000 UTC m=+2805.306867157" Nov 01 21:17:35.428019 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:35.427448 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-hdvw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.32:15021/healthz/ready\": dial tcp 10.134.1.32:15021: connect: connection refused" start-of-body= Nov 01 21:17:35.428019 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:35.427511 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-hdvw9" podUID="ec0b1a26-5b6c-405c-a042-7141708b6b22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.32:15021/healthz/ready\": dial tcp 10.134.1.32:15021: connect: connection refused" Nov 01 21:17:35.480020 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:35.479048 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-6zwld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused" start-of-body= Nov 01 21:17:35.480020 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:35.479110 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6zwld" podUID="8e6a4f7c-75e1-4506-afb5-4ab00107a7f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused" Nov 01 21:17:35.485345 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:35.485317 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-2n5h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" start-of-body= Nov 01 21:17:35.485495 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:35.485376 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2n5h6" podUID="35afbdf8-b193-42b5-9c8f-85471b4288b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" Nov 01 21:17:36.427490 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:36.427447 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-hdvw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.32:15021/healthz/ready\": dial tcp 10.134.1.32:15021: connect: connection refused" start-of-body= Nov 01 21:17:36.428312 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:36.428269 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-hdvw9" podUID="ec0b1a26-5b6c-405c-a042-7141708b6b22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.32:15021/healthz/ready\": dial tcp 10.134.1.32:15021: connect: connection refused" Nov 01 21:17:36.476494 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:36.476452 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-6zwld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused" start-of-body= Nov 01 21:17:36.476850 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:36.476811 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6zwld" podUID="8e6a4f7c-75e1-4506-afb5-4ab00107a7f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused" Nov 01 21:17:36.483179 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:36.483152 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-2n5h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" start-of-body= Nov 01 21:17:36.483330 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:36.483311 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2n5h6" podUID="35afbdf8-b193-42b5-9c8f-85471b4288b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" Nov 01 21:17:37.427670 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:37.427616 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-hdvw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.32:15021/healthz/ready\": dial tcp 10.134.1.32:15021: connect: connection refused" start-of-body= Nov 01 21:17:37.428049 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:37.427974 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-hdvw9" podUID="ec0b1a26-5b6c-405c-a042-7141708b6b22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.32:15021/healthz/ready\": dial tcp 10.134.1.32:15021: connect: connection refused" Nov 01 21:17:37.478079 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:37.478027 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-6zwld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused" start-of-body= Nov 01 21:17:37.478784 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:37.478701 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6zwld" podUID="8e6a4f7c-75e1-4506-afb5-4ab00107a7f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused" Nov 01 21:17:37.483505 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:37.483480 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-2n5h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" start-of-body= Nov 01 21:17:37.483640 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:37.483524 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2n5h6" podUID="35afbdf8-b193-42b5-9c8f-85471b4288b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" Nov 01 21:17:38.426453 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:38.426411 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-hdvw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.32:15021/healthz/ready\": dial tcp 10.134.1.32:15021: connect: connection refused" start-of-body= Nov 01 21:17:38.426795 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:38.426767 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-hdvw9" podUID="ec0b1a26-5b6c-405c-a042-7141708b6b22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.32:15021/healthz/ready\": dial tcp 10.134.1.32:15021: connect: connection refused" Nov 01 21:17:38.476382 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:38.476350 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-6zwld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused" start-of-body= Nov 01 21:17:38.476629 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:38.476609 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6zwld" podUID="8e6a4f7c-75e1-4506-afb5-4ab00107a7f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused" Nov 01 21:17:38.482918 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:38.482892 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-2n5h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" start-of-body= Nov 01 21:17:38.483370 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:38.483348 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2n5h6" podUID="35afbdf8-b193-42b5-9c8f-85471b4288b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" Nov 01 21:17:39.424526 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:39.424485 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-hdvw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.32:15021/healthz/ready\": dial tcp 10.134.1.32:15021: connect: connection refused" start-of-body= Nov 01 21:17:39.424797 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:39.424770 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-hdvw9" podUID="ec0b1a26-5b6c-405c-a042-7141708b6b22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.32:15021/healthz/ready\": dial tcp 10.134.1.32:15021: connect: connection refused" Nov 01 21:17:39.477524 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:39.477485 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-6zwld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused" start-of-body= Nov 01 21:17:39.477844 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:39.477815 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6zwld" podUID="8e6a4f7c-75e1-4506-afb5-4ab00107a7f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused" Nov 01 21:17:39.486022 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:39.483357 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-2n5h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" start-of-body= Nov 01 21:17:39.486022 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:39.483406 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2n5h6" podUID="35afbdf8-b193-42b5-9c8f-85471b4288b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" Nov 01 21:17:39.679546 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:39.679457 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:17:40.424049 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:40.423970 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-hdvw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.32:15021/healthz/ready\": dial tcp 10.134.1.32:15021: connect: connection refused" start-of-body= Nov 01 21:17:40.424286 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:40.424070 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-hdvw9" podUID="ec0b1a26-5b6c-405c-a042-7141708b6b22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.32:15021/healthz/ready\": dial tcp 10.134.1.32:15021: connect: connection refused" Nov 01 21:17:40.476518 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:40.476479 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-6zwld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused" start-of-body= Nov 01 21:17:40.476725 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:40.476546 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6zwld" podUID="8e6a4f7c-75e1-4506-afb5-4ab00107a7f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused" Nov 01 21:17:40.482877 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:40.482845 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-2n5h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" start-of-body= Nov 01 21:17:40.483048 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:40.482904 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2n5h6" podUID="35afbdf8-b193-42b5-9c8f-85471b4288b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" Nov 01 21:17:41.424191 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:41.424149 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-hdvw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.32:15021/healthz/ready\": dial tcp 10.134.1.32:15021: connect: connection refused" start-of-body= Nov 01 21:17:41.424612 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:41.424218 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-hdvw9" podUID="ec0b1a26-5b6c-405c-a042-7141708b6b22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.32:15021/healthz/ready\": dial tcp 10.134.1.32:15021: connect: connection refused" Nov 01 21:17:41.478543 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:41.478501 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-6zwld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused" start-of-body= Nov 01 21:17:41.478736 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:41.478571 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6zwld" podUID="8e6a4f7c-75e1-4506-afb5-4ab00107a7f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused" Nov 01 21:17:41.484734 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:41.484701 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-2n5h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" start-of-body= Nov 01 21:17:41.484953 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:41.484928 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2n5h6" podUID="35afbdf8-b193-42b5-9c8f-85471b4288b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" Nov 01 21:17:41.961517 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:41.961474 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:17:42.866794 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:42.866755 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-hdvw9" Nov 01 21:17:42.867386 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:42.867357 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-hdvw9" Nov 01 21:17:42.868141 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:42.868119 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-hdvw9" Nov 01 21:17:42.884478 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:42.884453 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2n5h6" Nov 01 21:17:42.885010 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:42.884969 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2n5h6" Nov 01 21:17:42.885788 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:42.885772 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2n5h6" Nov 01 21:17:42.908722 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:42.908371 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6zwld" Nov 01 21:17:42.909018 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:42.908965 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6zwld" Nov 01 21:17:42.909614 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:42.909597 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6zwld" Nov 01 21:17:45.766536 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:17:45.766499 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:17:48.977486 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:17:48.977437 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:18:01.440466 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:01.440389 4484 scope.go:122] "RemoveContainer" containerID="d265ed2597a7b992c73050b65dbc231df667281560d1fdf482377cfa3294b8ee" Nov 01 21:18:01.449532 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:01.449508 4484 scope.go:122] "RemoveContainer" containerID="f6043b38f21aee17b7bb3e2955efccd8f3fab033b9c70ab7834d63c3ce9988da" Nov 01 21:18:01.458257 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:01.458231 4484 scope.go:122] "RemoveContainer" containerID="1a1ff6ff2d01bc10df37e5c5f158dd1424bee557006b80d116df54938fd33e56" Nov 01 21:18:15.766748 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:18:15.766704 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:18:35.632475 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:35.632440 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jhjxx"] Nov 01 21:18:35.636258 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:35.636235 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jhjxx" Nov 01 21:18:35.645489 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:35.645466 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae151e4d-e3ee-451a-8776-2e9426c391c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-jhjxx\" (UID: \"ae151e4d-e3ee-451a-8776-2e9426c391c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jhjxx" Nov 01 21:18:35.645623 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:35.645527 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae151e4d-e3ee-451a-8776-2e9426c391c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-jhjxx\" (UID: \"ae151e4d-e3ee-451a-8776-2e9426c391c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jhjxx" Nov 01 21:18:35.645701 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:35.645645 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4gtz\" (UniqueName: \"kubernetes.io/projected/ae151e4d-e3ee-451a-8776-2e9426c391c2-kube-api-access-h4gtz\") pod \"maas-api-revoke-sub-c1e271a9-jhjxx\" (UID: \"ae151e4d-e3ee-451a-8776-2e9426c391c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jhjxx" Nov 01 21:18:35.692289 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:35.692256 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jhjxx"] Nov 01 21:18:35.740395 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:35.740344 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:18:35.740395 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:35.740387 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:18:35.740617 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:35.740394 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:18:35.740822 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:35.740803 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:18:35.746386 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:35.746364 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4gtz\" (UniqueName: \"kubernetes.io/projected/ae151e4d-e3ee-451a-8776-2e9426c391c2-kube-api-access-h4gtz\") pod \"maas-api-revoke-sub-c1e271a9-jhjxx\" (UID: \"ae151e4d-e3ee-451a-8776-2e9426c391c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jhjxx" Nov 01 21:18:35.746507 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:35.746413 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae151e4d-e3ee-451a-8776-2e9426c391c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-jhjxx\" (UID: \"ae151e4d-e3ee-451a-8776-2e9426c391c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jhjxx" Nov 01 21:18:35.746507 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:35.746464 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae151e4d-e3ee-451a-8776-2e9426c391c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-jhjxx\" (UID: \"ae151e4d-e3ee-451a-8776-2e9426c391c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jhjxx" Nov 01 21:18:35.747079 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:35.747063 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae151e4d-e3ee-451a-8776-2e9426c391c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-jhjxx\" (UID: \"ae151e4d-e3ee-451a-8776-2e9426c391c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jhjxx" Nov 01 21:18:35.748865 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:35.748849 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae151e4d-e3ee-451a-8776-2e9426c391c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-jhjxx\" (UID: \"ae151e4d-e3ee-451a-8776-2e9426c391c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jhjxx" Nov 01 21:18:35.804224 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:35.804190 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4gtz\" (UniqueName: \"kubernetes.io/projected/ae151e4d-e3ee-451a-8776-2e9426c391c2-kube-api-access-h4gtz\") pod \"maas-api-revoke-sub-c1e271a9-jhjxx\" (UID: \"ae151e4d-e3ee-451a-8776-2e9426c391c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jhjxx" Nov 01 21:18:35.952636 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:35.952535 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jhjxx" Nov 01 21:18:36.330857 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:36.330820 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jhjxx"] Nov 01 21:18:36.332134 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:18:36.332101 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae151e4d_e3ee_451a_8776_2e9426c391c2.slice/crio-0ac65f3115d3e08e62419bd89fbe6af0e464e5e89b0f1aa6b8b4a0357db446c3 WatchSource:0}: Error finding container 0ac65f3115d3e08e62419bd89fbe6af0e464e5e89b0f1aa6b8b4a0357db446c3: Status 404 returned error can't find the container with id 0ac65f3115d3e08e62419bd89fbe6af0e464e5e89b0f1aa6b8b4a0357db446c3 Nov 01 21:18:36.600558 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:36.600458 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jhjxx" event={"ID":"ae151e4d-e3ee-451a-8776-2e9426c391c2","Type":"ContainerStarted","Data":"f29df4c262341b45d7dee686156be2212fbda5de114702874782c62ebf0face8"} Nov 01 21:18:36.600558 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:36.600510 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jhjxx" event={"ID":"ae151e4d-e3ee-451a-8776-2e9426c391c2","Type":"ContainerStarted","Data":"0ac65f3115d3e08e62419bd89fbe6af0e464e5e89b0f1aa6b8b4a0357db446c3"} Nov 01 21:18:36.715839 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:36.715784 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jhjxx" podStartSLOduration=1.715772216 podStartE2EDuration="1.715772216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:18:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:18:36.714765156 +0000 UTC m=+2867.492405291" watchObservedRunningTime="2026-11-01 21:18:36.715772216 +0000 UTC m=+2867.493412350" Nov 01 21:18:37.607519 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:37.607483 4484 generic.go:356] "Generic (PLEG): container finished" podID="ae151e4d-e3ee-451a-8776-2e9426c391c2" containerID="f29df4c262341b45d7dee686156be2212fbda5de114702874782c62ebf0face8" exitCode=0 Nov 01 21:18:37.607706 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:37.607563 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jhjxx" event={"ID":"ae151e4d-e3ee-451a-8776-2e9426c391c2","Type":"ContainerDied","Data":"f29df4c262341b45d7dee686156be2212fbda5de114702874782c62ebf0face8"} Nov 01 21:18:38.753398 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:38.753371 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jhjxx" Nov 01 21:18:38.779470 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:38.779442 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae151e4d-e3ee-451a-8776-2e9426c391c2-kube-api-access-h4gtz\" (UniqueName: \"kubernetes.io/projected/ae151e4d-e3ee-451a-8776-2e9426c391c2-kube-api-access-h4gtz\") pod \"ae151e4d-e3ee-451a-8776-2e9426c391c2\" (UID: \"ae151e4d-e3ee-451a-8776-2e9426c391c2\") " Nov 01 21:18:38.779589 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:38.779490 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae151e4d-e3ee-451a-8776-2e9426c391c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae151e4d-e3ee-451a-8776-2e9426c391c2-maas-api-xks-ca\") pod \"ae151e4d-e3ee-451a-8776-2e9426c391c2\" (UID: \"ae151e4d-e3ee-451a-8776-2e9426c391c2\") " Nov 01 21:18:38.779640 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:38.779626 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae151e4d-e3ee-451a-8776-2e9426c391c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae151e4d-e3ee-451a-8776-2e9426c391c2-maas-api-openshift-ca\") pod \"ae151e4d-e3ee-451a-8776-2e9426c391c2\" (UID: \"ae151e4d-e3ee-451a-8776-2e9426c391c2\") " Nov 01 21:18:38.779990 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:38.779959 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae151e4d-e3ee-451a-8776-2e9426c391c2-maas-api-openshift-ca" pod "ae151e4d-e3ee-451a-8776-2e9426c391c2" (UID: "ae151e4d-e3ee-451a-8776-2e9426c391c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:18:38.781557 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:38.781533 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae151e4d-e3ee-451a-8776-2e9426c391c2-maas-api-xks-ca" pod "ae151e4d-e3ee-451a-8776-2e9426c391c2" (UID: "ae151e4d-e3ee-451a-8776-2e9426c391c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:18:38.781667 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:38.781615 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae151e4d-e3ee-451a-8776-2e9426c391c2-kube-api-access-h4gtz" pod "ae151e4d-e3ee-451a-8776-2e9426c391c2" (UID: "ae151e4d-e3ee-451a-8776-2e9426c391c2"). InnerVolumeSpecName "kube-api-access-h4gtz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:18:38.881422 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:38.881341 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae151e4d-e3ee-451a-8776-2e9426c391c2-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:18:38.881422 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:38.881369 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae151e4d-e3ee-451a-8776-2e9426c391c2-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:18:38.881422 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:38.881379 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4gtz\" (UniqueName: \"kubernetes.io/projected/ae151e4d-e3ee-451a-8776-2e9426c391c2-kube-api-access-h4gtz\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:18:39.619653 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:39.619623 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jhjxx" Nov 01 21:18:39.619848 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:39.619661 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jhjxx" event={"ID":"ae151e4d-e3ee-451a-8776-2e9426c391c2","Type":"ContainerDied","Data":"0ac65f3115d3e08e62419bd89fbe6af0e464e5e89b0f1aa6b8b4a0357db446c3"} Nov 01 21:18:39.619848 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:39.619710 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ac65f3115d3e08e62419bd89fbe6af0e464e5e89b0f1aa6b8b4a0357db446c3" Nov 01 21:18:45.766692 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:18:45.766652 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:18:48.726039 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:48.725982 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qm8qv"] Nov 01 21:18:48.726674 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:48.726653 4484 cpu_manager.go:414] "containerMap: removing container" podUID="ae151e4d-e3ee-451a-8776-2e9426c391c2" containerName="revoke-keys" Nov 01 21:18:48.726781 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:48.726675 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae151e4d-e3ee-451a-8776-2e9426c391c2" containerName="revoke-keys" Nov 01 21:18:48.726781 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:48.726772 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae151e4d-e3ee-451a-8776-2e9426c391c2" containerName="revoke-keys" Nov 01 21:18:48.729774 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:48.729751 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qm8qv" Nov 01 21:18:48.783194 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:48.783158 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3754b0a-165a-4693-b99c-c7e112c594c1-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-qm8qv\" (UID: \"f3754b0a-165a-4693-b99c-c7e112c594c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qm8qv" Nov 01 21:18:48.783411 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:48.783205 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3754b0a-165a-4693-b99c-c7e112c594c1-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-qm8qv\" (UID: \"f3754b0a-165a-4693-b99c-c7e112c594c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qm8qv" Nov 01 21:18:48.783411 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:48.783245 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3754b0a-165a-4693-b99c-c7e112c594c1-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-qm8qv\" (UID: \"f3754b0a-165a-4693-b99c-c7e112c594c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qm8qv" Nov 01 21:18:48.783411 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:48.783263 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3754b0a-165a-4693-b99c-c7e112c594c1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-qm8qv\" (UID: \"f3754b0a-165a-4693-b99c-c7e112c594c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qm8qv" Nov 01 21:18:48.783411 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:48.783290 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3754b0a-165a-4693-b99c-c7e112c594c1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-qm8qv\" (UID: \"f3754b0a-165a-4693-b99c-c7e112c594c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qm8qv" Nov 01 21:18:48.783411 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:48.783331 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trqvc\" (UniqueName: \"kubernetes.io/projected/f3754b0a-165a-4693-b99c-c7e112c594c1-kube-api-access-trqvc\") pod \"maas-default-gateway-openshift-default-58c6585447-qm8qv\" (UID: \"f3754b0a-165a-4693-b99c-c7e112c594c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qm8qv" Nov 01 21:18:48.783411 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:48.783349 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3754b0a-165a-4693-b99c-c7e112c594c1-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-qm8qv\" (UID: \"f3754b0a-165a-4693-b99c-c7e112c594c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qm8qv" Nov 01 21:18:48.783411 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:48.783382 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3754b0a-165a-4693-b99c-c7e112c594c1-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-qm8qv\" (UID: \"f3754b0a-165a-4693-b99c-c7e112c594c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qm8qv" Nov 01 21:18:48.783411 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:48.783402 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3754b0a-165a-4693-b99c-c7e112c594c1-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-qm8qv\" (UID: \"f3754b0a-165a-4693-b99c-c7e112c594c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qm8qv" Nov 01 21:18:48.836471 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:48.836438 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qm8qv"] Nov 01 21:18:48.884262 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:48.884224 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3754b0a-165a-4693-b99c-c7e112c594c1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-qm8qv\" (UID: \"f3754b0a-165a-4693-b99c-c7e112c594c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qm8qv" Nov 01 21:18:48.884262 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:48.884264 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3754b0a-165a-4693-b99c-c7e112c594c1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-qm8qv\" (UID: \"f3754b0a-165a-4693-b99c-c7e112c594c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qm8qv" Nov 01 21:18:48.884521 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:48.884307 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trqvc\" (UniqueName: \"kubernetes.io/projected/f3754b0a-165a-4693-b99c-c7e112c594c1-kube-api-access-trqvc\") pod \"maas-default-gateway-openshift-default-58c6585447-qm8qv\" (UID: \"f3754b0a-165a-4693-b99c-c7e112c594c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qm8qv" Nov 01 21:18:48.884521 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:48.884326 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3754b0a-165a-4693-b99c-c7e112c594c1-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-qm8qv\" (UID: \"f3754b0a-165a-4693-b99c-c7e112c594c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qm8qv" Nov 01 21:18:48.884521 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:48.884351 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3754b0a-165a-4693-b99c-c7e112c594c1-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-qm8qv\" (UID: \"f3754b0a-165a-4693-b99c-c7e112c594c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qm8qv" Nov 01 21:18:48.884521 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:48.884409 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3754b0a-165a-4693-b99c-c7e112c594c1-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-qm8qv\" (UID: \"f3754b0a-165a-4693-b99c-c7e112c594c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qm8qv" Nov 01 21:18:48.884521 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:48.884466 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3754b0a-165a-4693-b99c-c7e112c594c1-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-qm8qv\" (UID: \"f3754b0a-165a-4693-b99c-c7e112c594c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qm8qv" Nov 01 21:18:48.884521 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:48.884506 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3754b0a-165a-4693-b99c-c7e112c594c1-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-qm8qv\" (UID: \"f3754b0a-165a-4693-b99c-c7e112c594c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qm8qv" Nov 01 21:18:48.884876 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:48.884548 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3754b0a-165a-4693-b99c-c7e112c594c1-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-qm8qv\" (UID: \"f3754b0a-165a-4693-b99c-c7e112c594c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qm8qv" Nov 01 21:18:48.884876 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:48.884784 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3754b0a-165a-4693-b99c-c7e112c594c1-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-qm8qv\" (UID: \"f3754b0a-165a-4693-b99c-c7e112c594c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qm8qv" Nov 01 21:18:48.884876 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:48.884815 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3754b0a-165a-4693-b99c-c7e112c594c1-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-qm8qv\" (UID: \"f3754b0a-165a-4693-b99c-c7e112c594c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qm8qv" Nov 01 21:18:48.885081 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:48.885021 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3754b0a-165a-4693-b99c-c7e112c594c1-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-qm8qv\" (UID: \"f3754b0a-165a-4693-b99c-c7e112c594c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qm8qv" Nov 01 21:18:48.885147 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:48.885124 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3754b0a-165a-4693-b99c-c7e112c594c1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-qm8qv\" (UID: \"f3754b0a-165a-4693-b99c-c7e112c594c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qm8qv" Nov 01 21:18:48.885189 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:48.885174 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3754b0a-165a-4693-b99c-c7e112c594c1-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-qm8qv\" (UID: \"f3754b0a-165a-4693-b99c-c7e112c594c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qm8qv" Nov 01 21:18:48.886751 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:48.886726 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3754b0a-165a-4693-b99c-c7e112c594c1-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-qm8qv\" (UID: \"f3754b0a-165a-4693-b99c-c7e112c594c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qm8qv" Nov 01 21:18:48.886954 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:48.886936 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3754b0a-165a-4693-b99c-c7e112c594c1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-qm8qv\" (UID: \"f3754b0a-165a-4693-b99c-c7e112c594c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qm8qv" Nov 01 21:18:48.942715 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:48.942680 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3754b0a-165a-4693-b99c-c7e112c594c1-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-qm8qv\" (UID: \"f3754b0a-165a-4693-b99c-c7e112c594c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qm8qv" Nov 01 21:18:48.942916 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:48.942889 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trqvc\" (UniqueName: \"kubernetes.io/projected/f3754b0a-165a-4693-b99c-c7e112c594c1-kube-api-access-trqvc\") pod \"maas-default-gateway-openshift-default-58c6585447-qm8qv\" (UID: \"f3754b0a-165a-4693-b99c-c7e112c594c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qm8qv" Nov 01 21:18:49.042753 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:49.042715 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qm8qv" Nov 01 21:18:49.333135 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:49.333037 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:18:49.482173 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:49.482149 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qm8qv"] Nov 01 21:18:49.489603 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:18:49.489565 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3754b0a_165a_4693_b99c_c7e112c594c1.slice/crio-5b36dae681fe24a879a98bc55a8848c026d772ad8a2cd1208c4d33e3259616bf WatchSource:0}: Error finding container 5b36dae681fe24a879a98bc55a8848c026d772ad8a2cd1208c4d33e3259616bf: Status 404 returned error can't find the container with id 5b36dae681fe24a879a98bc55a8848c026d772ad8a2cd1208c4d33e3259616bf Nov 01 21:18:49.491946 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:49.491911 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:18:49.492095 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:49.491982 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:18:49.492095 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:49.492031 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:18:49.681243 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:49.681144 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qm8qv" event={"ID":"f3754b0a-165a-4693-b99c-c7e112c594c1","Type":"ContainerStarted","Data":"e575bd0a74419db47776fc2e709ea8b405756332bdc7354daefd9317312964ea"} Nov 01 21:18:49.681243 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:49.681203 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qm8qv" event={"ID":"f3754b0a-165a-4693-b99c-c7e112c594c1","Type":"ContainerStarted","Data":"5b36dae681fe24a879a98bc55a8848c026d772ad8a2cd1208c4d33e3259616bf"} Nov 01 21:18:50.046022 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:50.045305 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qm8qv" Nov 01 21:18:50.047409 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:50.046709 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qm8qv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused" start-of-body= Nov 01 21:18:50.047409 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:50.046758 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qm8qv" podUID="f3754b0a-165a-4693-b99c-c7e112c594c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused" Nov 01 21:18:51.044021 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:51.043457 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qm8qv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused" start-of-body= Nov 01 21:18:51.044021 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:51.043524 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qm8qv" podUID="f3754b0a-165a-4693-b99c-c7e112c594c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused" Nov 01 21:18:52.043658 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:52.043620 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qm8qv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused" start-of-body= Nov 01 21:18:52.044101 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:52.043675 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qm8qv" podUID="f3754b0a-165a-4693-b99c-c7e112c594c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused" Nov 01 21:18:53.044137 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:53.044095 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qm8qv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused" start-of-body= Nov 01 21:18:53.044597 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:53.044158 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qm8qv" podUID="f3754b0a-165a-4693-b99c-c7e112c594c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused" Nov 01 21:18:54.043572 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:54.043525 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qm8qv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused" start-of-body= Nov 01 21:18:54.043804 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:54.043579 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qm8qv" podUID="f3754b0a-165a-4693-b99c-c7e112c594c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused" Nov 01 21:18:55.046472 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:55.046431 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qm8qv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused" start-of-body= Nov 01 21:18:55.047138 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:55.047103 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qm8qv" podUID="f3754b0a-165a-4693-b99c-c7e112c594c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused" Nov 01 21:18:56.044020 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:56.043413 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qm8qv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused" start-of-body= Nov 01 21:18:56.044020 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:56.043477 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qm8qv" podUID="f3754b0a-165a-4693-b99c-c7e112c594c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused" Nov 01 21:18:57.044064 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:57.044028 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qm8qv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused" start-of-body= Nov 01 21:18:57.044505 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:57.044082 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qm8qv" podUID="f3754b0a-165a-4693-b99c-c7e112c594c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused" Nov 01 21:18:59.044351 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:59.044303 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qm8qv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 21:18:59.044782 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:18:59.044382 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qm8qv" podUID="f3754b0a-165a-4693-b99c-c7e112c594c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 21:19:00.086200 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:00.086160 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qm8qv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 21:19:00.086688 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:00.086237 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qm8qv" podUID="f3754b0a-165a-4693-b99c-c7e112c594c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 21:19:00.458720 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:00.458647 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qm8qv" Nov 01 21:19:00.458959 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:00.458937 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qm8qv" Nov 01 21:19:00.459780 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:00.459760 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qm8qv" Nov 01 21:19:00.679637 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:00.679577 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qm8qv" podStartSLOduration=12.679564799 podStartE2EDuration="12.679564799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:18:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:18:49.803846682 +0000 UTC m=+2880.581486820" watchObservedRunningTime="2026-11-01 21:19:00.679564799 +0000 UTC m=+2891.457204998" Nov 01 21:19:01.487955 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:01.487924 4484 scope.go:122] "RemoveContainer" containerID="0c9be017ab8b64cb30115390bd04c820f7634ea9d8f9d310ac130c4a67860617" Nov 01 21:19:01.497191 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:01.497165 4484 scope.go:122] "RemoveContainer" containerID="e7c95b69061038095e1d78802157374152db76b31d40dbd0b47ce97c9123c7b1" Nov 01 21:19:06.767025 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:06.766957 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qd4qk"] Nov 01 21:19:06.772055 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:06.772029 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qd4qk" Nov 01 21:19:06.782777 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:06.782741 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc9aa86d-e00e-461d-8443-3c28b1f1882c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-qd4qk\" (UID: \"dc9aa86d-e00e-461d-8443-3c28b1f1882c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qd4qk" Nov 01 21:19:06.783016 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:06.782814 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29gvz\" (UniqueName: \"kubernetes.io/projected/dc9aa86d-e00e-461d-8443-3c28b1f1882c-kube-api-access-29gvz\") pod \"maas-api-revoke-sub-2e1f5b5f-qd4qk\" (UID: \"dc9aa86d-e00e-461d-8443-3c28b1f1882c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qd4qk" Nov 01 21:19:06.783114 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:06.783032 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc9aa86d-e00e-461d-8443-3c28b1f1882c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-qd4qk\" (UID: \"dc9aa86d-e00e-461d-8443-3c28b1f1882c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qd4qk" Nov 01 21:19:06.827289 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:06.827254 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qd4qk"] Nov 01 21:19:06.874968 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:06.874927 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:19:06.875172 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:06.874987 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:19:06.876239 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:06.876219 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:19:06.876368 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:06.876350 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:19:06.883836 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:06.883813 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc9aa86d-e00e-461d-8443-3c28b1f1882c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-qd4qk\" (UID: \"dc9aa86d-e00e-461d-8443-3c28b1f1882c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qd4qk" Nov 01 21:19:06.883896 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:06.883858 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29gvz\" (UniqueName: \"kubernetes.io/projected/dc9aa86d-e00e-461d-8443-3c28b1f1882c-kube-api-access-29gvz\") pod \"maas-api-revoke-sub-2e1f5b5f-qd4qk\" (UID: \"dc9aa86d-e00e-461d-8443-3c28b1f1882c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qd4qk" Nov 01 21:19:06.883938 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:06.883898 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc9aa86d-e00e-461d-8443-3c28b1f1882c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-qd4qk\" (UID: \"dc9aa86d-e00e-461d-8443-3c28b1f1882c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qd4qk" Nov 01 21:19:06.884491 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:06.884471 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc9aa86d-e00e-461d-8443-3c28b1f1882c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-qd4qk\" (UID: \"dc9aa86d-e00e-461d-8443-3c28b1f1882c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qd4qk" Nov 01 21:19:06.886588 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:06.886563 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc9aa86d-e00e-461d-8443-3c28b1f1882c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-qd4qk\" (UID: \"dc9aa86d-e00e-461d-8443-3c28b1f1882c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qd4qk" Nov 01 21:19:06.942447 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:06.942406 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29gvz\" (UniqueName: \"kubernetes.io/projected/dc9aa86d-e00e-461d-8443-3c28b1f1882c-kube-api-access-29gvz\") pod \"maas-api-revoke-sub-2e1f5b5f-qd4qk\" (UID: \"dc9aa86d-e00e-461d-8443-3c28b1f1882c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qd4qk" Nov 01 21:19:07.087771 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:07.087673 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qd4qk" Nov 01 21:19:07.457238 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:07.457215 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qd4qk"] Nov 01 21:19:07.458935 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:19:07.458902 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc9aa86d_e00e_461d_8443_3c28b1f1882c.slice/crio-2f21a0ee2eef1387cb0728d5e489149d5861e33db89da4bfe666e7b5308d9cf1 WatchSource:0}: Error finding container 2f21a0ee2eef1387cb0728d5e489149d5861e33db89da4bfe666e7b5308d9cf1: Status 404 returned error can't find the container with id 2f21a0ee2eef1387cb0728d5e489149d5861e33db89da4bfe666e7b5308d9cf1 Nov 01 21:19:07.856032 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:07.855981 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qd4qk" event={"ID":"dc9aa86d-e00e-461d-8443-3c28b1f1882c","Type":"ContainerStarted","Data":"3c0c6fbefbae3a70b1d3556759efc1a653a8bb8f8c6f200545c7db438821ce73"} Nov 01 21:19:07.856445 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:07.856048 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qd4qk" event={"ID":"dc9aa86d-e00e-461d-8443-3c28b1f1882c","Type":"ContainerStarted","Data":"2f21a0ee2eef1387cb0728d5e489149d5861e33db89da4bfe666e7b5308d9cf1"} Nov 01 21:19:07.971700 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:07.971645 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qd4qk" podStartSLOduration=1.971629794 podStartE2EDuration="1.971629794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:19:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:19:07.970158859 +0000 UTC m=+2898.747798994" watchObservedRunningTime="2026-11-01 21:19:07.971629794 +0000 UTC m=+2898.749269946" Nov 01 21:19:08.863528 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:08.863488 4484 generic.go:356] "Generic (PLEG): container finished" podID="dc9aa86d-e00e-461d-8443-3c28b1f1882c" containerID="3c0c6fbefbae3a70b1d3556759efc1a653a8bb8f8c6f200545c7db438821ce73" exitCode=0 Nov 01 21:19:08.863965 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:08.863570 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qd4qk" event={"ID":"dc9aa86d-e00e-461d-8443-3c28b1f1882c","Type":"ContainerDied","Data":"3c0c6fbefbae3a70b1d3556759efc1a653a8bb8f8c6f200545c7db438821ce73"} Nov 01 21:19:10.007413 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:10.007387 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qd4qk" Nov 01 21:19:10.116975 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:10.116936 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc9aa86d-e00e-461d-8443-3c28b1f1882c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc9aa86d-e00e-461d-8443-3c28b1f1882c-maas-api-openshift-ca\") pod \"dc9aa86d-e00e-461d-8443-3c28b1f1882c\" (UID: \"dc9aa86d-e00e-461d-8443-3c28b1f1882c\") " Nov 01 21:19:10.117186 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:10.117037 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc9aa86d-e00e-461d-8443-3c28b1f1882c-kube-api-access-29gvz\" (UniqueName: \"kubernetes.io/projected/dc9aa86d-e00e-461d-8443-3c28b1f1882c-kube-api-access-29gvz\") pod \"dc9aa86d-e00e-461d-8443-3c28b1f1882c\" (UID: \"dc9aa86d-e00e-461d-8443-3c28b1f1882c\") " Nov 01 21:19:10.117248 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:10.117199 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc9aa86d-e00e-461d-8443-3c28b1f1882c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc9aa86d-e00e-461d-8443-3c28b1f1882c-maas-api-xks-ca\") pod \"dc9aa86d-e00e-461d-8443-3c28b1f1882c\" (UID: \"dc9aa86d-e00e-461d-8443-3c28b1f1882c\") " Nov 01 21:19:10.117380 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:10.117353 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc9aa86d-e00e-461d-8443-3c28b1f1882c-maas-api-openshift-ca" pod "dc9aa86d-e00e-461d-8443-3c28b1f1882c" (UID: "dc9aa86d-e00e-461d-8443-3c28b1f1882c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:19:10.117555 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:10.117531 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc9aa86d-e00e-461d-8443-3c28b1f1882c-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:19:10.119215 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:10.119184 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc9aa86d-e00e-461d-8443-3c28b1f1882c-maas-api-xks-ca" pod "dc9aa86d-e00e-461d-8443-3c28b1f1882c" (UID: "dc9aa86d-e00e-461d-8443-3c28b1f1882c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:19:10.119353 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:10.119238 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc9aa86d-e00e-461d-8443-3c28b1f1882c-kube-api-access-29gvz" pod "dc9aa86d-e00e-461d-8443-3c28b1f1882c" (UID: "dc9aa86d-e00e-461d-8443-3c28b1f1882c"). InnerVolumeSpecName "kube-api-access-29gvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:19:10.218835 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:10.218782 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc9aa86d-e00e-461d-8443-3c28b1f1882c-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:19:10.218835 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:10.218830 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29gvz\" (UniqueName: \"kubernetes.io/projected/dc9aa86d-e00e-461d-8443-3c28b1f1882c-kube-api-access-29gvz\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:19:10.875609 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:10.875580 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qd4qk" Nov 01 21:19:10.875765 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:10.875575 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qd4qk" event={"ID":"dc9aa86d-e00e-461d-8443-3c28b1f1882c","Type":"ContainerDied","Data":"2f21a0ee2eef1387cb0728d5e489149d5861e33db89da4bfe666e7b5308d9cf1"} Nov 01 21:19:10.875765 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:10.875735 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f21a0ee2eef1387cb0728d5e489149d5861e33db89da4bfe666e7b5308d9cf1" Nov 01 21:19:15.766056 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:19:15.766020 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:19:19.048845 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:19.048807 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3819a541-dh9c6"] Nov 01 21:19:19.100133 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:19.100091 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04f5a54f-88jsj"] Nov 01 21:19:19.102271 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:19.102240 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3819a541-dh9c6"] Nov 01 21:19:19.104123 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:19.104101 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04f5a54f-88jsj"] Nov 01 21:19:19.184823 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:19.184790 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-z2bbt"] Nov 01 21:19:19.185300 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:19.185282 4484 cpu_manager.go:414] "containerMap: removing container" podUID="dc9aa86d-e00e-461d-8443-3c28b1f1882c" containerName="revoke-keys" Nov 01 21:19:19.185419 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:19.185301 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc9aa86d-e00e-461d-8443-3c28b1f1882c" containerName="revoke-keys" Nov 01 21:19:19.185486 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:19.185438 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc9aa86d-e00e-461d-8443-3c28b1f1882c" containerName="revoke-keys" Nov 01 21:19:19.188541 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:19.188519 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-z2bbt" Nov 01 21:19:19.242805 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:19.242774 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:19:19.243176 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:19.243153 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:19:19.243176 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:19.243145 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:19:19.243340 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:19.243123 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:19:19.244181 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:19.244161 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-z2bbt"] Nov 01 21:19:19.311564 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:19.311475 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4deb7c07-9529-4d3e-8ca0-ecec5f4d86fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-z2bbt\" (UID: \"4deb7c07-9529-4d3e-8ca0-ecec5f4d86fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-z2bbt" Nov 01 21:19:19.311564 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:19.311533 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4deb7c07-9529-4d3e-8ca0-ecec5f4d86fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-z2bbt\" (UID: \"4deb7c07-9529-4d3e-8ca0-ecec5f4d86fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-z2bbt" Nov 01 21:19:19.311755 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:19.311643 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9brgl\" (UniqueName: \"kubernetes.io/projected/4deb7c07-9529-4d3e-8ca0-ecec5f4d86fe-kube-api-access-9brgl\") pod \"maas-api-revoke-sub-ec843018-z2bbt\" (UID: \"4deb7c07-9529-4d3e-8ca0-ecec5f4d86fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-z2bbt" Nov 01 21:19:19.371899 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:19.371863 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58935e56-56fd-4275-bd56-5a4b28f2c186" path="/var/lib/kubelet/pods/58935e56-56fd-4275-bd56-5a4b28f2c186/volumes" Nov 01 21:19:19.372310 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:19.372296 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ae8b00c-b101-4a28-a618-a7f2b0ce4bed" path="/var/lib/kubelet/pods/5ae8b00c-b101-4a28-a618-a7f2b0ce4bed/volumes" Nov 01 21:19:19.413107 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:19.413071 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4deb7c07-9529-4d3e-8ca0-ecec5f4d86fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-z2bbt\" (UID: \"4deb7c07-9529-4d3e-8ca0-ecec5f4d86fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-z2bbt" Nov 01 21:19:19.413318 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:19.413134 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4deb7c07-9529-4d3e-8ca0-ecec5f4d86fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-z2bbt\" (UID: \"4deb7c07-9529-4d3e-8ca0-ecec5f4d86fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-z2bbt" Nov 01 21:19:19.413318 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:19.413217 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9brgl\" (UniqueName: \"kubernetes.io/projected/4deb7c07-9529-4d3e-8ca0-ecec5f4d86fe-kube-api-access-9brgl\") pod \"maas-api-revoke-sub-ec843018-z2bbt\" (UID: \"4deb7c07-9529-4d3e-8ca0-ecec5f4d86fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-z2bbt" Nov 01 21:19:19.413736 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:19.413715 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4deb7c07-9529-4d3e-8ca0-ecec5f4d86fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-z2bbt\" (UID: \"4deb7c07-9529-4d3e-8ca0-ecec5f4d86fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-z2bbt" Nov 01 21:19:19.415543 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:19.415521 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4deb7c07-9529-4d3e-8ca0-ecec5f4d86fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-z2bbt\" (UID: \"4deb7c07-9529-4d3e-8ca0-ecec5f4d86fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-z2bbt" Nov 01 21:19:19.471176 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:19.471144 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9brgl\" (UniqueName: \"kubernetes.io/projected/4deb7c07-9529-4d3e-8ca0-ecec5f4d86fe-kube-api-access-9brgl\") pod \"maas-api-revoke-sub-ec843018-z2bbt\" (UID: \"4deb7c07-9529-4d3e-8ca0-ecec5f4d86fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-z2bbt" Nov 01 21:19:19.501454 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:19.501414 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-z2bbt" Nov 01 21:19:19.874229 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:19.874198 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-z2bbt"] Nov 01 21:19:19.875673 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:19:19.875643 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4deb7c07_9529_4d3e_8ca0_ecec5f4d86fe.slice/crio-5e6c7fc26f3478fa5f9f385fa6c91a3b95a3d0f02719dab6a618a03a9e80167a WatchSource:0}: Error finding container 5e6c7fc26f3478fa5f9f385fa6c91a3b95a3d0f02719dab6a618a03a9e80167a: Status 404 returned error can't find the container with id 5e6c7fc26f3478fa5f9f385fa6c91a3b95a3d0f02719dab6a618a03a9e80167a Nov 01 21:19:19.936508 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:19.936468 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-z2bbt" event={"ID":"4deb7c07-9529-4d3e-8ca0-ecec5f4d86fe","Type":"ContainerStarted","Data":"5e6c7fc26f3478fa5f9f385fa6c91a3b95a3d0f02719dab6a618a03a9e80167a"} Nov 01 21:19:20.946907 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:20.946869 4484 generic.go:356] "Generic (PLEG): container finished" podID="4deb7c07-9529-4d3e-8ca0-ecec5f4d86fe" containerID="32a3ffb0489e72b796f5b84776741766d6037d34284ee9d8a9fc573eef46e36a" exitCode=0 Nov 01 21:19:20.947342 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:20.946961 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-z2bbt" event={"ID":"4deb7c07-9529-4d3e-8ca0-ecec5f4d86fe","Type":"ContainerDied","Data":"32a3ffb0489e72b796f5b84776741766d6037d34284ee9d8a9fc573eef46e36a"} Nov 01 21:19:22.090739 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:22.090713 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-z2bbt" Nov 01 21:19:22.150302 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:22.150260 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4deb7c07-9529-4d3e-8ca0-ecec5f4d86fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4deb7c07-9529-4d3e-8ca0-ecec5f4d86fe-maas-api-xks-ca\") pod \"4deb7c07-9529-4d3e-8ca0-ecec5f4d86fe\" (UID: \"4deb7c07-9529-4d3e-8ca0-ecec5f4d86fe\") " Nov 01 21:19:22.150483 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:22.150365 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4deb7c07-9529-4d3e-8ca0-ecec5f4d86fe-kube-api-access-9brgl\" (UniqueName: \"kubernetes.io/projected/4deb7c07-9529-4d3e-8ca0-ecec5f4d86fe-kube-api-access-9brgl\") pod \"4deb7c07-9529-4d3e-8ca0-ecec5f4d86fe\" (UID: \"4deb7c07-9529-4d3e-8ca0-ecec5f4d86fe\") " Nov 01 21:19:22.150566 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:22.150540 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4deb7c07-9529-4d3e-8ca0-ecec5f4d86fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4deb7c07-9529-4d3e-8ca0-ecec5f4d86fe-maas-api-openshift-ca\") pod \"4deb7c07-9529-4d3e-8ca0-ecec5f4d86fe\" (UID: \"4deb7c07-9529-4d3e-8ca0-ecec5f4d86fe\") " Nov 01 21:19:22.150916 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:22.150889 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4deb7c07-9529-4d3e-8ca0-ecec5f4d86fe-maas-api-openshift-ca" pod "4deb7c07-9529-4d3e-8ca0-ecec5f4d86fe" (UID: "4deb7c07-9529-4d3e-8ca0-ecec5f4d86fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:19:22.152362 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:22.152335 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4deb7c07-9529-4d3e-8ca0-ecec5f4d86fe-maas-api-xks-ca" pod "4deb7c07-9529-4d3e-8ca0-ecec5f4d86fe" (UID: "4deb7c07-9529-4d3e-8ca0-ecec5f4d86fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:19:22.152557 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:22.152534 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4deb7c07-9529-4d3e-8ca0-ecec5f4d86fe-kube-api-access-9brgl" pod "4deb7c07-9529-4d3e-8ca0-ecec5f4d86fe" (UID: "4deb7c07-9529-4d3e-8ca0-ecec5f4d86fe"). InnerVolumeSpecName "kube-api-access-9brgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:19:22.251757 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:22.251722 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4deb7c07-9529-4d3e-8ca0-ecec5f4d86fe-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:19:22.251757 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:22.251759 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4deb7c07-9529-4d3e-8ca0-ecec5f4d86fe-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:19:22.251985 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:22.251770 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9brgl\" (UniqueName: \"kubernetes.io/projected/4deb7c07-9529-4d3e-8ca0-ecec5f4d86fe-kube-api-access-9brgl\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:19:22.959599 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:22.959558 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-z2bbt" event={"ID":"4deb7c07-9529-4d3e-8ca0-ecec5f4d86fe","Type":"ContainerDied","Data":"5e6c7fc26f3478fa5f9f385fa6c91a3b95a3d0f02719dab6a618a03a9e80167a"} Nov 01 21:19:22.959793 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:22.959612 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e6c7fc26f3478fa5f9f385fa6c91a3b95a3d0f02719dab6a618a03a9e80167a" Nov 01 21:19:22.959793 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:22.959572 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-z2bbt" Nov 01 21:19:45.767076 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:19:45.767034 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:19:49.672740 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:49.672703 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:19:50.564975 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:50.564889 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:19:54.261557 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:54.261517 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qfprx"] Nov 01 21:19:54.262022 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:54.261984 4484 cpu_manager.go:414] "containerMap: removing container" podUID="4deb7c07-9529-4d3e-8ca0-ecec5f4d86fe" containerName="revoke-keys" Nov 01 21:19:54.262022 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:54.262011 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4deb7c07-9529-4d3e-8ca0-ecec5f4d86fe" containerName="revoke-keys" Nov 01 21:19:54.262116 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:54.262088 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="4deb7c07-9529-4d3e-8ca0-ecec5f4d86fe" containerName="revoke-keys" Nov 01 21:19:54.265542 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:54.265526 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qfprx" Nov 01 21:19:54.321151 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:54.321113 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qfprx"] Nov 01 21:19:54.325323 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:54.325290 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3c70aaa-58e4-41f8-8a2c-7b723c51d82d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-qfprx\" (UID: \"f3c70aaa-58e4-41f8-8a2c-7b723c51d82d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qfprx" Nov 01 21:19:54.325496 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:54.325404 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3c70aaa-58e4-41f8-8a2c-7b723c51d82d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-qfprx\" (UID: \"f3c70aaa-58e4-41f8-8a2c-7b723c51d82d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qfprx" Nov 01 21:19:54.325496 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:54.325481 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5w5s\" (UniqueName: \"kubernetes.io/projected/f3c70aaa-58e4-41f8-8a2c-7b723c51d82d-kube-api-access-n5w5s\") pod \"maas-api-revoke-sub-046e1343-qfprx\" (UID: \"f3c70aaa-58e4-41f8-8a2c-7b723c51d82d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qfprx" Nov 01 21:19:54.368809 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:54.368775 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:19:54.368809 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:54.368818 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:19:54.369072 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:54.368829 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:19:54.372298 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:54.372279 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:19:54.426937 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:54.426901 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3c70aaa-58e4-41f8-8a2c-7b723c51d82d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-qfprx\" (UID: \"f3c70aaa-58e4-41f8-8a2c-7b723c51d82d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qfprx" Nov 01 21:19:54.426937 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:54.426939 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3c70aaa-58e4-41f8-8a2c-7b723c51d82d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-qfprx\" (UID: \"f3c70aaa-58e4-41f8-8a2c-7b723c51d82d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qfprx" Nov 01 21:19:54.427186 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:54.426960 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5w5s\" (UniqueName: \"kubernetes.io/projected/f3c70aaa-58e4-41f8-8a2c-7b723c51d82d-kube-api-access-n5w5s\") pod \"maas-api-revoke-sub-046e1343-qfprx\" (UID: \"f3c70aaa-58e4-41f8-8a2c-7b723c51d82d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qfprx" Nov 01 21:19:54.427649 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:54.427625 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3c70aaa-58e4-41f8-8a2c-7b723c51d82d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-qfprx\" (UID: \"f3c70aaa-58e4-41f8-8a2c-7b723c51d82d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qfprx" Nov 01 21:19:54.429744 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:54.429710 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3c70aaa-58e4-41f8-8a2c-7b723c51d82d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-qfprx\" (UID: \"f3c70aaa-58e4-41f8-8a2c-7b723c51d82d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qfprx" Nov 01 21:19:54.485089 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:54.485049 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5w5s\" (UniqueName: \"kubernetes.io/projected/f3c70aaa-58e4-41f8-8a2c-7b723c51d82d-kube-api-access-n5w5s\") pod \"maas-api-revoke-sub-046e1343-qfprx\" (UID: \"f3c70aaa-58e4-41f8-8a2c-7b723c51d82d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qfprx" Nov 01 21:19:54.578949 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:54.578857 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qfprx" Nov 01 21:19:54.950628 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:54.950587 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qfprx"] Nov 01 21:19:54.951935 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:19:54.951906 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3c70aaa_58e4_41f8_8a2c_7b723c51d82d.slice/crio-f7249ec6b55da9199f92d24cdb2bcafe415a81e616fe9243cd7cb8f535457cf5 WatchSource:0}: Error finding container f7249ec6b55da9199f92d24cdb2bcafe415a81e616fe9243cd7cb8f535457cf5: Status 404 returned error can't find the container with id f7249ec6b55da9199f92d24cdb2bcafe415a81e616fe9243cd7cb8f535457cf5 Nov 01 21:19:55.177745 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:55.177710 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qfprx" event={"ID":"f3c70aaa-58e4-41f8-8a2c-7b723c51d82d","Type":"ContainerStarted","Data":"f7249ec6b55da9199f92d24cdb2bcafe415a81e616fe9243cd7cb8f535457cf5"} Nov 01 21:19:56.185317 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:56.185279 4484 generic.go:356] "Generic (PLEG): container finished" podID="f3c70aaa-58e4-41f8-8a2c-7b723c51d82d" containerID="3a2f970ae1f6e8163d4ab731b32b57801226321d5ac90b8c79fb4f98cec85bbe" exitCode=0 Nov 01 21:19:56.185780 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:56.185369 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qfprx" event={"ID":"f3c70aaa-58e4-41f8-8a2c-7b723c51d82d","Type":"ContainerDied","Data":"3a2f970ae1f6e8163d4ab731b32b57801226321d5ac90b8c79fb4f98cec85bbe"} Nov 01 21:19:57.332211 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:57.332182 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qfprx" Nov 01 21:19:57.455501 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:57.455419 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3c70aaa-58e4-41f8-8a2c-7b723c51d82d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3c70aaa-58e4-41f8-8a2c-7b723c51d82d-maas-api-openshift-ca\") pod \"f3c70aaa-58e4-41f8-8a2c-7b723c51d82d\" (UID: \"f3c70aaa-58e4-41f8-8a2c-7b723c51d82d\") " Nov 01 21:19:57.455681 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:57.455510 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3c70aaa-58e4-41f8-8a2c-7b723c51d82d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3c70aaa-58e4-41f8-8a2c-7b723c51d82d-maas-api-xks-ca\") pod \"f3c70aaa-58e4-41f8-8a2c-7b723c51d82d\" (UID: \"f3c70aaa-58e4-41f8-8a2c-7b723c51d82d\") " Nov 01 21:19:57.455788 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:57.455697 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3c70aaa-58e4-41f8-8a2c-7b723c51d82d-kube-api-access-n5w5s\" (UniqueName: \"kubernetes.io/projected/f3c70aaa-58e4-41f8-8a2c-7b723c51d82d-kube-api-access-n5w5s\") pod \"f3c70aaa-58e4-41f8-8a2c-7b723c51d82d\" (UID: \"f3c70aaa-58e4-41f8-8a2c-7b723c51d82d\") " Nov 01 21:19:57.456026 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:57.455973 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3c70aaa-58e4-41f8-8a2c-7b723c51d82d-maas-api-openshift-ca" pod "f3c70aaa-58e4-41f8-8a2c-7b723c51d82d" (UID: "f3c70aaa-58e4-41f8-8a2c-7b723c51d82d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:19:57.456184 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:57.456164 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3c70aaa-58e4-41f8-8a2c-7b723c51d82d-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:19:57.457742 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:57.457713 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3c70aaa-58e4-41f8-8a2c-7b723c51d82d-maas-api-xks-ca" pod "f3c70aaa-58e4-41f8-8a2c-7b723c51d82d" (UID: "f3c70aaa-58e4-41f8-8a2c-7b723c51d82d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:19:57.457833 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:57.457808 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3c70aaa-58e4-41f8-8a2c-7b723c51d82d-kube-api-access-n5w5s" pod "f3c70aaa-58e4-41f8-8a2c-7b723c51d82d" (UID: "f3c70aaa-58e4-41f8-8a2c-7b723c51d82d"). InnerVolumeSpecName "kube-api-access-n5w5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:19:57.557766 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:57.557732 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5w5s\" (UniqueName: \"kubernetes.io/projected/f3c70aaa-58e4-41f8-8a2c-7b723c51d82d-kube-api-access-n5w5s\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:19:57.557766 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:57.557762 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3c70aaa-58e4-41f8-8a2c-7b723c51d82d-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:19:58.198320 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:58.198289 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qfprx" Nov 01 21:19:58.198502 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:58.198320 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qfprx" event={"ID":"f3c70aaa-58e4-41f8-8a2c-7b723c51d82d","Type":"ContainerDied","Data":"f7249ec6b55da9199f92d24cdb2bcafe415a81e616fe9243cd7cb8f535457cf5"} Nov 01 21:19:58.198502 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:19:58.198366 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7249ec6b55da9199f92d24cdb2bcafe415a81e616fe9243cd7cb8f535457cf5" Nov 01 21:20:01.523783 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:01.523750 4484 scope.go:122] "RemoveContainer" containerID="9d3642cfeab94c1bee39a239537eb8b2e1c2d07be25c1df1ede20b92dae8dfda" Nov 01 21:20:01.533077 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:01.533054 4484 scope.go:122] "RemoveContainer" containerID="edd06cad245f511642863abf4ca35484a6eaf6647204b9f2f26b9f9fab41c1ca" Nov 01 21:20:01.541418 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:01.541393 4484 scope.go:122] "RemoveContainer" containerID="e848dca57f980c5b40af5b7e753a9fe3394ec47c035c7c03547c11b82431d8a0" Nov 01 21:20:01.549656 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:01.549631 4484 scope.go:122] "RemoveContainer" containerID="dc1c3c34e21cbbfb2932534448e3e950c228f511dc50741739e51bc79f550cc3" Nov 01 21:20:04.997896 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:04.997858 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-t5q9j"] Nov 01 21:20:04.998343 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:04.998304 4484 cpu_manager.go:414] "containerMap: removing container" podUID="f3c70aaa-58e4-41f8-8a2c-7b723c51d82d" containerName="revoke-keys" Nov 01 21:20:04.998343 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:04.998316 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3c70aaa-58e4-41f8-8a2c-7b723c51d82d" containerName="revoke-keys" Nov 01 21:20:04.998445 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:04.998382 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3c70aaa-58e4-41f8-8a2c-7b723c51d82d" containerName="revoke-keys" Nov 01 21:20:05.003013 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:05.002977 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-t5q9j" Nov 01 21:20:05.057742 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:05.057703 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-t5q9j"] Nov 01 21:20:05.105342 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:05.105309 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:20:05.105342 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:05.105320 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:20:05.105578 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:05.105463 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:20:05.107196 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:05.107181 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:20:05.135814 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:05.135773 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6zkv\" (UniqueName: \"kubernetes.io/projected/abf725d3-0862-4578-9102-049f532baf95-kube-api-access-s6zkv\") pod \"maas-api-revoke-sub-d3fba056-t5q9j\" (UID: \"abf725d3-0862-4578-9102-049f532baf95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-t5q9j" Nov 01 21:20:05.135988 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:05.135833 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abf725d3-0862-4578-9102-049f532baf95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-t5q9j\" (UID: \"abf725d3-0862-4578-9102-049f532baf95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-t5q9j" Nov 01 21:20:05.135988 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:05.135926 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abf725d3-0862-4578-9102-049f532baf95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-t5q9j\" (UID: \"abf725d3-0862-4578-9102-049f532baf95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-t5q9j" Nov 01 21:20:05.237095 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:05.237055 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6zkv\" (UniqueName: \"kubernetes.io/projected/abf725d3-0862-4578-9102-049f532baf95-kube-api-access-s6zkv\") pod \"maas-api-revoke-sub-d3fba056-t5q9j\" (UID: \"abf725d3-0862-4578-9102-049f532baf95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-t5q9j" Nov 01 21:20:05.237306 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:05.237118 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abf725d3-0862-4578-9102-049f532baf95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-t5q9j\" (UID: \"abf725d3-0862-4578-9102-049f532baf95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-t5q9j" Nov 01 21:20:05.237306 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:05.237139 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abf725d3-0862-4578-9102-049f532baf95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-t5q9j\" (UID: \"abf725d3-0862-4578-9102-049f532baf95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-t5q9j" Nov 01 21:20:05.237831 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:05.237807 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abf725d3-0862-4578-9102-049f532baf95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-t5q9j\" (UID: \"abf725d3-0862-4578-9102-049f532baf95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-t5q9j" Nov 01 21:20:05.239516 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:05.239499 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abf725d3-0862-4578-9102-049f532baf95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-t5q9j\" (UID: \"abf725d3-0862-4578-9102-049f532baf95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-t5q9j" Nov 01 21:20:05.295497 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:05.295421 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6zkv\" (UniqueName: \"kubernetes.io/projected/abf725d3-0862-4578-9102-049f532baf95-kube-api-access-s6zkv\") pod \"maas-api-revoke-sub-d3fba056-t5q9j\" (UID: \"abf725d3-0862-4578-9102-049f532baf95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-t5q9j" Nov 01 21:20:05.316021 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:05.315973 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-t5q9j" Nov 01 21:20:05.692690 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:05.692659 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-t5q9j"] Nov 01 21:20:05.693984 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:20:05.693956 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabf725d3_0862_4578_9102_049f532baf95.slice/crio-b092b1dd8aec3089ce2f8c413ff12c7e1d238ca1ab8e6a3de193f6f87947c4f3 WatchSource:0}: Error finding container b092b1dd8aec3089ce2f8c413ff12c7e1d238ca1ab8e6a3de193f6f87947c4f3: Status 404 returned error can't find the container with id b092b1dd8aec3089ce2f8c413ff12c7e1d238ca1ab8e6a3de193f6f87947c4f3 Nov 01 21:20:06.246540 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:06.246507 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-t5q9j" event={"ID":"abf725d3-0862-4578-9102-049f532baf95","Type":"ContainerStarted","Data":"2a72172308db3f7ffa01deb33fce07c9c352d34fa30ffe7a0a760b0f7b404de9"} Nov 01 21:20:06.246956 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:06.246557 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-t5q9j" event={"ID":"abf725d3-0862-4578-9102-049f532baf95","Type":"ContainerStarted","Data":"b092b1dd8aec3089ce2f8c413ff12c7e1d238ca1ab8e6a3de193f6f87947c4f3"} Nov 01 21:20:06.363193 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:06.363073 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-t5q9j" podStartSLOduration=2.363058397 podStartE2EDuration="2.363058397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:20:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:20:06.361615309 +0000 UTC m=+2957.139255444" watchObservedRunningTime="2026-11-01 21:20:06.363058397 +0000 UTC m=+2957.140698529" Nov 01 21:20:07.255391 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:07.255355 4484 generic.go:356] "Generic (PLEG): container finished" podID="abf725d3-0862-4578-9102-049f532baf95" containerID="2a72172308db3f7ffa01deb33fce07c9c352d34fa30ffe7a0a760b0f7b404de9" exitCode=0 Nov 01 21:20:07.255832 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:07.255443 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-t5q9j" event={"ID":"abf725d3-0862-4578-9102-049f532baf95","Type":"ContainerDied","Data":"2a72172308db3f7ffa01deb33fce07c9c352d34fa30ffe7a0a760b0f7b404de9"} Nov 01 21:20:08.989219 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:08.989184 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-t5q9j" Nov 01 21:20:09.023183 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:09.023148 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abf725d3-0862-4578-9102-049f532baf95-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abf725d3-0862-4578-9102-049f532baf95-maas-api-xks-ca\") pod \"abf725d3-0862-4578-9102-049f532baf95\" (UID: \"abf725d3-0862-4578-9102-049f532baf95\") " Nov 01 21:20:09.023889 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:09.023866 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abf725d3-0862-4578-9102-049f532baf95-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abf725d3-0862-4578-9102-049f532baf95-maas-api-openshift-ca\") pod \"abf725d3-0862-4578-9102-049f532baf95\" (UID: \"abf725d3-0862-4578-9102-049f532baf95\") " Nov 01 21:20:09.024591 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:09.024570 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abf725d3-0862-4578-9102-049f532baf95-kube-api-access-s6zkv\" (UniqueName: \"kubernetes.io/projected/abf725d3-0862-4578-9102-049f532baf95-kube-api-access-s6zkv\") pod \"abf725d3-0862-4578-9102-049f532baf95\" (UID: \"abf725d3-0862-4578-9102-049f532baf95\") " Nov 01 21:20:09.025371 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:09.024369 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abf725d3-0862-4578-9102-049f532baf95-maas-api-openshift-ca" pod "abf725d3-0862-4578-9102-049f532baf95" (UID: "abf725d3-0862-4578-9102-049f532baf95"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:20:09.061288 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:09.059357 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abf725d3-0862-4578-9102-049f532baf95-maas-api-xks-ca" pod "abf725d3-0862-4578-9102-049f532baf95" (UID: "abf725d3-0862-4578-9102-049f532baf95"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:20:09.061685 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:09.061636 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abf725d3-0862-4578-9102-049f532baf95-kube-api-access-s6zkv" pod "abf725d3-0862-4578-9102-049f532baf95" (UID: "abf725d3-0862-4578-9102-049f532baf95"). InnerVolumeSpecName "kube-api-access-s6zkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:20:09.128208 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:09.128121 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6zkv\" (UniqueName: \"kubernetes.io/projected/abf725d3-0862-4578-9102-049f532baf95-kube-api-access-s6zkv\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:20:09.128208 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:09.128157 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abf725d3-0862-4578-9102-049f532baf95-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:20:09.128208 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:09.128177 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abf725d3-0862-4578-9102-049f532baf95-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:20:09.453301 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:09.446055 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-t5q9j" Nov 01 21:20:09.453301 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:09.448170 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-t5q9j" event={"ID":"abf725d3-0862-4578-9102-049f532baf95","Type":"ContainerDied","Data":"b092b1dd8aec3089ce2f8c413ff12c7e1d238ca1ab8e6a3de193f6f87947c4f3"} Nov 01 21:20:09.453301 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:09.448218 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b092b1dd8aec3089ce2f8c413ff12c7e1d238ca1ab8e6a3de193f6f87947c4f3" Nov 01 21:20:15.767123 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:20:15.767078 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:20:29.390220 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:29.390179 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:20:29.542742 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:29.542708 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:20:42.674714 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:42.674683 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-c5lpf"] Nov 01 21:20:42.675161 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:42.675134 4484 cpu_manager.go:414] "containerMap: removing container" podUID="abf725d3-0862-4578-9102-049f532baf95" containerName="revoke-keys" Nov 01 21:20:42.675161 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:42.675147 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abf725d3-0862-4578-9102-049f532baf95" containerName="revoke-keys" Nov 01 21:20:42.675241 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:42.675223 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="abf725d3-0862-4578-9102-049f532baf95" containerName="revoke-keys" Nov 01 21:20:42.678652 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:42.678628 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-c5lpf" Nov 01 21:20:42.739613 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:42.739574 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-c5lpf"] Nov 01 21:20:42.781974 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:42.781942 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:20:42.781974 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:42.781940 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:20:42.781974 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:42.781942 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:20:42.783482 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:42.783460 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:20:42.875133 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:42.875098 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d6d00d0-eb9a-474a-a942-d6186f886d85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-c5lpf\" (UID: \"0d6d00d0-eb9a-474a-a942-d6186f886d85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-c5lpf" Nov 01 21:20:42.875133 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:42.875147 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72hd5\" (UniqueName: \"kubernetes.io/projected/0d6d00d0-eb9a-474a-a942-d6186f886d85-kube-api-access-72hd5\") pod \"maas-api-revoke-sub-ae594979-c5lpf\" (UID: \"0d6d00d0-eb9a-474a-a942-d6186f886d85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-c5lpf" Nov 01 21:20:42.875401 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:42.875315 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d6d00d0-eb9a-474a-a942-d6186f886d85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-c5lpf\" (UID: \"0d6d00d0-eb9a-474a-a942-d6186f886d85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-c5lpf" Nov 01 21:20:42.975984 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:42.975879 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d6d00d0-eb9a-474a-a942-d6186f886d85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-c5lpf\" (UID: \"0d6d00d0-eb9a-474a-a942-d6186f886d85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-c5lpf" Nov 01 21:20:42.975984 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:42.975942 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d6d00d0-eb9a-474a-a942-d6186f886d85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-c5lpf\" (UID: \"0d6d00d0-eb9a-474a-a942-d6186f886d85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-c5lpf" Nov 01 21:20:42.976247 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:42.976023 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72hd5\" (UniqueName: \"kubernetes.io/projected/0d6d00d0-eb9a-474a-a942-d6186f886d85-kube-api-access-72hd5\") pod \"maas-api-revoke-sub-ae594979-c5lpf\" (UID: \"0d6d00d0-eb9a-474a-a942-d6186f886d85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-c5lpf" Nov 01 21:20:42.976717 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:42.976684 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d6d00d0-eb9a-474a-a942-d6186f886d85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-c5lpf\" (UID: \"0d6d00d0-eb9a-474a-a942-d6186f886d85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-c5lpf" Nov 01 21:20:42.978315 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:42.978290 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d6d00d0-eb9a-474a-a942-d6186f886d85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-c5lpf\" (UID: \"0d6d00d0-eb9a-474a-a942-d6186f886d85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-c5lpf" Nov 01 21:20:43.033799 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:43.033765 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72hd5\" (UniqueName: \"kubernetes.io/projected/0d6d00d0-eb9a-474a-a942-d6186f886d85-kube-api-access-72hd5\") pod \"maas-api-revoke-sub-ae594979-c5lpf\" (UID: \"0d6d00d0-eb9a-474a-a942-d6186f886d85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-c5lpf" Nov 01 21:20:43.291476 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:43.291434 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-c5lpf" Nov 01 21:20:43.662652 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:43.662618 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-c5lpf"] Nov 01 21:20:43.663948 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:20:43.663917 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d6d00d0_eb9a_474a_a942_d6186f886d85.slice/crio-4cab8e0d4cdca4b3208d8a241133f6e8fce3d45fd60ff91f8f165b0d3738e416 WatchSource:0}: Error finding container 4cab8e0d4cdca4b3208d8a241133f6e8fce3d45fd60ff91f8f165b0d3738e416: Status 404 returned error can't find the container with id 4cab8e0d4cdca4b3208d8a241133f6e8fce3d45fd60ff91f8f165b0d3738e416 Nov 01 21:20:43.731545 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:43.731510 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-c5lpf" event={"ID":"0d6d00d0-eb9a-474a-a942-d6186f886d85","Type":"ContainerStarted","Data":"4cab8e0d4cdca4b3208d8a241133f6e8fce3d45fd60ff91f8f165b0d3738e416"} Nov 01 21:20:44.738923 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:44.738884 4484 generic.go:356] "Generic (PLEG): container finished" podID="0d6d00d0-eb9a-474a-a942-d6186f886d85" containerID="43509f35fba6724b6922ccd12f9f8df646362f41df75bb8b46ac6ceb88824629" exitCode=0 Nov 01 21:20:44.739366 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:44.738979 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-c5lpf" event={"ID":"0d6d00d0-eb9a-474a-a942-d6186f886d85","Type":"ContainerDied","Data":"43509f35fba6724b6922ccd12f9f8df646362f41df75bb8b46ac6ceb88824629"} Nov 01 21:20:45.769731 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:20:45.769692 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:20:45.883764 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:45.883738 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-c5lpf" Nov 01 21:20:45.899768 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:45.899736 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d6d00d0-eb9a-474a-a942-d6186f886d85-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d6d00d0-eb9a-474a-a942-d6186f886d85-maas-api-xks-ca\") pod \"0d6d00d0-eb9a-474a-a942-d6186f886d85\" (UID: \"0d6d00d0-eb9a-474a-a942-d6186f886d85\") " Nov 01 21:20:45.899870 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:45.899818 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d6d00d0-eb9a-474a-a942-d6186f886d85-kube-api-access-72hd5\" (UniqueName: \"kubernetes.io/projected/0d6d00d0-eb9a-474a-a942-d6186f886d85-kube-api-access-72hd5\") pod \"0d6d00d0-eb9a-474a-a942-d6186f886d85\" (UID: \"0d6d00d0-eb9a-474a-a942-d6186f886d85\") " Nov 01 21:20:45.901797 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:45.901765 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d6d00d0-eb9a-474a-a942-d6186f886d85-maas-api-xks-ca" pod "0d6d00d0-eb9a-474a-a942-d6186f886d85" (UID: "0d6d00d0-eb9a-474a-a942-d6186f886d85"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:20:45.901895 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:45.901805 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d6d00d0-eb9a-474a-a942-d6186f886d85-kube-api-access-72hd5" pod "0d6d00d0-eb9a-474a-a942-d6186f886d85" (UID: "0d6d00d0-eb9a-474a-a942-d6186f886d85"). InnerVolumeSpecName "kube-api-access-72hd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:20:46.000775 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:46.000683 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d6d00d0-eb9a-474a-a942-d6186f886d85-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d6d00d0-eb9a-474a-a942-d6186f886d85-maas-api-openshift-ca\") pod \"0d6d00d0-eb9a-474a-a942-d6186f886d85\" (UID: \"0d6d00d0-eb9a-474a-a942-d6186f886d85\") " Nov 01 21:20:46.001080 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:46.001062 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72hd5\" (UniqueName: \"kubernetes.io/projected/0d6d00d0-eb9a-474a-a942-d6186f886d85-kube-api-access-72hd5\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:20:46.001174 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:46.001084 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d6d00d0-eb9a-474a-a942-d6186f886d85-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:20:46.001174 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:46.001087 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d6d00d0-eb9a-474a-a942-d6186f886d85-maas-api-openshift-ca" pod "0d6d00d0-eb9a-474a-a942-d6186f886d85" (UID: "0d6d00d0-eb9a-474a-a942-d6186f886d85"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:20:46.101987 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:46.101954 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d6d00d0-eb9a-474a-a942-d6186f886d85-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:20:46.752440 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:46.752409 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-c5lpf" Nov 01 21:20:46.752723 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:46.752448 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-c5lpf" event={"ID":"0d6d00d0-eb9a-474a-a942-d6186f886d85","Type":"ContainerDied","Data":"4cab8e0d4cdca4b3208d8a241133f6e8fce3d45fd60ff91f8f165b0d3738e416"} Nov 01 21:20:46.752723 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:46.752499 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4cab8e0d4cdca4b3208d8a241133f6e8fce3d45fd60ff91f8f165b0d3738e416" Nov 01 21:20:50.018782 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:20:50.018747 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:21:01.050437 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:01.050341 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56018f3e-dbs8g"] Nov 01 21:21:01.118865 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:01.113930 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56018f3e-dbs8g"] Nov 01 21:21:01.374143 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:01.374039 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="684eee5e-908a-4f3c-9c04-782732b39560" path="/var/lib/kubelet/pods/684eee5e-908a-4f3c-9c04-782732b39560/volumes" Nov 01 21:21:01.601370 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:01.601337 4484 scope.go:122] "RemoveContainer" containerID="5b843468db3131a27782cc0d7a399657c2be47ac4c09c0f32b2d196425c7927b" Nov 01 21:21:01.686271 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:01.683154 4484 scope.go:122] "RemoveContainer" containerID="10e406f0d707285b6072f28068f51a4104fe5fc16c121f1fce29070d297b21e8" Nov 01 21:21:01.764408 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:01.764010 4484 scope.go:122] "RemoveContainer" containerID="79b38e0622b00db51a870e62e46420f69d75b6e84d94e18dc45c03037c8c8d43" Nov 01 21:21:01.846125 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:01.841194 4484 scope.go:122] "RemoveContainer" containerID="470dc8d703080ade25b2e65b5f8ea500a1199dc1cc6f2736df1ebef44b6d63d1" Nov 01 21:21:01.910384 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:01.910005 4484 scope.go:122] "RemoveContainer" containerID="1e7fac1909090041b59086d6b03c0a3d0807a204641f4c05caa3da1d63f5fb9c" Nov 01 21:21:02.079959 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:02.079379 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee43e1ee-cq2d9"] Nov 01 21:21:02.103364 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:02.103324 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee43e1ee-cq2d9"] Nov 01 21:21:03.377771 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:03.377327 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e61432fd-9976-4b09-b225-b9750d67aa3a" path="/var/lib/kubelet/pods/e61432fd-9976-4b09-b225-b9750d67aa3a/volumes" Nov 01 21:21:10.586940 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:10.586892 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7cbqg"] Nov 01 21:21:10.587868 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:10.587842 4484 cpu_manager.go:414] "containerMap: removing container" podUID="0d6d00d0-eb9a-474a-a942-d6186f886d85" containerName="revoke-keys" Nov 01 21:21:10.587868 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:10.587870 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d6d00d0-eb9a-474a-a942-d6186f886d85" containerName="revoke-keys" Nov 01 21:21:10.588162 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:10.588147 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d6d00d0-eb9a-474a-a942-d6186f886d85" containerName="revoke-keys" Nov 01 21:21:10.595301 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:10.595273 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7cbqg" Nov 01 21:21:10.645114 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:10.645075 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7cbqg"] Nov 01 21:21:10.693240 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:10.693208 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:21:10.693240 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:10.693208 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:21:10.693240 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:10.693236 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:21:10.693542 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:10.693520 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kw5v\" (UniqueName: \"kubernetes.io/projected/b9f59ba2-6a67-404f-853e-6009607c8f0c-kube-api-access-8kw5v\") pod \"maas-api-revoke-sub-a4aef7d7-7cbqg\" (UID: \"b9f59ba2-6a67-404f-853e-6009607c8f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7cbqg" Nov 01 21:21:10.693595 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:10.693569 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9f59ba2-6a67-404f-853e-6009607c8f0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-7cbqg\" (UID: \"b9f59ba2-6a67-404f-853e-6009607c8f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7cbqg" Nov 01 21:21:10.693650 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:10.693631 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9f59ba2-6a67-404f-853e-6009607c8f0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-7cbqg\" (UID: \"b9f59ba2-6a67-404f-853e-6009607c8f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7cbqg" Nov 01 21:21:10.699253 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:10.699233 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:21:10.794372 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:10.794336 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kw5v\" (UniqueName: \"kubernetes.io/projected/b9f59ba2-6a67-404f-853e-6009607c8f0c-kube-api-access-8kw5v\") pod \"maas-api-revoke-sub-a4aef7d7-7cbqg\" (UID: \"b9f59ba2-6a67-404f-853e-6009607c8f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7cbqg" Nov 01 21:21:10.794584 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:10.794395 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9f59ba2-6a67-404f-853e-6009607c8f0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-7cbqg\" (UID: \"b9f59ba2-6a67-404f-853e-6009607c8f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7cbqg" Nov 01 21:21:10.794584 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:10.794420 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9f59ba2-6a67-404f-853e-6009607c8f0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-7cbqg\" (UID: \"b9f59ba2-6a67-404f-853e-6009607c8f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7cbqg" Nov 01 21:21:10.795183 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:10.795157 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9f59ba2-6a67-404f-853e-6009607c8f0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-7cbqg\" (UID: \"b9f59ba2-6a67-404f-853e-6009607c8f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7cbqg" Nov 01 21:21:10.796963 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:10.796937 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9f59ba2-6a67-404f-853e-6009607c8f0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-7cbqg\" (UID: \"b9f59ba2-6a67-404f-853e-6009607c8f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7cbqg" Nov 01 21:21:10.852228 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:10.852157 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kw5v\" (UniqueName: \"kubernetes.io/projected/b9f59ba2-6a67-404f-853e-6009607c8f0c-kube-api-access-8kw5v\") pod \"maas-api-revoke-sub-a4aef7d7-7cbqg\" (UID: \"b9f59ba2-6a67-404f-853e-6009607c8f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7cbqg" Nov 01 21:21:10.908420 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:10.908386 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7cbqg" Nov 01 21:21:11.279821 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:11.279791 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7cbqg"] Nov 01 21:21:11.280901 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:21:11.280875 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9f59ba2_6a67_404f_853e_6009607c8f0c.slice/crio-8c48b8fb3a66eb75cbe4e76ad60b7f60717af37843a579f4b1ba5bb364622397 WatchSource:0}: Error finding container 8c48b8fb3a66eb75cbe4e76ad60b7f60717af37843a579f4b1ba5bb364622397: Status 404 returned error can't find the container with id 8c48b8fb3a66eb75cbe4e76ad60b7f60717af37843a579f4b1ba5bb364622397 Nov 01 21:21:12.123291 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:12.123256 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7cbqg" event={"ID":"b9f59ba2-6a67-404f-853e-6009607c8f0c","Type":"ContainerStarted","Data":"98992f0381ed70d93404ab8cd60ec97bef1da25246966ec9ff749490b155af5f"} Nov 01 21:21:12.123722 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:12.123313 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7cbqg" event={"ID":"b9f59ba2-6a67-404f-853e-6009607c8f0c","Type":"ContainerStarted","Data":"8c48b8fb3a66eb75cbe4e76ad60b7f60717af37843a579f4b1ba5bb364622397"} Nov 01 21:21:12.239601 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:12.239546 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7cbqg" podStartSLOduration=2.2395306 podStartE2EDuration="2.2395306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:21:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:21:12.237327754 +0000 UTC m=+3023.014967890" watchObservedRunningTime="2026-11-01 21:21:12.2395306 +0000 UTC m=+3023.017170788" Nov 01 21:21:13.130278 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:13.130239 4484 generic.go:356] "Generic (PLEG): container finished" podID="b9f59ba2-6a67-404f-853e-6009607c8f0c" containerID="98992f0381ed70d93404ab8cd60ec97bef1da25246966ec9ff749490b155af5f" exitCode=0 Nov 01 21:21:13.130696 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:13.130287 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7cbqg" event={"ID":"b9f59ba2-6a67-404f-853e-6009607c8f0c","Type":"ContainerDied","Data":"98992f0381ed70d93404ab8cd60ec97bef1da25246966ec9ff749490b155af5f"} Nov 01 21:21:14.045516 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:14.045483 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76e99db3-4z8lg"] Nov 01 21:21:14.096970 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:14.096920 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e67bc43-bs7hs"] Nov 01 21:21:14.099078 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:14.099050 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76e99db3-4z8lg"] Nov 01 21:21:14.100913 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:14.100885 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e67bc43-bs7hs"] Nov 01 21:21:14.284338 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:14.284305 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7cbqg" Nov 01 21:21:14.327549 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:14.327443 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9f59ba2-6a67-404f-853e-6009607c8f0c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9f59ba2-6a67-404f-853e-6009607c8f0c-maas-api-openshift-ca\") pod \"b9f59ba2-6a67-404f-853e-6009607c8f0c\" (UID: \"b9f59ba2-6a67-404f-853e-6009607c8f0c\") " Nov 01 21:21:14.327549 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:14.327509 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9f59ba2-6a67-404f-853e-6009607c8f0c-kube-api-access-8kw5v\" (UniqueName: \"kubernetes.io/projected/b9f59ba2-6a67-404f-853e-6009607c8f0c-kube-api-access-8kw5v\") pod \"b9f59ba2-6a67-404f-853e-6009607c8f0c\" (UID: \"b9f59ba2-6a67-404f-853e-6009607c8f0c\") " Nov 01 21:21:14.327795 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:14.327658 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9f59ba2-6a67-404f-853e-6009607c8f0c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9f59ba2-6a67-404f-853e-6009607c8f0c-maas-api-xks-ca\") pod \"b9f59ba2-6a67-404f-853e-6009607c8f0c\" (UID: \"b9f59ba2-6a67-404f-853e-6009607c8f0c\") " Nov 01 21:21:14.327915 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:14.327887 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9f59ba2-6a67-404f-853e-6009607c8f0c-maas-api-openshift-ca" pod "b9f59ba2-6a67-404f-853e-6009607c8f0c" (UID: "b9f59ba2-6a67-404f-853e-6009607c8f0c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:21:14.328023 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:14.327979 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9f59ba2-6a67-404f-853e-6009607c8f0c-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:21:14.329752 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:14.329728 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9f59ba2-6a67-404f-853e-6009607c8f0c-kube-api-access-8kw5v" pod "b9f59ba2-6a67-404f-853e-6009607c8f0c" (UID: "b9f59ba2-6a67-404f-853e-6009607c8f0c"). InnerVolumeSpecName "kube-api-access-8kw5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:21:14.329826 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:14.329778 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9f59ba2-6a67-404f-853e-6009607c8f0c-maas-api-xks-ca" pod "b9f59ba2-6a67-404f-853e-6009607c8f0c" (UID: "b9f59ba2-6a67-404f-853e-6009607c8f0c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:21:14.429565 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:14.429526 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kw5v\" (UniqueName: \"kubernetes.io/projected/b9f59ba2-6a67-404f-853e-6009607c8f0c-kube-api-access-8kw5v\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:21:14.429565 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:14.429558 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9f59ba2-6a67-404f-853e-6009607c8f0c-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:21:15.142230 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:15.142194 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7cbqg" Nov 01 21:21:15.142230 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:15.142218 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7cbqg" event={"ID":"b9f59ba2-6a67-404f-853e-6009607c8f0c","Type":"ContainerDied","Data":"8c48b8fb3a66eb75cbe4e76ad60b7f60717af37843a579f4b1ba5bb364622397"} Nov 01 21:21:15.142453 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:15.142268 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c48b8fb3a66eb75cbe4e76ad60b7f60717af37843a579f4b1ba5bb364622397" Nov 01 21:21:15.370848 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:15.370814 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39899bb0-5323-441a-8267-e70ae6fe4e5e" path="/var/lib/kubelet/pods/39899bb0-5323-441a-8267-e70ae6fe4e5e/volumes" Nov 01 21:21:15.371296 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:15.371281 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64efda00-ea04-476c-b80e-9deba0bc9827" path="/var/lib/kubelet/pods/64efda00-ea04-476c-b80e-9deba0bc9827/volumes" Nov 01 21:21:15.766080 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:21:15.766043 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:21:38.046643 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:38.046599 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6911ec62-s7w9h"] Nov 01 21:21:38.097521 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:38.097487 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6911ec62-s7w9h"] Nov 01 21:21:39.373675 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:39.373638 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8fbc4e0-a109-4059-87d6-d785b84eabe0" path="/var/lib/kubelet/pods/f8fbc4e0-a109-4059-87d6-d785b84eabe0/volumes" Nov 01 21:21:41.171166 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:41.171132 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bfs4p"] Nov 01 21:21:41.171560 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:41.171545 4484 cpu_manager.go:414] "containerMap: removing container" podUID="b9f59ba2-6a67-404f-853e-6009607c8f0c" containerName="revoke-keys" Nov 01 21:21:41.171560 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:41.171555 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9f59ba2-6a67-404f-853e-6009607c8f0c" containerName="revoke-keys" Nov 01 21:21:41.172064 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:41.172028 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9f59ba2-6a67-404f-853e-6009607c8f0c" containerName="revoke-keys" Nov 01 21:21:41.178162 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:41.176638 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bfs4p" Nov 01 21:21:41.231040 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:41.230989 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bfs4p"] Nov 01 21:21:41.232908 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:41.232881 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a646bf7-f90c-4ea9-8056-9968f5f4b957-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-bfs4p\" (UID: \"1a646bf7-f90c-4ea9-8056-9968f5f4b957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bfs4p" Nov 01 21:21:41.233076 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:41.233044 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97sg2\" (UniqueName: \"kubernetes.io/projected/1a646bf7-f90c-4ea9-8056-9968f5f4b957-kube-api-access-97sg2\") pod \"maas-api-revoke-sub-79805ca9-bfs4p\" (UID: \"1a646bf7-f90c-4ea9-8056-9968f5f4b957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bfs4p" Nov 01 21:21:41.233130 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:41.233079 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a646bf7-f90c-4ea9-8056-9968f5f4b957-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-bfs4p\" (UID: \"1a646bf7-f90c-4ea9-8056-9968f5f4b957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bfs4p" Nov 01 21:21:41.278686 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:41.278649 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:21:41.278686 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:41.278661 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:21:41.278686 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:41.278661 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:21:41.282273 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:41.282255 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:21:41.334180 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:41.334152 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a646bf7-f90c-4ea9-8056-9968f5f4b957-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-bfs4p\" (UID: \"1a646bf7-f90c-4ea9-8056-9968f5f4b957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bfs4p" Nov 01 21:21:41.334336 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:41.334219 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a646bf7-f90c-4ea9-8056-9968f5f4b957-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-bfs4p\" (UID: \"1a646bf7-f90c-4ea9-8056-9968f5f4b957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bfs4p" Nov 01 21:21:41.334388 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:41.334345 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97sg2\" (UniqueName: \"kubernetes.io/projected/1a646bf7-f90c-4ea9-8056-9968f5f4b957-kube-api-access-97sg2\") pod \"maas-api-revoke-sub-79805ca9-bfs4p\" (UID: \"1a646bf7-f90c-4ea9-8056-9968f5f4b957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bfs4p" Nov 01 21:21:41.334924 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:41.334904 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a646bf7-f90c-4ea9-8056-9968f5f4b957-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-bfs4p\" (UID: \"1a646bf7-f90c-4ea9-8056-9968f5f4b957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bfs4p" Nov 01 21:21:41.336622 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:41.336598 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a646bf7-f90c-4ea9-8056-9968f5f4b957-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-bfs4p\" (UID: \"1a646bf7-f90c-4ea9-8056-9968f5f4b957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bfs4p" Nov 01 21:21:41.391813 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:41.391775 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97sg2\" (UniqueName: \"kubernetes.io/projected/1a646bf7-f90c-4ea9-8056-9968f5f4b957-kube-api-access-97sg2\") pod \"maas-api-revoke-sub-79805ca9-bfs4p\" (UID: \"1a646bf7-f90c-4ea9-8056-9968f5f4b957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bfs4p" Nov 01 21:21:41.492226 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:41.492191 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bfs4p" Nov 01 21:21:41.863508 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:41.863484 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bfs4p"] Nov 01 21:21:41.864616 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:21:41.864583 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a646bf7_f90c_4ea9_8056_9968f5f4b957.slice/crio-c453841bb90c8406b054105f075a57139f3db6918ac00acde17e4ef47c5d5692 WatchSource:0}: Error finding container c453841bb90c8406b054105f075a57139f3db6918ac00acde17e4ef47c5d5692: Status 404 returned error can't find the container with id c453841bb90c8406b054105f075a57139f3db6918ac00acde17e4ef47c5d5692 Nov 01 21:21:42.319497 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:42.319456 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bfs4p" event={"ID":"1a646bf7-f90c-4ea9-8056-9968f5f4b957","Type":"ContainerStarted","Data":"0000aa5a0fdf30882cadbe444029c3cdb56e62ec95f5902f06661a6078c60446"} Nov 01 21:21:42.320028 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:42.319526 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bfs4p" event={"ID":"1a646bf7-f90c-4ea9-8056-9968f5f4b957","Type":"ContainerStarted","Data":"c453841bb90c8406b054105f075a57139f3db6918ac00acde17e4ef47c5d5692"} Nov 01 21:21:42.436112 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:42.436051 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bfs4p" podStartSLOduration=1.436036567 podStartE2EDuration="1.436036567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:21:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:21:42.433552582 +0000 UTC m=+3053.211192740" watchObservedRunningTime="2026-11-01 21:21:42.436036567 +0000 UTC m=+3053.213676702" Nov 01 21:21:43.045696 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:43.045656 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65195188-2f5s5"] Nov 01 21:21:43.096613 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:43.096579 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65195188-2f5s5"] Nov 01 21:21:43.327210 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:43.327118 4484 generic.go:356] "Generic (PLEG): container finished" podID="1a646bf7-f90c-4ea9-8056-9968f5f4b957" containerID="0000aa5a0fdf30882cadbe444029c3cdb56e62ec95f5902f06661a6078c60446" exitCode=0 Nov 01 21:21:43.327210 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:43.327177 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bfs4p" event={"ID":"1a646bf7-f90c-4ea9-8056-9968f5f4b957","Type":"ContainerDied","Data":"0000aa5a0fdf30882cadbe444029c3cdb56e62ec95f5902f06661a6078c60446"} Nov 01 21:21:43.370239 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:43.370207 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0f691d8-cc30-4367-901f-cc880d12d118" path="/var/lib/kubelet/pods/b0f691d8-cc30-4367-901f-cc880d12d118/volumes" Nov 01 21:21:44.471783 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:44.471757 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bfs4p" Nov 01 21:21:44.570159 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:44.570128 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a646bf7-f90c-4ea9-8056-9968f5f4b957-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a646bf7-f90c-4ea9-8056-9968f5f4b957-maas-api-xks-ca\") pod \"1a646bf7-f90c-4ea9-8056-9968f5f4b957\" (UID: \"1a646bf7-f90c-4ea9-8056-9968f5f4b957\") " Nov 01 21:21:44.570347 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:44.570182 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a646bf7-f90c-4ea9-8056-9968f5f4b957-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a646bf7-f90c-4ea9-8056-9968f5f4b957-maas-api-openshift-ca\") pod \"1a646bf7-f90c-4ea9-8056-9968f5f4b957\" (UID: \"1a646bf7-f90c-4ea9-8056-9968f5f4b957\") " Nov 01 21:21:44.570347 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:44.570259 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a646bf7-f90c-4ea9-8056-9968f5f4b957-kube-api-access-97sg2\" (UniqueName: \"kubernetes.io/projected/1a646bf7-f90c-4ea9-8056-9968f5f4b957-kube-api-access-97sg2\") pod \"1a646bf7-f90c-4ea9-8056-9968f5f4b957\" (UID: \"1a646bf7-f90c-4ea9-8056-9968f5f4b957\") " Nov 01 21:21:44.570643 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:44.570612 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a646bf7-f90c-4ea9-8056-9968f5f4b957-maas-api-openshift-ca" pod "1a646bf7-f90c-4ea9-8056-9968f5f4b957" (UID: "1a646bf7-f90c-4ea9-8056-9968f5f4b957"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:21:44.572370 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:44.572344 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a646bf7-f90c-4ea9-8056-9968f5f4b957-maas-api-xks-ca" pod "1a646bf7-f90c-4ea9-8056-9968f5f4b957" (UID: "1a646bf7-f90c-4ea9-8056-9968f5f4b957"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:21:44.572509 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:44.572491 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a646bf7-f90c-4ea9-8056-9968f5f4b957-kube-api-access-97sg2" pod "1a646bf7-f90c-4ea9-8056-9968f5f4b957" (UID: "1a646bf7-f90c-4ea9-8056-9968f5f4b957"). InnerVolumeSpecName "kube-api-access-97sg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:21:44.671503 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:44.671422 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a646bf7-f90c-4ea9-8056-9968f5f4b957-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:21:44.671503 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:44.671450 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a646bf7-f90c-4ea9-8056-9968f5f4b957-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:21:44.671503 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:44.671459 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97sg2\" (UniqueName: \"kubernetes.io/projected/1a646bf7-f90c-4ea9-8056-9968f5f4b957-kube-api-access-97sg2\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:21:45.339411 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:45.339379 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bfs4p" Nov 01 21:21:45.339411 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:45.339389 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bfs4p" event={"ID":"1a646bf7-f90c-4ea9-8056-9968f5f4b957","Type":"ContainerDied","Data":"c453841bb90c8406b054105f075a57139f3db6918ac00acde17e4ef47c5d5692"} Nov 01 21:21:45.339628 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:45.339443 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c453841bb90c8406b054105f075a57139f3db6918ac00acde17e4ef47c5d5692" Nov 01 21:21:45.766044 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:21:45.765977 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:21:50.412383 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:50.412331 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:21:58.926021 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:58.914085 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-hdvw9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 21:21:58.926021 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:58.914167 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-hdvw9" podUID="ec0b1a26-5b6c-405c-a042-7141708b6b22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 21:21:58.935155 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:58.935092 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-2n5h6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 21:21:58.935498 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:58.935469 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2n5h6" podUID="35afbdf8-b193-42b5-9c8f-85471b4288b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 21:21:58.956031 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:58.954107 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-6zwld container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 21:21:58.956031 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:21:58.954166 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6zwld" podUID="8e6a4f7c-75e1-4506-afb5-4ab00107a7f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 21:22:02.148915 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:02.148880 4484 scope.go:122] "RemoveContainer" containerID="7d361c6ae6b1328446fbe82824322f7de911699b2f3f40a1b69d7bc5088813fe" Nov 01 21:22:02.158555 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:02.158519 4484 scope.go:122] "RemoveContainer" containerID="e57aabf609e00da2996a6d6a9cd3cd110e72dd3626958c81b2d0a395e0030d36" Nov 01 21:22:02.168239 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:02.168184 4484 scope.go:122] "RemoveContainer" containerID="8344400d481c0a87bc25f773a889fbe380d00e8ee9d2c3c84d8a8a3cba54c1b7" Nov 01 21:22:02.178568 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:02.178548 4484 scope.go:122] "RemoveContainer" containerID="2c07bac71e1347752f1cc099eb3c05ebd66434cc23a0417037cbeb6bd854760c" Nov 01 21:22:02.190166 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:02.190142 4484 scope.go:122] "RemoveContainer" containerID="31aa46d356be0b2610bf24d726b7f2d9214181a100f4c127131be981cad380cc" Nov 01 21:22:02.204146 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:02.204111 4484 scope.go:122] "RemoveContainer" containerID="122624c96d30804d0282e0b7ddf0f9666781ce9580db37fc59f0f35c367a5632" Nov 01 21:22:04.873670 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:04.873633 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zkx4k"] Nov 01 21:22:04.874239 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:04.874070 4484 cpu_manager.go:414] "containerMap: removing container" podUID="1a646bf7-f90c-4ea9-8056-9968f5f4b957" containerName="revoke-keys" Nov 01 21:22:04.874239 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:04.874082 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a646bf7-f90c-4ea9-8056-9968f5f4b957" containerName="revoke-keys" Nov 01 21:22:04.874239 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:04.874154 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a646bf7-f90c-4ea9-8056-9968f5f4b957" containerName="revoke-keys" Nov 01 21:22:04.878978 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:04.878947 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zkx4k" Nov 01 21:22:04.923731 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:04.923685 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8cbbf27-4eda-4002-921e-e1d090b7d983-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-zkx4k\" (UID: \"b8cbbf27-4eda-4002-921e-e1d090b7d983\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zkx4k" Nov 01 21:22:04.923954 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:04.923759 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg585\" (UniqueName: \"kubernetes.io/projected/b8cbbf27-4eda-4002-921e-e1d090b7d983-kube-api-access-qg585\") pod \"maas-api-revoke-sub-0b7e8edb-zkx4k\" (UID: \"b8cbbf27-4eda-4002-921e-e1d090b7d983\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zkx4k" Nov 01 21:22:04.923954 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:04.923852 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8cbbf27-4eda-4002-921e-e1d090b7d983-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-zkx4k\" (UID: \"b8cbbf27-4eda-4002-921e-e1d090b7d983\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zkx4k" Nov 01 21:22:04.934801 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:04.934768 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zkx4k"] Nov 01 21:22:04.981892 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:04.981853 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:22:04.981892 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:04.981866 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:22:04.983413 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:04.983394 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:22:04.983466 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:04.983406 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:22:05.024684 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:05.024649 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8cbbf27-4eda-4002-921e-e1d090b7d983-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-zkx4k\" (UID: \"b8cbbf27-4eda-4002-921e-e1d090b7d983\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zkx4k" Nov 01 21:22:05.024871 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:05.024711 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg585\" (UniqueName: \"kubernetes.io/projected/b8cbbf27-4eda-4002-921e-e1d090b7d983-kube-api-access-qg585\") pod \"maas-api-revoke-sub-0b7e8edb-zkx4k\" (UID: \"b8cbbf27-4eda-4002-921e-e1d090b7d983\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zkx4k" Nov 01 21:22:05.024871 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:05.024764 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8cbbf27-4eda-4002-921e-e1d090b7d983-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-zkx4k\" (UID: \"b8cbbf27-4eda-4002-921e-e1d090b7d983\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zkx4k" Nov 01 21:22:05.025496 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:05.025468 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8cbbf27-4eda-4002-921e-e1d090b7d983-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-zkx4k\" (UID: \"b8cbbf27-4eda-4002-921e-e1d090b7d983\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zkx4k" Nov 01 21:22:05.027060 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:05.027037 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8cbbf27-4eda-4002-921e-e1d090b7d983-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-zkx4k\" (UID: \"b8cbbf27-4eda-4002-921e-e1d090b7d983\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zkx4k" Nov 01 21:22:05.081965 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:05.081931 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg585\" (UniqueName: \"kubernetes.io/projected/b8cbbf27-4eda-4002-921e-e1d090b7d983-kube-api-access-qg585\") pod \"maas-api-revoke-sub-0b7e8edb-zkx4k\" (UID: \"b8cbbf27-4eda-4002-921e-e1d090b7d983\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zkx4k" Nov 01 21:22:05.192162 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:05.192078 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zkx4k" Nov 01 21:22:05.567299 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:05.567260 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zkx4k"] Nov 01 21:22:05.567553 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:22:05.567451 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8cbbf27_4eda_4002_921e_e1d090b7d983.slice/crio-b6c24f0e3de649219389445c90edccd36f743b4b7d187c9ca83a2731b641199e WatchSource:0}: Error finding container b6c24f0e3de649219389445c90edccd36f743b4b7d187c9ca83a2731b641199e: Status 404 returned error can't find the container with id b6c24f0e3de649219389445c90edccd36f743b4b7d187c9ca83a2731b641199e Nov 01 21:22:06.507904 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:06.507875 4484 generic.go:356] "Generic (PLEG): container finished" podID="b8cbbf27-4eda-4002-921e-e1d090b7d983" containerID="2a23753baaaa56a57641399efbee8788319ceac974ad596f11e53aebc43d80be" exitCode=0 Nov 01 21:22:06.508297 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:06.507954 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zkx4k" event={"ID":"b8cbbf27-4eda-4002-921e-e1d090b7d983","Type":"ContainerDied","Data":"2a23753baaaa56a57641399efbee8788319ceac974ad596f11e53aebc43d80be"} Nov 01 21:22:06.508297 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:06.508095 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zkx4k" event={"ID":"b8cbbf27-4eda-4002-921e-e1d090b7d983","Type":"ContainerStarted","Data":"b6c24f0e3de649219389445c90edccd36f743b4b7d187c9ca83a2731b641199e"} Nov 01 21:22:07.657786 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:07.657760 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zkx4k" Nov 01 21:22:07.751257 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:07.751215 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8cbbf27-4eda-4002-921e-e1d090b7d983-kube-api-access-qg585\" (UniqueName: \"kubernetes.io/projected/b8cbbf27-4eda-4002-921e-e1d090b7d983-kube-api-access-qg585\") pod \"b8cbbf27-4eda-4002-921e-e1d090b7d983\" (UID: \"b8cbbf27-4eda-4002-921e-e1d090b7d983\") " Nov 01 21:22:07.751468 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:07.751317 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8cbbf27-4eda-4002-921e-e1d090b7d983-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8cbbf27-4eda-4002-921e-e1d090b7d983-maas-api-openshift-ca\") pod \"b8cbbf27-4eda-4002-921e-e1d090b7d983\" (UID: \"b8cbbf27-4eda-4002-921e-e1d090b7d983\") " Nov 01 21:22:07.751468 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:07.751358 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8cbbf27-4eda-4002-921e-e1d090b7d983-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8cbbf27-4eda-4002-921e-e1d090b7d983-maas-api-xks-ca\") pod \"b8cbbf27-4eda-4002-921e-e1d090b7d983\" (UID: \"b8cbbf27-4eda-4002-921e-e1d090b7d983\") " Nov 01 21:22:07.751743 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:07.751716 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8cbbf27-4eda-4002-921e-e1d090b7d983-maas-api-openshift-ca" pod "b8cbbf27-4eda-4002-921e-e1d090b7d983" (UID: "b8cbbf27-4eda-4002-921e-e1d090b7d983"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:22:07.753388 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:07.753361 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8cbbf27-4eda-4002-921e-e1d090b7d983-maas-api-xks-ca" pod "b8cbbf27-4eda-4002-921e-e1d090b7d983" (UID: "b8cbbf27-4eda-4002-921e-e1d090b7d983"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:22:07.753503 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:07.753457 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8cbbf27-4eda-4002-921e-e1d090b7d983-kube-api-access-qg585" pod "b8cbbf27-4eda-4002-921e-e1d090b7d983" (UID: "b8cbbf27-4eda-4002-921e-e1d090b7d983"). InnerVolumeSpecName "kube-api-access-qg585". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:22:07.852472 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:07.852366 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qg585\" (UniqueName: \"kubernetes.io/projected/b8cbbf27-4eda-4002-921e-e1d090b7d983-kube-api-access-qg585\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:22:07.852472 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:07.852411 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8cbbf27-4eda-4002-921e-e1d090b7d983-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:22:07.852472 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:07.852422 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8cbbf27-4eda-4002-921e-e1d090b7d983-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:22:08.519703 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:08.519669 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zkx4k" Nov 01 21:22:08.519703 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:08.519687 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zkx4k" event={"ID":"b8cbbf27-4eda-4002-921e-e1d090b7d983","Type":"ContainerDied","Data":"b6c24f0e3de649219389445c90edccd36f743b4b7d187c9ca83a2731b641199e"} Nov 01 21:22:08.519955 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:08.519739 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6c24f0e3de649219389445c90edccd36f743b4b7d187c9ca83a2731b641199e" Nov 01 21:22:10.451793 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:10.451723 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:22:15.766072 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:22:15.766033 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:22:16.228202 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:16.228123 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4zldl"] Nov 01 21:22:16.228567 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:16.228554 4484 cpu_manager.go:414] "containerMap: removing container" podUID="b8cbbf27-4eda-4002-921e-e1d090b7d983" containerName="revoke-keys" Nov 01 21:22:16.228631 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:16.228569 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8cbbf27-4eda-4002-921e-e1d090b7d983" containerName="revoke-keys" Nov 01 21:22:16.228673 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:16.228659 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8cbbf27-4eda-4002-921e-e1d090b7d983" containerName="revoke-keys" Nov 01 21:22:16.231858 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:16.231835 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4zldl" Nov 01 21:22:16.288549 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:16.288498 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4zldl"] Nov 01 21:22:16.336108 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:16.336066 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:22:16.336108 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:16.336078 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:22:16.336108 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:16.336103 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:22:16.336380 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:16.336176 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:22:16.336429 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:16.336379 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc859\" (UniqueName: \"kubernetes.io/projected/a6b68760-d700-43fa-b8a6-f38b88acb5d1-kube-api-access-wc859\") pod \"maas-api-revoke-sub-3d4156f1-4zldl\" (UID: \"a6b68760-d700-43fa-b8a6-f38b88acb5d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4zldl" Nov 01 21:22:16.336508 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:16.336490 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6b68760-d700-43fa-b8a6-f38b88acb5d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4zldl\" (UID: \"a6b68760-d700-43fa-b8a6-f38b88acb5d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4zldl" Nov 01 21:22:16.336578 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:16.336565 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6b68760-d700-43fa-b8a6-f38b88acb5d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4zldl\" (UID: \"a6b68760-d700-43fa-b8a6-f38b88acb5d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4zldl" Nov 01 21:22:16.437683 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:16.437648 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6b68760-d700-43fa-b8a6-f38b88acb5d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4zldl\" (UID: \"a6b68760-d700-43fa-b8a6-f38b88acb5d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4zldl" Nov 01 21:22:16.437926 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:16.437743 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6b68760-d700-43fa-b8a6-f38b88acb5d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4zldl\" (UID: \"a6b68760-d700-43fa-b8a6-f38b88acb5d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4zldl" Nov 01 21:22:16.437926 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:16.437800 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc859\" (UniqueName: \"kubernetes.io/projected/a6b68760-d700-43fa-b8a6-f38b88acb5d1-kube-api-access-wc859\") pod \"maas-api-revoke-sub-3d4156f1-4zldl\" (UID: \"a6b68760-d700-43fa-b8a6-f38b88acb5d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4zldl" Nov 01 21:22:16.438447 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:16.438428 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6b68760-d700-43fa-b8a6-f38b88acb5d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4zldl\" (UID: \"a6b68760-d700-43fa-b8a6-f38b88acb5d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4zldl" Nov 01 21:22:16.440032 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:16.439982 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6b68760-d700-43fa-b8a6-f38b88acb5d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4zldl\" (UID: \"a6b68760-d700-43fa-b8a6-f38b88acb5d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4zldl" Nov 01 21:22:16.497187 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:16.497153 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc859\" (UniqueName: \"kubernetes.io/projected/a6b68760-d700-43fa-b8a6-f38b88acb5d1-kube-api-access-wc859\") pod \"maas-api-revoke-sub-3d4156f1-4zldl\" (UID: \"a6b68760-d700-43fa-b8a6-f38b88acb5d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4zldl" Nov 01 21:22:16.545138 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:16.545099 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4zldl" Nov 01 21:22:16.917973 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:16.917945 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4zldl"] Nov 01 21:22:16.919815 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:22:16.919788 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6b68760_d700_43fa_b8a6_f38b88acb5d1.slice/crio-558e51b469c96fc1a82f6df471a2b2836019e18cfb0a0caa6c2ef9a09cbe489a WatchSource:0}: Error finding container 558e51b469c96fc1a82f6df471a2b2836019e18cfb0a0caa6c2ef9a09cbe489a: Status 404 returned error can't find the container with id 558e51b469c96fc1a82f6df471a2b2836019e18cfb0a0caa6c2ef9a09cbe489a Nov 01 21:22:17.565414 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:17.565376 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4zldl" event={"ID":"a6b68760-d700-43fa-b8a6-f38b88acb5d1","Type":"ContainerStarted","Data":"8b6bafdebdf490aa0ab63270eb60ffa73944f852f4e0800f846eb0bb4cb218e3"} Nov 01 21:22:17.565607 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:17.565432 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4zldl" event={"ID":"a6b68760-d700-43fa-b8a6-f38b88acb5d1","Type":"ContainerStarted","Data":"558e51b469c96fc1a82f6df471a2b2836019e18cfb0a0caa6c2ef9a09cbe489a"} Nov 01 21:22:17.681885 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:17.681831 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4zldl" podStartSLOduration=1.6818173779999999 podStartE2EDuration="1.681817378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:22:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:22:17.679189256 +0000 UTC m=+3088.456829388" watchObservedRunningTime="2026-11-01 21:22:17.681817378 +0000 UTC m=+3088.459457506" Nov 01 21:22:18.587020 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:18.586767 4484 generic.go:356] "Generic (PLEG): container finished" podID="a6b68760-d700-43fa-b8a6-f38b88acb5d1" containerID="8b6bafdebdf490aa0ab63270eb60ffa73944f852f4e0800f846eb0bb4cb218e3" exitCode=0 Nov 01 21:22:18.587020 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:18.586925 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4zldl" event={"ID":"a6b68760-d700-43fa-b8a6-f38b88acb5d1","Type":"ContainerDied","Data":"8b6bafdebdf490aa0ab63270eb60ffa73944f852f4e0800f846eb0bb4cb218e3"} Nov 01 21:22:20.130020 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:20.128456 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4zldl" Nov 01 21:22:20.214014 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:20.209637 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6b68760-d700-43fa-b8a6-f38b88acb5d1-kube-api-access-wc859\" (UniqueName: \"kubernetes.io/projected/a6b68760-d700-43fa-b8a6-f38b88acb5d1-kube-api-access-wc859\") pod \"a6b68760-d700-43fa-b8a6-f38b88acb5d1\" (UID: \"a6b68760-d700-43fa-b8a6-f38b88acb5d1\") " Nov 01 21:22:20.214014 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:20.209713 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6b68760-d700-43fa-b8a6-f38b88acb5d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6b68760-d700-43fa-b8a6-f38b88acb5d1-maas-api-openshift-ca\") pod \"a6b68760-d700-43fa-b8a6-f38b88acb5d1\" (UID: \"a6b68760-d700-43fa-b8a6-f38b88acb5d1\") " Nov 01 21:22:20.214014 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:20.209770 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6b68760-d700-43fa-b8a6-f38b88acb5d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6b68760-d700-43fa-b8a6-f38b88acb5d1-maas-api-xks-ca\") pod \"a6b68760-d700-43fa-b8a6-f38b88acb5d1\" (UID: \"a6b68760-d700-43fa-b8a6-f38b88acb5d1\") " Nov 01 21:22:20.214014 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:20.210985 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6b68760-d700-43fa-b8a6-f38b88acb5d1-maas-api-openshift-ca" pod "a6b68760-d700-43fa-b8a6-f38b88acb5d1" (UID: "a6b68760-d700-43fa-b8a6-f38b88acb5d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:22:20.228253 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:20.228208 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6b68760-d700-43fa-b8a6-f38b88acb5d1-maas-api-xks-ca" pod "a6b68760-d700-43fa-b8a6-f38b88acb5d1" (UID: "a6b68760-d700-43fa-b8a6-f38b88acb5d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:22:20.228454 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:20.228242 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6b68760-d700-43fa-b8a6-f38b88acb5d1-kube-api-access-wc859" pod "a6b68760-d700-43fa-b8a6-f38b88acb5d1" (UID: "a6b68760-d700-43fa-b8a6-f38b88acb5d1"). InnerVolumeSpecName "kube-api-access-wc859". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:22:20.311363 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:20.311320 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6b68760-d700-43fa-b8a6-f38b88acb5d1-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:22:20.311363 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:20.311366 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6b68760-d700-43fa-b8a6-f38b88acb5d1-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:22:20.311575 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:20.311383 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc859\" (UniqueName: \"kubernetes.io/projected/a6b68760-d700-43fa-b8a6-f38b88acb5d1-kube-api-access-wc859\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:22:20.602189 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:20.602158 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4zldl" Nov 01 21:22:20.602189 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:20.602177 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4zldl" event={"ID":"a6b68760-d700-43fa-b8a6-f38b88acb5d1","Type":"ContainerDied","Data":"558e51b469c96fc1a82f6df471a2b2836019e18cfb0a0caa6c2ef9a09cbe489a"} Nov 01 21:22:20.602417 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:20.602218 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="558e51b469c96fc1a82f6df471a2b2836019e18cfb0a0caa6c2ef9a09cbe489a" Nov 01 21:22:31.045451 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:31.045368 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6bjrx"] Nov 01 21:22:31.047576 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:31.047549 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6bjrx"] Nov 01 21:22:31.374238 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:31.374142 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fff41a8-bef5-464b-89b6-2bc7247d528c" path="/var/lib/kubelet/pods/4fff41a8-bef5-464b-89b6-2bc7247d528c/volumes" Nov 01 21:22:45.765677 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:22:45.765638 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:22:49.759968 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:49.759865 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4zldl"] Nov 01 21:22:49.811075 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:49.811040 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4zldl"] Nov 01 21:22:49.881591 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:49.881557 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m78x6"] Nov 01 21:22:49.882057 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:49.882044 4484 cpu_manager.go:414] "containerMap: removing container" podUID="a6b68760-d700-43fa-b8a6-f38b88acb5d1" containerName="revoke-keys" Nov 01 21:22:49.882112 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:49.882059 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6b68760-d700-43fa-b8a6-f38b88acb5d1" containerName="revoke-keys" Nov 01 21:22:49.882150 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:49.882137 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6b68760-d700-43fa-b8a6-f38b88acb5d1" containerName="revoke-keys" Nov 01 21:22:49.885488 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:49.885472 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m78x6" Nov 01 21:22:49.939385 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:49.939332 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:22:49.939636 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:49.939505 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:22:49.939748 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:49.939664 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:22:49.941434 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:49.941412 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m78x6"] Nov 01 21:22:49.989492 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:49.989468 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:22:50.020903 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:50.020826 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c6fa59d-a1e5-4863-89b7-df655ab1da3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-m78x6\" (UID: \"1c6fa59d-a1e5-4863-89b7-df655ab1da3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m78x6" Nov 01 21:22:50.020903 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:50.020860 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c6fa59d-a1e5-4863-89b7-df655ab1da3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-m78x6\" (UID: \"1c6fa59d-a1e5-4863-89b7-df655ab1da3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m78x6" Nov 01 21:22:50.021170 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:50.021110 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k525\" (UniqueName: \"kubernetes.io/projected/1c6fa59d-a1e5-4863-89b7-df655ab1da3a-kube-api-access-7k525\") pod \"maas-api-revoke-sub-3d4156f1-m78x6\" (UID: \"1c6fa59d-a1e5-4863-89b7-df655ab1da3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m78x6" Nov 01 21:22:50.122317 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:50.122273 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k525\" (UniqueName: \"kubernetes.io/projected/1c6fa59d-a1e5-4863-89b7-df655ab1da3a-kube-api-access-7k525\") pod \"maas-api-revoke-sub-3d4156f1-m78x6\" (UID: \"1c6fa59d-a1e5-4863-89b7-df655ab1da3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m78x6" Nov 01 21:22:50.122511 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:50.122373 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c6fa59d-a1e5-4863-89b7-df655ab1da3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-m78x6\" (UID: \"1c6fa59d-a1e5-4863-89b7-df655ab1da3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m78x6" Nov 01 21:22:50.122511 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:50.122400 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c6fa59d-a1e5-4863-89b7-df655ab1da3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-m78x6\" (UID: \"1c6fa59d-a1e5-4863-89b7-df655ab1da3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m78x6" Nov 01 21:22:50.123091 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:50.123068 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c6fa59d-a1e5-4863-89b7-df655ab1da3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-m78x6\" (UID: \"1c6fa59d-a1e5-4863-89b7-df655ab1da3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m78x6" Nov 01 21:22:50.124838 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:50.124814 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c6fa59d-a1e5-4863-89b7-df655ab1da3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-m78x6\" (UID: \"1c6fa59d-a1e5-4863-89b7-df655ab1da3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m78x6" Nov 01 21:22:50.179889 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:50.179851 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k525\" (UniqueName: \"kubernetes.io/projected/1c6fa59d-a1e5-4863-89b7-df655ab1da3a-kube-api-access-7k525\") pod \"maas-api-revoke-sub-3d4156f1-m78x6\" (UID: \"1c6fa59d-a1e5-4863-89b7-df655ab1da3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m78x6" Nov 01 21:22:50.198512 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:50.198479 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m78x6" Nov 01 21:22:50.570037 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:50.570008 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m78x6"] Nov 01 21:22:50.571150 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:22:50.571122 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c6fa59d_a1e5_4863_89b7_df655ab1da3a.slice/crio-7f68c991e4c975e87b914b024ceae977f173d1c75e861ba4e8f00da553e20a5d WatchSource:0}: Error finding container 7f68c991e4c975e87b914b024ceae977f173d1c75e861ba4e8f00da553e20a5d: Status 404 returned error can't find the container with id 7f68c991e4c975e87b914b024ceae977f173d1c75e861ba4e8f00da553e20a5d Nov 01 21:22:50.704648 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:50.704587 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:22:50.822122 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:50.822084 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m78x6" event={"ID":"1c6fa59d-a1e5-4863-89b7-df655ab1da3a","Type":"ContainerStarted","Data":"7f68c991e4c975e87b914b024ceae977f173d1c75e861ba4e8f00da553e20a5d"} Nov 01 21:22:51.090579 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:51.090475 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:22:51.370828 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:51.370750 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6b68760-d700-43fa-b8a6-f38b88acb5d1" path="/var/lib/kubelet/pods/a6b68760-d700-43fa-b8a6-f38b88acb5d1/volumes" Nov 01 21:22:51.828990 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:51.828955 4484 generic.go:356] "Generic (PLEG): container finished" podID="1c6fa59d-a1e5-4863-89b7-df655ab1da3a" containerID="b5436ac4e60510cb4a593ddef5a850bd5f8cb85d667f8f96c1d57c805f1c4b8a" exitCode=0 Nov 01 21:22:51.829461 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:51.829029 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m78x6" event={"ID":"1c6fa59d-a1e5-4863-89b7-df655ab1da3a","Type":"ContainerDied","Data":"b5436ac4e60510cb4a593ddef5a850bd5f8cb85d667f8f96c1d57c805f1c4b8a"} Nov 01 21:22:52.971819 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:52.971796 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m78x6" Nov 01 21:22:53.151800 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:53.151715 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c6fa59d-a1e5-4863-89b7-df655ab1da3a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c6fa59d-a1e5-4863-89b7-df655ab1da3a-maas-api-openshift-ca\") pod \"1c6fa59d-a1e5-4863-89b7-df655ab1da3a\" (UID: \"1c6fa59d-a1e5-4863-89b7-df655ab1da3a\") " Nov 01 21:22:53.151800 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:53.151759 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c6fa59d-a1e5-4863-89b7-df655ab1da3a-kube-api-access-7k525\" (UniqueName: \"kubernetes.io/projected/1c6fa59d-a1e5-4863-89b7-df655ab1da3a-kube-api-access-7k525\") pod \"1c6fa59d-a1e5-4863-89b7-df655ab1da3a\" (UID: \"1c6fa59d-a1e5-4863-89b7-df655ab1da3a\") " Nov 01 21:22:53.152105 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:53.151858 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c6fa59d-a1e5-4863-89b7-df655ab1da3a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c6fa59d-a1e5-4863-89b7-df655ab1da3a-maas-api-xks-ca\") pod \"1c6fa59d-a1e5-4863-89b7-df655ab1da3a\" (UID: \"1c6fa59d-a1e5-4863-89b7-df655ab1da3a\") " Nov 01 21:22:53.152105 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:53.152095 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c6fa59d-a1e5-4863-89b7-df655ab1da3a-maas-api-openshift-ca" pod "1c6fa59d-a1e5-4863-89b7-df655ab1da3a" (UID: "1c6fa59d-a1e5-4863-89b7-df655ab1da3a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:22:53.152234 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:53.152192 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c6fa59d-a1e5-4863-89b7-df655ab1da3a-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:22:53.154066 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:53.154042 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c6fa59d-a1e5-4863-89b7-df655ab1da3a-kube-api-access-7k525" pod "1c6fa59d-a1e5-4863-89b7-df655ab1da3a" (UID: "1c6fa59d-a1e5-4863-89b7-df655ab1da3a"). InnerVolumeSpecName "kube-api-access-7k525". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:22:53.154175 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:53.154064 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c6fa59d-a1e5-4863-89b7-df655ab1da3a-maas-api-xks-ca" pod "1c6fa59d-a1e5-4863-89b7-df655ab1da3a" (UID: "1c6fa59d-a1e5-4863-89b7-df655ab1da3a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:22:53.253241 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:53.253207 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k525\" (UniqueName: \"kubernetes.io/projected/1c6fa59d-a1e5-4863-89b7-df655ab1da3a-kube-api-access-7k525\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:22:53.253241 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:53.253238 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c6fa59d-a1e5-4863-89b7-df655ab1da3a-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:22:53.841754 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:53.841725 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m78x6" Nov 01 21:22:53.841944 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:53.841764 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m78x6" event={"ID":"1c6fa59d-a1e5-4863-89b7-df655ab1da3a","Type":"ContainerDied","Data":"7f68c991e4c975e87b914b024ceae977f173d1c75e861ba4e8f00da553e20a5d"} Nov 01 21:22:53.841944 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:22:53.841833 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f68c991e4c975e87b914b024ceae977f173d1c75e861ba4e8f00da553e20a5d" Nov 01 21:23:02.252216 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:02.252170 4484 scope.go:122] "RemoveContainer" containerID="6f27b48ef9958eb2b6ffa1d3db2ad0ccc8a036a6becf65fbc03a3652873725a2" Nov 01 21:23:02.261442 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:02.261421 4484 scope.go:122] "RemoveContainer" containerID="49e41a02b1d29dc4d48163884adc3dcbc1167f6b93e337cfca19325f494ec07a" Nov 01 21:23:02.271295 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:02.271268 4484 scope.go:122] "RemoveContainer" containerID="bf7cd31a6c28f30e1df47bb433e7bb3cf4cbd7bc43b2818898c15cbc1fcf0915" Nov 01 21:23:02.279432 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:02.279410 4484 scope.go:122] "RemoveContainer" containerID="e370babedebd99baec75b329b2ab52a05c74d448551236c89cc38928b63d2840" Nov 01 21:23:02.287724 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:02.287692 4484 scope.go:122] "RemoveContainer" containerID="09021bf263260cd19389c2a6b85e9df13034f21c56eabe380a191b97c4cb058d" Nov 01 21:23:02.296206 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:02.296182 4484 scope.go:122] "RemoveContainer" containerID="5bf567914a60ad7a46214015df4d59bbb5c6a09f776bb228f164f68cbe9072be" Nov 01 21:23:15.766824 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:23:15.766785 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:23:39.044730 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:39.044691 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jhjxx"] Nov 01 21:23:39.095496 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:39.095454 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jhjxx"] Nov 01 21:23:39.369991 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:39.369904 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae151e4d-e3ee-451a-8776-2e9426c391c2" path="/var/lib/kubelet/pods/ae151e4d-e3ee-451a-8776-2e9426c391c2/volumes" Nov 01 21:23:39.664534 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:39.664437 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6str6"] Nov 01 21:23:39.664910 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:39.664890 4484 cpu_manager.go:414] "containerMap: removing container" podUID="1c6fa59d-a1e5-4863-89b7-df655ab1da3a" containerName="revoke-keys" Nov 01 21:23:39.664910 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:39.664909 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c6fa59d-a1e5-4863-89b7-df655ab1da3a" containerName="revoke-keys" Nov 01 21:23:39.665127 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:39.665026 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c6fa59d-a1e5-4863-89b7-df655ab1da3a" containerName="revoke-keys" Nov 01 21:23:39.668476 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:39.668457 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6str6" Nov 01 21:23:39.724147 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:39.724115 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6str6"] Nov 01 21:23:39.758286 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:39.758240 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ff52970-facf-4ef5-b12a-c9cbc3bed639-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-6str6\" (UID: \"0ff52970-facf-4ef5-b12a-c9cbc3bed639\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6str6" Nov 01 21:23:39.758500 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:39.758300 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ff52970-facf-4ef5-b12a-c9cbc3bed639-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-6str6\" (UID: \"0ff52970-facf-4ef5-b12a-c9cbc3bed639\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6str6" Nov 01 21:23:39.758500 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:39.758415 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpdqh\" (UniqueName: \"kubernetes.io/projected/0ff52970-facf-4ef5-b12a-c9cbc3bed639-kube-api-access-jpdqh\") pod \"maas-api-revoke-sub-e2fc9c5b-6str6\" (UID: \"0ff52970-facf-4ef5-b12a-c9cbc3bed639\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6str6" Nov 01 21:23:39.772037 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:39.771982 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:23:39.772790 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:39.772774 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:23:39.772790 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:39.772778 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:23:39.772894 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:39.772777 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:23:39.859455 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:39.859414 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ff52970-facf-4ef5-b12a-c9cbc3bed639-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-6str6\" (UID: \"0ff52970-facf-4ef5-b12a-c9cbc3bed639\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6str6" Nov 01 21:23:39.859638 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:39.859501 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpdqh\" (UniqueName: \"kubernetes.io/projected/0ff52970-facf-4ef5-b12a-c9cbc3bed639-kube-api-access-jpdqh\") pod \"maas-api-revoke-sub-e2fc9c5b-6str6\" (UID: \"0ff52970-facf-4ef5-b12a-c9cbc3bed639\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6str6" Nov 01 21:23:39.859683 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:39.859636 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ff52970-facf-4ef5-b12a-c9cbc3bed639-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-6str6\" (UID: \"0ff52970-facf-4ef5-b12a-c9cbc3bed639\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6str6" Nov 01 21:23:39.860084 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:39.860066 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ff52970-facf-4ef5-b12a-c9cbc3bed639-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-6str6\" (UID: \"0ff52970-facf-4ef5-b12a-c9cbc3bed639\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6str6" Nov 01 21:23:39.861913 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:39.861895 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ff52970-facf-4ef5-b12a-c9cbc3bed639-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-6str6\" (UID: \"0ff52970-facf-4ef5-b12a-c9cbc3bed639\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6str6" Nov 01 21:23:39.917441 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:39.917373 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpdqh\" (UniqueName: \"kubernetes.io/projected/0ff52970-facf-4ef5-b12a-c9cbc3bed639-kube-api-access-jpdqh\") pod \"maas-api-revoke-sub-e2fc9c5b-6str6\" (UID: \"0ff52970-facf-4ef5-b12a-c9cbc3bed639\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6str6" Nov 01 21:23:39.981016 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:39.980953 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6str6" Nov 01 21:23:40.352204 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:40.352177 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6str6"] Nov 01 21:23:40.353969 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:23:40.353944 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ff52970_facf_4ef5_b12a_c9cbc3bed639.slice/crio-e9a491b10b3f87c682bf1c6fea431b1ec3f53821b435807fb32d42889cb062c4 WatchSource:0}: Error finding container e9a491b10b3f87c682bf1c6fea431b1ec3f53821b435807fb32d42889cb062c4: Status 404 returned error can't find the container with id e9a491b10b3f87c682bf1c6fea431b1ec3f53821b435807fb32d42889cb062c4 Nov 01 21:23:41.118136 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:41.118104 4484 generic.go:356] "Generic (PLEG): container finished" podID="0ff52970-facf-4ef5-b12a-c9cbc3bed639" containerID="af7ac39d3c6a7474cf841ef458cf14abb8ac2cd62dec91c8e3f467481717cc1e" exitCode=0 Nov 01 21:23:41.118264 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:41.118187 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6str6" event={"ID":"0ff52970-facf-4ef5-b12a-c9cbc3bed639","Type":"ContainerDied","Data":"af7ac39d3c6a7474cf841ef458cf14abb8ac2cd62dec91c8e3f467481717cc1e"} Nov 01 21:23:41.118308 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:41.118266 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6str6" event={"ID":"0ff52970-facf-4ef5-b12a-c9cbc3bed639","Type":"ContainerStarted","Data":"e9a491b10b3f87c682bf1c6fea431b1ec3f53821b435807fb32d42889cb062c4"} Nov 01 21:23:42.255738 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:42.255711 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6str6" Nov 01 21:23:42.284107 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:42.284061 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ff52970-facf-4ef5-b12a-c9cbc3bed639-kube-api-access-jpdqh\" (UniqueName: \"kubernetes.io/projected/0ff52970-facf-4ef5-b12a-c9cbc3bed639-kube-api-access-jpdqh\") pod \"0ff52970-facf-4ef5-b12a-c9cbc3bed639\" (UID: \"0ff52970-facf-4ef5-b12a-c9cbc3bed639\") " Nov 01 21:23:42.284310 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:42.284263 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ff52970-facf-4ef5-b12a-c9cbc3bed639-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ff52970-facf-4ef5-b12a-c9cbc3bed639-maas-api-openshift-ca\") pod \"0ff52970-facf-4ef5-b12a-c9cbc3bed639\" (UID: \"0ff52970-facf-4ef5-b12a-c9cbc3bed639\") " Nov 01 21:23:42.284389 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:42.284305 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ff52970-facf-4ef5-b12a-c9cbc3bed639-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ff52970-facf-4ef5-b12a-c9cbc3bed639-maas-api-xks-ca\") pod \"0ff52970-facf-4ef5-b12a-c9cbc3bed639\" (UID: \"0ff52970-facf-4ef5-b12a-c9cbc3bed639\") " Nov 01 21:23:42.284601 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:42.284573 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ff52970-facf-4ef5-b12a-c9cbc3bed639-maas-api-openshift-ca" pod "0ff52970-facf-4ef5-b12a-c9cbc3bed639" (UID: "0ff52970-facf-4ef5-b12a-c9cbc3bed639"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:23:42.286191 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:42.286167 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ff52970-facf-4ef5-b12a-c9cbc3bed639-kube-api-access-jpdqh" pod "0ff52970-facf-4ef5-b12a-c9cbc3bed639" (UID: "0ff52970-facf-4ef5-b12a-c9cbc3bed639"). InnerVolumeSpecName "kube-api-access-jpdqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:23:42.286320 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:42.286301 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ff52970-facf-4ef5-b12a-c9cbc3bed639-maas-api-xks-ca" pod "0ff52970-facf-4ef5-b12a-c9cbc3bed639" (UID: "0ff52970-facf-4ef5-b12a-c9cbc3bed639"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:23:42.385535 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:42.385448 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ff52970-facf-4ef5-b12a-c9cbc3bed639-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:23:42.385535 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:42.385483 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ff52970-facf-4ef5-b12a-c9cbc3bed639-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:23:42.385535 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:42.385499 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpdqh\" (UniqueName: \"kubernetes.io/projected/0ff52970-facf-4ef5-b12a-c9cbc3bed639-kube-api-access-jpdqh\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:23:43.129875 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:43.129850 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6str6" Nov 01 21:23:43.130070 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:43.129850 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6str6" event={"ID":"0ff52970-facf-4ef5-b12a-c9cbc3bed639","Type":"ContainerDied","Data":"e9a491b10b3f87c682bf1c6fea431b1ec3f53821b435807fb32d42889cb062c4"} Nov 01 21:23:43.130070 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:43.129972 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9a491b10b3f87c682bf1c6fea431b1ec3f53821b435807fb32d42889cb062c4" Nov 01 21:23:45.766703 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:23:45.766665 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:23:50.455203 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:50.455169 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j7lwr"] Nov 01 21:23:50.455664 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:50.455621 4484 cpu_manager.go:414] "containerMap: removing container" podUID="0ff52970-facf-4ef5-b12a-c9cbc3bed639" containerName="revoke-keys" Nov 01 21:23:50.455664 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:50.455634 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ff52970-facf-4ef5-b12a-c9cbc3bed639" containerName="revoke-keys" Nov 01 21:23:50.455761 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:50.455728 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ff52970-facf-4ef5-b12a-c9cbc3bed639" containerName="revoke-keys" Nov 01 21:23:50.459327 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:50.459309 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j7lwr" Nov 01 21:23:50.514810 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:50.514772 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j7lwr"] Nov 01 21:23:50.563051 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:50.563013 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:23:50.563051 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:50.563017 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:23:50.563051 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:50.563016 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:23:50.563988 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:50.563962 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:23:50.570428 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:50.570408 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/153c5475-eb19-47b8-bba6-f9ce73945692-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-j7lwr\" (UID: \"153c5475-eb19-47b8-bba6-f9ce73945692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j7lwr" Nov 01 21:23:50.570526 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:50.570440 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/153c5475-eb19-47b8-bba6-f9ce73945692-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-j7lwr\" (UID: \"153c5475-eb19-47b8-bba6-f9ce73945692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j7lwr" Nov 01 21:23:50.570582 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:50.570566 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpdd9\" (UniqueName: \"kubernetes.io/projected/153c5475-eb19-47b8-bba6-f9ce73945692-kube-api-access-zpdd9\") pod \"maas-api-revoke-sub-5a29efb3-j7lwr\" (UID: \"153c5475-eb19-47b8-bba6-f9ce73945692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j7lwr" Nov 01 21:23:50.671595 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:50.671559 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/153c5475-eb19-47b8-bba6-f9ce73945692-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-j7lwr\" (UID: \"153c5475-eb19-47b8-bba6-f9ce73945692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j7lwr" Nov 01 21:23:50.671595 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:50.671601 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/153c5475-eb19-47b8-bba6-f9ce73945692-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-j7lwr\" (UID: \"153c5475-eb19-47b8-bba6-f9ce73945692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j7lwr" Nov 01 21:23:50.671826 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:50.671672 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpdd9\" (UniqueName: \"kubernetes.io/projected/153c5475-eb19-47b8-bba6-f9ce73945692-kube-api-access-zpdd9\") pod \"maas-api-revoke-sub-5a29efb3-j7lwr\" (UID: \"153c5475-eb19-47b8-bba6-f9ce73945692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j7lwr" Nov 01 21:23:50.672230 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:50.672208 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/153c5475-eb19-47b8-bba6-f9ce73945692-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-j7lwr\" (UID: \"153c5475-eb19-47b8-bba6-f9ce73945692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j7lwr" Nov 01 21:23:50.674108 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:50.674087 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/153c5475-eb19-47b8-bba6-f9ce73945692-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-j7lwr\" (UID: \"153c5475-eb19-47b8-bba6-f9ce73945692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j7lwr" Nov 01 21:23:50.729276 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:50.729211 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpdd9\" (UniqueName: \"kubernetes.io/projected/153c5475-eb19-47b8-bba6-f9ce73945692-kube-api-access-zpdd9\") pod \"maas-api-revoke-sub-5a29efb3-j7lwr\" (UID: \"153c5475-eb19-47b8-bba6-f9ce73945692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j7lwr" Nov 01 21:23:50.773400 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:50.773372 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j7lwr" Nov 01 21:23:51.144378 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:51.144351 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j7lwr"] Nov 01 21:23:51.145211 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:23:51.145184 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod153c5475_eb19_47b8_bba6_f9ce73945692.slice/crio-aad59eb50fcc6dd03c3770bd25cbb39554ed2bddfa23e67c6019c14bde97e89c WatchSource:0}: Error finding container aad59eb50fcc6dd03c3770bd25cbb39554ed2bddfa23e67c6019c14bde97e89c: Status 404 returned error can't find the container with id aad59eb50fcc6dd03c3770bd25cbb39554ed2bddfa23e67c6019c14bde97e89c Nov 01 21:23:51.171748 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:51.171722 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j7lwr" event={"ID":"153c5475-eb19-47b8-bba6-f9ce73945692","Type":"ContainerStarted","Data":"aad59eb50fcc6dd03c3770bd25cbb39554ed2bddfa23e67c6019c14bde97e89c"} Nov 01 21:23:52.179241 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:52.179150 4484 generic.go:356] "Generic (PLEG): container finished" podID="153c5475-eb19-47b8-bba6-f9ce73945692" containerID="7b709062397a2ed5118d6d9f047f8b38b74debf8cb31bee32912c1a9fff87716" exitCode=0 Nov 01 21:23:52.179241 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:52.179211 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j7lwr" event={"ID":"153c5475-eb19-47b8-bba6-f9ce73945692","Type":"ContainerDied","Data":"7b709062397a2ed5118d6d9f047f8b38b74debf8cb31bee32912c1a9fff87716"} Nov 01 21:23:53.384709 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:53.384673 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j7lwr" Nov 01 21:23:53.414973 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:53.414906 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:23:53.503373 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:53.503335 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/153c5475-eb19-47b8-bba6-f9ce73945692-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/153c5475-eb19-47b8-bba6-f9ce73945692-maas-api-openshift-ca\") pod \"153c5475-eb19-47b8-bba6-f9ce73945692\" (UID: \"153c5475-eb19-47b8-bba6-f9ce73945692\") " Nov 01 21:23:53.503577 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:53.503449 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/153c5475-eb19-47b8-bba6-f9ce73945692-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/153c5475-eb19-47b8-bba6-f9ce73945692-maas-api-xks-ca\") pod \"153c5475-eb19-47b8-bba6-f9ce73945692\" (UID: \"153c5475-eb19-47b8-bba6-f9ce73945692\") " Nov 01 21:23:53.503577 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:53.503520 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/153c5475-eb19-47b8-bba6-f9ce73945692-kube-api-access-zpdd9\" (UniqueName: \"kubernetes.io/projected/153c5475-eb19-47b8-bba6-f9ce73945692-kube-api-access-zpdd9\") pod \"153c5475-eb19-47b8-bba6-f9ce73945692\" (UID: \"153c5475-eb19-47b8-bba6-f9ce73945692\") " Nov 01 21:23:53.505557 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:53.505523 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/153c5475-eb19-47b8-bba6-f9ce73945692-maas-api-openshift-ca" pod "153c5475-eb19-47b8-bba6-f9ce73945692" (UID: "153c5475-eb19-47b8-bba6-f9ce73945692"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:23:53.510845 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:53.507825 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/153c5475-eb19-47b8-bba6-f9ce73945692-maas-api-xks-ca" pod "153c5475-eb19-47b8-bba6-f9ce73945692" (UID: "153c5475-eb19-47b8-bba6-f9ce73945692"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:23:53.510845 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:53.508836 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/153c5475-eb19-47b8-bba6-f9ce73945692-kube-api-access-zpdd9" pod "153c5475-eb19-47b8-bba6-f9ce73945692" (UID: "153c5475-eb19-47b8-bba6-f9ce73945692"). InnerVolumeSpecName "kube-api-access-zpdd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:23:53.605016 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:53.604950 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/153c5475-eb19-47b8-bba6-f9ce73945692-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:23:53.605016 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:53.605013 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpdd9\" (UniqueName: \"kubernetes.io/projected/153c5475-eb19-47b8-bba6-f9ce73945692-kube-api-access-zpdd9\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:23:53.605307 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:53.605035 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/153c5475-eb19-47b8-bba6-f9ce73945692-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:23:54.211019 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:54.199197 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j7lwr" Nov 01 21:23:54.211019 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:54.199286 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j7lwr" event={"ID":"153c5475-eb19-47b8-bba6-f9ce73945692","Type":"ContainerDied","Data":"aad59eb50fcc6dd03c3770bd25cbb39554ed2bddfa23e67c6019c14bde97e89c"} Nov 01 21:23:54.211019 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:23:54.199345 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aad59eb50fcc6dd03c3770bd25cbb39554ed2bddfa23e67c6019c14bde97e89c" Nov 01 21:24:02.347977 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:24:02.347898 4484 scope.go:122] "RemoveContainer" containerID="7da3462e5c91133c44b0c89bbadb01097da4d0acdd5edce5bf3a4b959daaaa15" Nov 01 21:24:10.045913 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:24:10.045865 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qd4qk"] Nov 01 21:24:10.096566 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:24:10.096526 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qd4qk"] Nov 01 21:24:11.370546 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:24:11.370502 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc9aa86d-e00e-461d-8443-3c28b1f1882c" path="/var/lib/kubelet/pods/dc9aa86d-e00e-461d-8443-3c28b1f1882c/volumes" Nov 01 21:24:15.766347 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:24:15.766309 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:24:23.044826 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:24:23.044792 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-z2bbt"] Nov 01 21:24:23.095491 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:24:23.095454 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-z2bbt"] Nov 01 21:24:23.370375 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:24:23.370301 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4deb7c07-9529-4d3e-8ca0-ecec5f4d86fe" path="/var/lib/kubelet/pods/4deb7c07-9529-4d3e-8ca0-ecec5f4d86fe/volumes" Nov 01 21:24:35.841457 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:24:35.841420 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m78x6"] Nov 01 21:24:35.892332 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:24:35.892298 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m78x6"] Nov 01 21:24:35.954088 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:24:35.954049 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kpr7v"] Nov 01 21:24:35.954469 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:24:35.954457 4484 cpu_manager.go:414] "containerMap: removing container" podUID="153c5475-eb19-47b8-bba6-f9ce73945692" containerName="revoke-keys" Nov 01 21:24:35.954537 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:24:35.954470 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="153c5475-eb19-47b8-bba6-f9ce73945692" containerName="revoke-keys" Nov 01 21:24:35.954537 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:24:35.954533 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="153c5475-eb19-47b8-bba6-f9ce73945692" containerName="revoke-keys" Nov 01 21:24:35.957838 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:24:35.957819 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kpr7v" Nov 01 21:24:36.011706 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:24:36.011678 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:24:36.011937 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:24:36.011912 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:24:36.013958 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:24:36.013931 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kpr7v"] Nov 01 21:24:36.035403 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:24:36.035366 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:24:36.050044 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:24:36.049988 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbq4k\" (UniqueName: \"kubernetes.io/projected/c3c4a858-de7c-47af-9718-ef53f733df59-kube-api-access-qbq4k\") pod \"maas-api-revoke-sub-3d4156f1-kpr7v\" (UID: \"c3c4a858-de7c-47af-9718-ef53f733df59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kpr7v" Nov 01 21:24:36.050243 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:24:36.050221 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3c4a858-de7c-47af-9718-ef53f733df59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-kpr7v\" (UID: \"c3c4a858-de7c-47af-9718-ef53f733df59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kpr7v" Nov 01 21:24:36.050311 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:24:36.050270 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3c4a858-de7c-47af-9718-ef53f733df59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-kpr7v\" (UID: \"c3c4a858-de7c-47af-9718-ef53f733df59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kpr7v" Nov 01 21:24:36.061658 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:24:36.061635 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:24:36.150878 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:24:36.150792 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3c4a858-de7c-47af-9718-ef53f733df59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-kpr7v\" (UID: \"c3c4a858-de7c-47af-9718-ef53f733df59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kpr7v" Nov 01 21:24:36.150878 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:24:36.150829 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3c4a858-de7c-47af-9718-ef53f733df59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-kpr7v\" (UID: \"c3c4a858-de7c-47af-9718-ef53f733df59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kpr7v" Nov 01 21:24:36.150878 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:24:36.150865 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbq4k\" (UniqueName: \"kubernetes.io/projected/c3c4a858-de7c-47af-9718-ef53f733df59-kube-api-access-qbq4k\") pod \"maas-api-revoke-sub-3d4156f1-kpr7v\" (UID: \"c3c4a858-de7c-47af-9718-ef53f733df59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kpr7v" Nov 01 21:24:36.151549 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:24:36.151523 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3c4a858-de7c-47af-9718-ef53f733df59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-kpr7v\" (UID: \"c3c4a858-de7c-47af-9718-ef53f733df59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kpr7v" Nov 01 21:24:36.153210 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:24:36.153189 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3c4a858-de7c-47af-9718-ef53f733df59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-kpr7v\" (UID: \"c3c4a858-de7c-47af-9718-ef53f733df59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kpr7v" Nov 01 21:24:36.208633 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:24:36.208600 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbq4k\" (UniqueName: \"kubernetes.io/projected/c3c4a858-de7c-47af-9718-ef53f733df59-kube-api-access-qbq4k\") pod \"maas-api-revoke-sub-3d4156f1-kpr7v\" (UID: \"c3c4a858-de7c-47af-9718-ef53f733df59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kpr7v" Nov 01 21:24:36.270394 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:24:36.270356 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kpr7v" Nov 01 21:24:36.639820 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:24:36.639795 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kpr7v"] Nov 01 21:24:36.641746 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:24:36.641716 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3c4a858_de7c_47af_9718_ef53f733df59.slice/crio-2499c10772d1995082597e81251b5f7038a599f42bc5b61aa1dd1896d816a011 WatchSource:0}: Error finding container 2499c10772d1995082597e81251b5f7038a599f42bc5b61aa1dd1896d816a011: Status 404 returned error can't find the container with id 2499c10772d1995082597e81251b5f7038a599f42bc5b61aa1dd1896d816a011 Nov 01 21:24:37.370949 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:24:37.370908 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c6fa59d-a1e5-4863-89b7-df655ab1da3a" path="/var/lib/kubelet/pods/1c6fa59d-a1e5-4863-89b7-df655ab1da3a/volumes" Nov 01 21:24:37.442720 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:24:37.442681 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kpr7v" event={"ID":"c3c4a858-de7c-47af-9718-ef53f733df59","Type":"ContainerStarted","Data":"48db40f94ffc4b64e30aaf555a59b0eda483a0f7ae32efc831cb5fdb3d29be32"} Nov 01 21:24:37.442946 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:24:37.442774 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kpr7v" event={"ID":"c3c4a858-de7c-47af-9718-ef53f733df59","Type":"ContainerStarted","Data":"2499c10772d1995082597e81251b5f7038a599f42bc5b61aa1dd1896d816a011"} Nov 01 21:24:37.609863 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:24:37.609809 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kpr7v" podStartSLOduration=2.609795132 podStartE2EDuration="2.609795132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:24:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:24:37.558192547 +0000 UTC m=+3228.335832683" watchObservedRunningTime="2026-11-01 21:24:37.609795132 +0000 UTC m=+3228.387435267" Nov 01 21:24:38.449750 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:24:38.449713 4484 generic.go:356] "Generic (PLEG): container finished" podID="c3c4a858-de7c-47af-9718-ef53f733df59" containerID="48db40f94ffc4b64e30aaf555a59b0eda483a0f7ae32efc831cb5fdb3d29be32" exitCode=0 Nov 01 21:24:38.450210 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:24:38.449781 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kpr7v" event={"ID":"c3c4a858-de7c-47af-9718-ef53f733df59","Type":"ContainerDied","Data":"48db40f94ffc4b64e30aaf555a59b0eda483a0f7ae32efc831cb5fdb3d29be32"} Nov 01 21:24:39.599274 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:24:39.599249 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kpr7v" Nov 01 21:24:39.684458 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:24:39.684424 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3c4a858-de7c-47af-9718-ef53f733df59-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3c4a858-de7c-47af-9718-ef53f733df59-maas-api-openshift-ca\") pod \"c3c4a858-de7c-47af-9718-ef53f733df59\" (UID: \"c3c4a858-de7c-47af-9718-ef53f733df59\") " Nov 01 21:24:39.684458 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:24:39.684465 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3c4a858-de7c-47af-9718-ef53f733df59-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3c4a858-de7c-47af-9718-ef53f733df59-maas-api-xks-ca\") pod \"c3c4a858-de7c-47af-9718-ef53f733df59\" (UID: \"c3c4a858-de7c-47af-9718-ef53f733df59\") " Nov 01 21:24:39.684724 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:24:39.684618 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3c4a858-de7c-47af-9718-ef53f733df59-kube-api-access-qbq4k\" (UniqueName: \"kubernetes.io/projected/c3c4a858-de7c-47af-9718-ef53f733df59-kube-api-access-qbq4k\") pod \"c3c4a858-de7c-47af-9718-ef53f733df59\" (UID: \"c3c4a858-de7c-47af-9718-ef53f733df59\") " Nov 01 21:24:39.684865 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:24:39.684842 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3c4a858-de7c-47af-9718-ef53f733df59-maas-api-openshift-ca" pod "c3c4a858-de7c-47af-9718-ef53f733df59" (UID: "c3c4a858-de7c-47af-9718-ef53f733df59"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:24:39.684991 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:24:39.684975 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3c4a858-de7c-47af-9718-ef53f733df59-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:24:39.686681 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:24:39.686651 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3c4a858-de7c-47af-9718-ef53f733df59-maas-api-xks-ca" pod "c3c4a858-de7c-47af-9718-ef53f733df59" (UID: "c3c4a858-de7c-47af-9718-ef53f733df59"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:24:39.686781 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:24:39.686719 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3c4a858-de7c-47af-9718-ef53f733df59-kube-api-access-qbq4k" pod "c3c4a858-de7c-47af-9718-ef53f733df59" (UID: "c3c4a858-de7c-47af-9718-ef53f733df59"). InnerVolumeSpecName "kube-api-access-qbq4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:24:39.786477 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:24:39.786440 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbq4k\" (UniqueName: \"kubernetes.io/projected/c3c4a858-de7c-47af-9718-ef53f733df59-kube-api-access-qbq4k\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:24:39.786477 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:24:39.786472 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3c4a858-de7c-47af-9718-ef53f733df59-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:24:40.461604 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:24:40.461570 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kpr7v" event={"ID":"c3c4a858-de7c-47af-9718-ef53f733df59","Type":"ContainerDied","Data":"2499c10772d1995082597e81251b5f7038a599f42bc5b61aa1dd1896d816a011"} Nov 01 21:24:40.461604 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:24:40.461600 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kpr7v" Nov 01 21:24:40.461825 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:24:40.461619 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2499c10772d1995082597e81251b5f7038a599f42bc5b61aa1dd1896d816a011" Nov 01 21:24:45.766666 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:24:45.766625 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:24:51.455025 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:24:51.444109 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:24:58.045613 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:24:58.045576 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qfprx"] Nov 01 21:24:58.096499 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:24:58.096465 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qfprx"] Nov 01 21:24:59.370562 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:24:59.370521 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3c70aaa-58e4-41f8-8a2c-7b723c51d82d" path="/var/lib/kubelet/pods/f3c70aaa-58e4-41f8-8a2c-7b723c51d82d/volumes" Nov 01 21:25:02.371222 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:02.371187 4484 scope.go:122] "RemoveContainer" containerID="f29df4c262341b45d7dee686156be2212fbda5de114702874782c62ebf0face8" Nov 01 21:25:07.540037 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:07.539988 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:25:08.012182 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:08.012148 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n"] Nov 01 21:25:08.012607 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:08.012593 4484 cpu_manager.go:414] "containerMap: removing container" podUID="c3c4a858-de7c-47af-9718-ef53f733df59" containerName="revoke-keys" Nov 01 21:25:08.012659 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:08.012608 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3c4a858-de7c-47af-9718-ef53f733df59" containerName="revoke-keys" Nov 01 21:25:08.012696 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:08.012691 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3c4a858-de7c-47af-9718-ef53f733df59" containerName="revoke-keys" Nov 01 21:25:08.017376 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:08.017357 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n" Nov 01 21:25:08.084042 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:08.082941 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-7249f4-openshift-default-dockercfg-wdzpc\"" Nov 01 21:25:08.101368 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:08.101337 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acd7c6cc-09ae-4036-b815-64b107627c63-workload-certs\") pod \"e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n\" (UID: \"acd7c6cc-09ae-4036-b815-64b107627c63\") " pod="openshift-ingress/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n" Nov 01 21:25:08.101561 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:08.101387 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmq97\" (UniqueName: \"kubernetes.io/projected/acd7c6cc-09ae-4036-b815-64b107627c63-kube-api-access-zmq97\") pod \"e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n\" (UID: \"acd7c6cc-09ae-4036-b815-64b107627c63\") " pod="openshift-ingress/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n" Nov 01 21:25:08.101561 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:08.101524 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acd7c6cc-09ae-4036-b815-64b107627c63-istiod-ca-cert\") pod \"e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n\" (UID: \"acd7c6cc-09ae-4036-b815-64b107627c63\") " pod="openshift-ingress/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n" Nov 01 21:25:08.101654 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:08.101570 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acd7c6cc-09ae-4036-b815-64b107627c63-istio-envoy\") pod \"e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n\" (UID: \"acd7c6cc-09ae-4036-b815-64b107627c63\") " pod="openshift-ingress/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n" Nov 01 21:25:08.101709 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:08.101685 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acd7c6cc-09ae-4036-b815-64b107627c63-istio-podinfo\") pod \"e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n\" (UID: \"acd7c6cc-09ae-4036-b815-64b107627c63\") " pod="openshift-ingress/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n" Nov 01 21:25:08.101762 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:08.101729 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acd7c6cc-09ae-4036-b815-64b107627c63-istio-token\") pod \"e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n\" (UID: \"acd7c6cc-09ae-4036-b815-64b107627c63\") " pod="openshift-ingress/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n" Nov 01 21:25:08.101862 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:08.101846 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acd7c6cc-09ae-4036-b815-64b107627c63-credential-socket\") pod \"e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n\" (UID: \"acd7c6cc-09ae-4036-b815-64b107627c63\") " pod="openshift-ingress/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n" Nov 01 21:25:08.101912 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:08.101895 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acd7c6cc-09ae-4036-b815-64b107627c63-workload-socket\") pod \"e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n\" (UID: \"acd7c6cc-09ae-4036-b815-64b107627c63\") " pod="openshift-ingress/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n" Nov 01 21:25:08.102042 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:08.102023 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acd7c6cc-09ae-4036-b815-64b107627c63-istio-data\") pod \"e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n\" (UID: \"acd7c6cc-09ae-4036-b815-64b107627c63\") " pod="openshift-ingress/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n" Nov 01 21:25:08.132031 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:08.131974 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n"] Nov 01 21:25:08.134124 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:08.134094 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:25:08.202892 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:08.202851 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acd7c6cc-09ae-4036-b815-64b107627c63-istio-envoy\") pod \"e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n\" (UID: \"acd7c6cc-09ae-4036-b815-64b107627c63\") " pod="openshift-ingress/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n" Nov 01 21:25:08.203158 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:08.202907 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acd7c6cc-09ae-4036-b815-64b107627c63-istio-podinfo\") pod \"e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n\" (UID: \"acd7c6cc-09ae-4036-b815-64b107627c63\") " pod="openshift-ingress/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n" Nov 01 21:25:08.203158 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:08.202926 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acd7c6cc-09ae-4036-b815-64b107627c63-istio-token\") pod \"e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n\" (UID: \"acd7c6cc-09ae-4036-b815-64b107627c63\") " pod="openshift-ingress/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n" Nov 01 21:25:08.203158 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:08.202974 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acd7c6cc-09ae-4036-b815-64b107627c63-credential-socket\") pod \"e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n\" (UID: \"acd7c6cc-09ae-4036-b815-64b107627c63\") " pod="openshift-ingress/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n" Nov 01 21:25:08.203372 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:08.203166 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acd7c6cc-09ae-4036-b815-64b107627c63-workload-socket\") pod \"e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n\" (UID: \"acd7c6cc-09ae-4036-b815-64b107627c63\") " pod="openshift-ingress/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n" Nov 01 21:25:08.203372 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:08.203234 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acd7c6cc-09ae-4036-b815-64b107627c63-istio-data\") pod \"e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n\" (UID: \"acd7c6cc-09ae-4036-b815-64b107627c63\") " pod="openshift-ingress/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n" Nov 01 21:25:08.203372 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:08.203266 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acd7c6cc-09ae-4036-b815-64b107627c63-workload-certs\") pod \"e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n\" (UID: \"acd7c6cc-09ae-4036-b815-64b107627c63\") " pod="openshift-ingress/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n" Nov 01 21:25:08.203372 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:08.203308 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmq97\" (UniqueName: \"kubernetes.io/projected/acd7c6cc-09ae-4036-b815-64b107627c63-kube-api-access-zmq97\") pod \"e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n\" (UID: \"acd7c6cc-09ae-4036-b815-64b107627c63\") " pod="openshift-ingress/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n" Nov 01 21:25:08.203597 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:08.203394 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acd7c6cc-09ae-4036-b815-64b107627c63-istiod-ca-cert\") pod \"e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n\" (UID: \"acd7c6cc-09ae-4036-b815-64b107627c63\") " pod="openshift-ingress/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n" Nov 01 21:25:08.203597 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:08.203414 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acd7c6cc-09ae-4036-b815-64b107627c63-credential-socket\") pod \"e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n\" (UID: \"acd7c6cc-09ae-4036-b815-64b107627c63\") " pod="openshift-ingress/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n" Nov 01 21:25:08.203597 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:08.203512 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acd7c6cc-09ae-4036-b815-64b107627c63-workload-socket\") pod \"e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n\" (UID: \"acd7c6cc-09ae-4036-b815-64b107627c63\") " pod="openshift-ingress/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n" Nov 01 21:25:08.203597 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:08.203593 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acd7c6cc-09ae-4036-b815-64b107627c63-istio-data\") pod \"e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n\" (UID: \"acd7c6cc-09ae-4036-b815-64b107627c63\") " pod="openshift-ingress/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n" Nov 01 21:25:08.203814 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:08.203668 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acd7c6cc-09ae-4036-b815-64b107627c63-workload-certs\") pod \"e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n\" (UID: \"acd7c6cc-09ae-4036-b815-64b107627c63\") " pod="openshift-ingress/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n" Nov 01 21:25:08.204137 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:08.204117 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acd7c6cc-09ae-4036-b815-64b107627c63-istiod-ca-cert\") pod \"e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n\" (UID: \"acd7c6cc-09ae-4036-b815-64b107627c63\") " pod="openshift-ingress/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n" Nov 01 21:25:08.205456 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:08.205431 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acd7c6cc-09ae-4036-b815-64b107627c63-istio-envoy\") pod \"e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n\" (UID: \"acd7c6cc-09ae-4036-b815-64b107627c63\") " pod="openshift-ingress/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n" Nov 01 21:25:08.205873 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:08.205852 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acd7c6cc-09ae-4036-b815-64b107627c63-istio-podinfo\") pod \"e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n\" (UID: \"acd7c6cc-09ae-4036-b815-64b107627c63\") " pod="openshift-ingress/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n" Nov 01 21:25:08.263678 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:08.263603 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acd7c6cc-09ae-4036-b815-64b107627c63-istio-token\") pod \"e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n\" (UID: \"acd7c6cc-09ae-4036-b815-64b107627c63\") " pod="openshift-ingress/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n" Nov 01 21:25:08.264022 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:08.263977 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmq97\" (UniqueName: \"kubernetes.io/projected/acd7c6cc-09ae-4036-b815-64b107627c63-kube-api-access-zmq97\") pod \"e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n\" (UID: \"acd7c6cc-09ae-4036-b815-64b107627c63\") " pod="openshift-ingress/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n" Nov 01 21:25:08.355425 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:08.355379 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n" Nov 01 21:25:08.610295 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:08.610190 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:25:08.783804 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:08.783773 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n"] Nov 01 21:25:08.785208 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:25:08.785183 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacd7c6cc_09ae_4036_b815_64b107627c63.slice/crio-ef81e4296bdcaf80e5abc4ee2bf2e563b8f2ae90fff691fada2a0174bb43c137 WatchSource:0}: Error finding container ef81e4296bdcaf80e5abc4ee2bf2e563b8f2ae90fff691fada2a0174bb43c137: Status 404 returned error can't find the container with id ef81e4296bdcaf80e5abc4ee2bf2e563b8f2ae90fff691fada2a0174bb43c137 Nov 01 21:25:08.787396 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:08.787362 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:25:08.787460 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:08.787432 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:25:08.787504 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:08.787462 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:25:09.045010 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:09.044965 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-t5q9j"] Nov 01 21:25:09.047028 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:09.046975 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-t5q9j"] Nov 01 21:25:09.371351 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:09.371273 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abf725d3-0862-4578-9102-049f532baf95" path="/var/lib/kubelet/pods/abf725d3-0862-4578-9102-049f532baf95/volumes" Nov 01 21:25:09.642294 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:09.642204 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n" event={"ID":"acd7c6cc-09ae-4036-b815-64b107627c63","Type":"ContainerStarted","Data":"a3d5f2005bd09da34404ff2c334541c8c568a7ff9da7bb2a3c3b7abbaceeba39"} Nov 01 21:25:09.642294 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:09.642264 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n" event={"ID":"acd7c6cc-09ae-4036-b815-64b107627c63","Type":"ContainerStarted","Data":"ef81e4296bdcaf80e5abc4ee2bf2e563b8f2ae90fff691fada2a0174bb43c137"} Nov 01 21:25:09.761127 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:09.761072 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n" podStartSLOduration=2.761057083 podStartE2EDuration="2.761057083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:25:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:25:09.758692316 +0000 UTC m=+3260.536332481" watchObservedRunningTime="2026-11-01 21:25:09.761057083 +0000 UTC m=+3260.538697211" Nov 01 21:25:10.355536 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:10.355500 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n" Nov 01 21:25:10.360987 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:10.360955 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n" Nov 01 21:25:10.647977 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:10.647897 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n" Nov 01 21:25:10.648927 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:10.648908 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n" Nov 01 21:25:12.703946 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:12.703907 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:25:12.929251 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:12.929217 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:25:13.101138 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:13.101103 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:25:15.767309 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:25:15.767273 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:25:15.925297 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:15.925259 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:25:16.083167 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:16.083039 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:25:16.607887 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:16.607855 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7249f4-544589757t"] Nov 01 21:25:16.611751 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:16.611729 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7249f4-544589757t" Nov 01 21:25:16.668416 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:16.668384 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7249f4-544589757t"] Nov 01 21:25:16.703562 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:16.703525 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a8b983c-41b2-4020-bf69-f7641dbcca4d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-7249f4-544589757t\" (UID: \"3a8b983c-41b2-4020-bf69-f7641dbcca4d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7249f4-544589757t" Nov 01 21:25:16.703772 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:16.703658 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gctf5\" (UniqueName: \"kubernetes.io/projected/3a8b983c-41b2-4020-bf69-f7641dbcca4d-kube-api-access-gctf5\") pod \"payload-pre-processing-e2e-worker-single-main-7249f4-544589757t\" (UID: \"3a8b983c-41b2-4020-bf69-f7641dbcca4d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7249f4-544589757t" Nov 01 21:25:16.716337 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:16.716311 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-7249f4\"" Nov 01 21:25:16.716337 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:16.716322 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-e65bae51-dockercfg-mrpkb\"" Nov 01 21:25:16.804455 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:16.804414 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a8b983c-41b2-4020-bf69-f7641dbcca4d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-7249f4-544589757t\" (UID: \"3a8b983c-41b2-4020-bf69-f7641dbcca4d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7249f4-544589757t" Nov 01 21:25:16.804982 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:16.804494 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gctf5\" (UniqueName: \"kubernetes.io/projected/3a8b983c-41b2-4020-bf69-f7641dbcca4d-kube-api-access-gctf5\") pod \"payload-pre-processing-e2e-worker-single-main-7249f4-544589757t\" (UID: \"3a8b983c-41b2-4020-bf69-f7641dbcca4d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7249f4-544589757t" Nov 01 21:25:16.805369 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:16.805342 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a8b983c-41b2-4020-bf69-f7641dbcca4d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-7249f4-544589757t\" (UID: \"3a8b983c-41b2-4020-bf69-f7641dbcca4d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7249f4-544589757t" Nov 01 21:25:16.862909 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:16.862834 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gctf5\" (UniqueName: \"kubernetes.io/projected/3a8b983c-41b2-4020-bf69-f7641dbcca4d-kube-api-access-gctf5\") pod \"payload-pre-processing-e2e-worker-single-main-7249f4-544589757t\" (UID: \"3a8b983c-41b2-4020-bf69-f7641dbcca4d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7249f4-544589757t" Nov 01 21:25:16.925671 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:16.925630 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7249f4-544589757t" Nov 01 21:25:17.296957 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:17.296932 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7249f4-544589757t"] Nov 01 21:25:17.298878 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:25:17.298847 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a8b983c_41b2_4020_bf69_f7641dbcca4d.slice/crio-0a48f7794215ce5f7df741b599d7e2d874521ab3376148df68f51bbc1899d426 WatchSource:0}: Error finding container 0a48f7794215ce5f7df741b599d7e2d874521ab3376148df68f51bbc1899d426: Status 404 returned error can't find the container with id 0a48f7794215ce5f7df741b599d7e2d874521ab3376148df68f51bbc1899d426 Nov 01 21:25:17.686667 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:17.686578 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7249f4-544589757t" event={"ID":"3a8b983c-41b2-4020-bf69-f7641dbcca4d","Type":"ContainerStarted","Data":"6ebd9f85ef6b8e9626790d475a85a6876cc481f6b85dc1c49bb2451f0fc2c2b8"} Nov 01 21:25:17.686667 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:17.686638 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7249f4-544589757t" event={"ID":"3a8b983c-41b2-4020-bf69-f7641dbcca4d","Type":"ContainerStarted","Data":"0a48f7794215ce5f7df741b599d7e2d874521ab3376148df68f51bbc1899d426"} Nov 01 21:25:17.686887 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:17.686671 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7249f4-544589757t" Nov 01 21:25:17.801488 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:17.801439 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7249f4-544589757t" podStartSLOduration=1.801424887 podStartE2EDuration="1.801424887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:25:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:25:17.8009769 +0000 UTC m=+3268.578617036" watchObservedRunningTime="2026-11-01 21:25:17.801424887 +0000 UTC m=+3268.579065022" Nov 01 21:25:18.987620 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:18.987590 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:25:19.153701 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:19.153659 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:25:19.313223 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:19.313187 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:25:23.937500 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:23.937466 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:25:28.695546 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:28.695516 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7249f4-544589757t" Nov 01 21:25:28.752074 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:28.752032 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:25:33.542273 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:33.542234 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:25:36.806888 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:36.806852 4484 generic.go:356] "Generic (PLEG): container finished" podID="acd7c6cc-09ae-4036-b815-64b107627c63" containerID="a3d5f2005bd09da34404ff2c334541c8c568a7ff9da7bb2a3c3b7abbaceeba39" exitCode=0 Nov 01 21:25:36.807415 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:36.806927 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n" event={"ID":"acd7c6cc-09ae-4036-b815-64b107627c63","Type":"ContainerDied","Data":"a3d5f2005bd09da34404ff2c334541c8c568a7ff9da7bb2a3c3b7abbaceeba39"} Nov 01 21:25:36.807415 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:36.807327 4484 scope.go:122] "RemoveContainer" containerID="a3d5f2005bd09da34404ff2c334541c8c568a7ff9da7bb2a3c3b7abbaceeba39" Nov 01 21:25:36.808510 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:36.808480 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:25:36.808579 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:36.808548 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:25:36.808627 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:36.808580 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:25:37.267214 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:37.267176 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:25:37.784888 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:37.784855 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:25:37.833506 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:37.833476 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n" event={"ID":"acd7c6cc-09ae-4036-b815-64b107627c63","Type":"ContainerStarted","Data":"4efbb9775c502815b64a18674c1bbcbc4ccbdee756fa60673d3c4dd2b4ce8f2b"} Nov 01 21:25:37.945659 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:37.945619 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:25:38.355529 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:38.355464 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n" Nov 01 21:25:38.356941 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:38.356915 4484 patch_prober.go:29] interesting pod/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.50:15021/healthz/ready\": dial tcp 10.134.1.50:15021: connect: connection refused" start-of-body= Nov 01 21:25:38.357100 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:38.356968 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n" podUID="acd7c6cc-09ae-4036-b815-64b107627c63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.50:15021/healthz/ready\": dial tcp 10.134.1.50:15021: connect: connection refused" Nov 01 21:25:38.639389 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:38.639305 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:25:39.355778 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:39.355739 4484 patch_prober.go:29] interesting pod/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.50:15021/healthz/ready\": dial tcp 10.134.1.50:15021: connect: connection refused" start-of-body= Nov 01 21:25:39.356232 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:39.355793 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n" podUID="acd7c6cc-09ae-4036-b815-64b107627c63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.50:15021/healthz/ready\": dial tcp 10.134.1.50:15021: connect: connection refused" Nov 01 21:25:40.356686 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:40.356651 4484 patch_prober.go:29] interesting pod/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.50:15021/healthz/ready\": dial tcp 10.134.1.50:15021: connect: connection refused" start-of-body= Nov 01 21:25:40.357164 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:40.356703 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n" podUID="acd7c6cc-09ae-4036-b815-64b107627c63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.50:15021/healthz/ready\": dial tcp 10.134.1.50:15021: connect: connection refused" Nov 01 21:25:40.648764 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:40.648680 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n" Nov 01 21:25:40.986708 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:40.986626 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:25:41.360484 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:41.360450 4484 patch_prober.go:29] interesting pod/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.50:15021/healthz/ready\": dial tcp 10.134.1.50:15021: connect: connection refused" start-of-body= Nov 01 21:25:41.361133 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:41.361098 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n" podUID="acd7c6cc-09ae-4036-b815-64b107627c63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.50:15021/healthz/ready\": dial tcp 10.134.1.50:15021: connect: connection refused" Nov 01 21:25:42.356557 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:42.356517 4484 patch_prober.go:29] interesting pod/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.50:15021/healthz/ready\": dial tcp 10.134.1.50:15021: connect: connection refused" start-of-body= Nov 01 21:25:42.356751 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:42.356577 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n" podUID="acd7c6cc-09ae-4036-b815-64b107627c63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.50:15021/healthz/ready\": dial tcp 10.134.1.50:15021: connect: connection refused" Nov 01 21:25:43.360363 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:43.360331 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n" Nov 01 21:25:43.361577 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:43.361557 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n" Nov 01 21:25:43.420731 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:43.420636 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:25:45.775233 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:25:45.775192 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:25:46.045197 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:46.045120 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-c5lpf"] Nov 01 21:25:46.095692 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:46.095657 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-c5lpf"] Nov 01 21:25:47.370197 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:47.370155 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d6d00d0-eb9a-474a-a942-d6186f886d85" path="/var/lib/kubelet/pods/0d6d00d0-eb9a-474a-a942-d6186f886d85/volumes" Nov 01 21:25:48.236095 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:48.236062 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:25:51.740665 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:51.740633 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:25:53.404188 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:53.404151 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:25:57.732130 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:25:57.732091 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:26:02.397055 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:02.397019 4484 scope.go:122] "RemoveContainer" containerID="3c0c6fbefbae3a70b1d3556759efc1a653a8bb8f8c6f200545c7db438821ce73" Nov 01 21:26:02.406678 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:02.406653 4484 scope.go:122] "RemoveContainer" containerID="3a2f970ae1f6e8163d4ab731b32b57801226321d5ac90b8c79fb4f98cec85bbe" Nov 01 21:26:02.415373 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:02.415346 4484 scope.go:122] "RemoveContainer" containerID="32a3ffb0489e72b796f5b84776741766d6037d34284ee9d8a9fc573eef46e36a" Nov 01 21:26:02.444329 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:02.444301 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:26:07.273018 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:07.272956 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:26:12.045771 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:12.045732 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:26:15.044584 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:15.044545 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7cbqg"] Nov 01 21:26:15.095287 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:15.095250 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7cbqg"] Nov 01 21:26:15.369904 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:15.369826 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9f59ba2-6a67-404f-853e-6009607c8f0c" path="/var/lib/kubelet/pods/b9f59ba2-6a67-404f-853e-6009607c8f0c/volumes" Nov 01 21:26:15.765882 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:26:15.765844 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:26:16.958313 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:16.958274 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:26:21.659979 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:21.659941 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:26:25.396741 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:25.396705 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd00b00d-bbfdh"] Nov 01 21:26:25.400315 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:25.400297 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd00b00d-bbfdh" Nov 01 21:26:25.453828 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:25.453791 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:26:25.504692 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:25.504659 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:26:25.504692 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:25.504659 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:26:25.504913 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:25.504795 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:26:25.506906 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:25.506885 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd00b00d-bbfdh"] Nov 01 21:26:25.545982 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:25.545942 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ded6c12-5f9a-4279-ae09-014bde385042-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd00b00d-bbfdh\" (UID: \"7ded6c12-5f9a-4279-ae09-014bde385042\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd00b00d-bbfdh" Nov 01 21:26:25.546194 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:25.546062 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cbhq\" (UniqueName: \"kubernetes.io/projected/7ded6c12-5f9a-4279-ae09-014bde385042-kube-api-access-6cbhq\") pod \"maas-api-revoke-sub-dd00b00d-bbfdh\" (UID: \"7ded6c12-5f9a-4279-ae09-014bde385042\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd00b00d-bbfdh" Nov 01 21:26:25.546194 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:25.546084 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ded6c12-5f9a-4279-ae09-014bde385042-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd00b00d-bbfdh\" (UID: \"7ded6c12-5f9a-4279-ae09-014bde385042\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd00b00d-bbfdh" Nov 01 21:26:25.647225 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:25.647126 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ded6c12-5f9a-4279-ae09-014bde385042-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd00b00d-bbfdh\" (UID: \"7ded6c12-5f9a-4279-ae09-014bde385042\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd00b00d-bbfdh" Nov 01 21:26:25.647225 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:25.647202 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cbhq\" (UniqueName: \"kubernetes.io/projected/7ded6c12-5f9a-4279-ae09-014bde385042-kube-api-access-6cbhq\") pod \"maas-api-revoke-sub-dd00b00d-bbfdh\" (UID: \"7ded6c12-5f9a-4279-ae09-014bde385042\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd00b00d-bbfdh" Nov 01 21:26:25.647447 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:25.647233 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ded6c12-5f9a-4279-ae09-014bde385042-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd00b00d-bbfdh\" (UID: \"7ded6c12-5f9a-4279-ae09-014bde385042\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd00b00d-bbfdh" Nov 01 21:26:25.647939 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:25.647907 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ded6c12-5f9a-4279-ae09-014bde385042-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd00b00d-bbfdh\" (UID: \"7ded6c12-5f9a-4279-ae09-014bde385042\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd00b00d-bbfdh" Nov 01 21:26:25.649572 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:25.649542 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ded6c12-5f9a-4279-ae09-014bde385042-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd00b00d-bbfdh\" (UID: \"7ded6c12-5f9a-4279-ae09-014bde385042\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd00b00d-bbfdh" Nov 01 21:26:25.705009 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:25.704968 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cbhq\" (UniqueName: \"kubernetes.io/projected/7ded6c12-5f9a-4279-ae09-014bde385042-kube-api-access-6cbhq\") pod \"maas-api-revoke-sub-dd00b00d-bbfdh\" (UID: \"7ded6c12-5f9a-4279-ae09-014bde385042\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd00b00d-bbfdh" Nov 01 21:26:25.712702 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:25.712674 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd00b00d-bbfdh" Nov 01 21:26:26.083939 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:26.083914 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd00b00d-bbfdh"] Nov 01 21:26:26.085567 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:26:26.085542 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ded6c12_5f9a_4279_ae09_014bde385042.slice/crio-1aaa47ae514747abea2a147c82c159b101ee2aff95ca206987c9a460afd3d17c WatchSource:0}: Error finding container 1aaa47ae514747abea2a147c82c159b101ee2aff95ca206987c9a460afd3d17c: Status 404 returned error can't find the container with id 1aaa47ae514747abea2a147c82c159b101ee2aff95ca206987c9a460afd3d17c Nov 01 21:26:26.454454 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:26.454234 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:26:27.098283 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:27.098245 4484 generic.go:356] "Generic (PLEG): container finished" podID="7ded6c12-5f9a-4279-ae09-014bde385042" containerID="f218ac21528d52f316b51affa0f892d1754d77df85ef052b7c5157c876742fb6" exitCode=0 Nov 01 21:26:27.098465 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:27.098330 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd00b00d-bbfdh" event={"ID":"7ded6c12-5f9a-4279-ae09-014bde385042","Type":"ContainerDied","Data":"f218ac21528d52f316b51affa0f892d1754d77df85ef052b7c5157c876742fb6"} Nov 01 21:26:27.098465 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:27.098382 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd00b00d-bbfdh" event={"ID":"7ded6c12-5f9a-4279-ae09-014bde385042","Type":"ContainerStarted","Data":"1aaa47ae514747abea2a147c82c159b101ee2aff95ca206987c9a460afd3d17c"} Nov 01 21:26:28.238015 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:28.237967 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd00b00d-bbfdh" Nov 01 21:26:28.376336 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:28.376237 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ded6c12-5f9a-4279-ae09-014bde385042-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ded6c12-5f9a-4279-ae09-014bde385042-maas-api-xks-ca\") pod \"7ded6c12-5f9a-4279-ae09-014bde385042\" (UID: \"7ded6c12-5f9a-4279-ae09-014bde385042\") " Nov 01 21:26:28.376336 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:28.376312 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ded6c12-5f9a-4279-ae09-014bde385042-kube-api-access-6cbhq\" (UniqueName: \"kubernetes.io/projected/7ded6c12-5f9a-4279-ae09-014bde385042-kube-api-access-6cbhq\") pod \"7ded6c12-5f9a-4279-ae09-014bde385042\" (UID: \"7ded6c12-5f9a-4279-ae09-014bde385042\") " Nov 01 21:26:28.376597 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:28.376427 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ded6c12-5f9a-4279-ae09-014bde385042-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ded6c12-5f9a-4279-ae09-014bde385042-maas-api-openshift-ca\") pod \"7ded6c12-5f9a-4279-ae09-014bde385042\" (UID: \"7ded6c12-5f9a-4279-ae09-014bde385042\") " Nov 01 21:26:28.376772 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:28.376752 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ded6c12-5f9a-4279-ae09-014bde385042-maas-api-openshift-ca" pod "7ded6c12-5f9a-4279-ae09-014bde385042" (UID: "7ded6c12-5f9a-4279-ae09-014bde385042"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:26:28.378563 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:28.378534 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ded6c12-5f9a-4279-ae09-014bde385042-maas-api-xks-ca" pod "7ded6c12-5f9a-4279-ae09-014bde385042" (UID: "7ded6c12-5f9a-4279-ae09-014bde385042"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:26:28.378698 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:28.378613 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ded6c12-5f9a-4279-ae09-014bde385042-kube-api-access-6cbhq" pod "7ded6c12-5f9a-4279-ae09-014bde385042" (UID: "7ded6c12-5f9a-4279-ae09-014bde385042"). InnerVolumeSpecName "kube-api-access-6cbhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:26:28.477391 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:28.477338 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ded6c12-5f9a-4279-ae09-014bde385042-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:26:28.477391 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:28.477384 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ded6c12-5f9a-4279-ae09-014bde385042-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:26:28.477391 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:28.477397 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cbhq\" (UniqueName: \"kubernetes.io/projected/7ded6c12-5f9a-4279-ae09-014bde385042-kube-api-access-6cbhq\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:26:29.110057 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:29.110028 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd00b00d-bbfdh" Nov 01 21:26:29.110237 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:29.110055 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd00b00d-bbfdh" event={"ID":"7ded6c12-5f9a-4279-ae09-014bde385042","Type":"ContainerDied","Data":"1aaa47ae514747abea2a147c82c159b101ee2aff95ca206987c9a460afd3d17c"} Nov 01 21:26:29.110237 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:29.110117 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1aaa47ae514747abea2a147c82c159b101ee2aff95ca206987c9a460afd3d17c" Nov 01 21:26:31.236785 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:31.236735 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:26:33.255652 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:33.255619 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:26:33.407128 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:33.407101 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:26:36.043670 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:36.043633 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:26:37.150172 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:37.150138 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:26:37.591811 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:37.591767 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7249f4-xq5hx"] Nov 01 21:26:37.592296 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:37.592277 4484 cpu_manager.go:414] "containerMap: removing container" podUID="7ded6c12-5f9a-4279-ae09-014bde385042" containerName="revoke-keys" Nov 01 21:26:37.592414 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:37.592296 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ded6c12-5f9a-4279-ae09-014bde385042" containerName="revoke-keys" Nov 01 21:26:37.592414 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:37.592404 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ded6c12-5f9a-4279-ae09-014bde385042" containerName="revoke-keys" Nov 01 21:26:37.595815 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:37.595797 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7249f4-xq5hx" Nov 01 21:26:37.652448 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:37.652414 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7249f4-xq5hx"] Nov 01 21:26:37.700804 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:37.700772 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:26:37.700804 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:37.700772 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:26:37.700804 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:37.700773 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:26:37.701094 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:37.700773 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:26:37.771904 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:37.771866 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a7273a2-100d-4424-8c9b-406b797a38bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7249f4-xq5hx\" (UID: \"5a7273a2-100d-4424-8c9b-406b797a38bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7249f4-xq5hx" Nov 01 21:26:37.772103 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:37.771989 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjvcf\" (UniqueName: \"kubernetes.io/projected/5a7273a2-100d-4424-8c9b-406b797a38bb-kube-api-access-jjvcf\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7249f4-xq5hx\" (UID: \"5a7273a2-100d-4424-8c9b-406b797a38bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7249f4-xq5hx" Nov 01 21:26:37.772185 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:37.772100 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a7273a2-100d-4424-8c9b-406b797a38bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7249f4-xq5hx\" (UID: \"5a7273a2-100d-4424-8c9b-406b797a38bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7249f4-xq5hx" Nov 01 21:26:37.873549 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:37.873451 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a7273a2-100d-4424-8c9b-406b797a38bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7249f4-xq5hx\" (UID: \"5a7273a2-100d-4424-8c9b-406b797a38bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7249f4-xq5hx" Nov 01 21:26:37.873549 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:37.873548 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjvcf\" (UniqueName: \"kubernetes.io/projected/5a7273a2-100d-4424-8c9b-406b797a38bb-kube-api-access-jjvcf\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7249f4-xq5hx\" (UID: \"5a7273a2-100d-4424-8c9b-406b797a38bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7249f4-xq5hx" Nov 01 21:26:37.873803 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:37.873593 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a7273a2-100d-4424-8c9b-406b797a38bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7249f4-xq5hx\" (UID: \"5a7273a2-100d-4424-8c9b-406b797a38bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7249f4-xq5hx" Nov 01 21:26:37.874459 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:37.874433 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a7273a2-100d-4424-8c9b-406b797a38bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7249f4-xq5hx\" (UID: \"5a7273a2-100d-4424-8c9b-406b797a38bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7249f4-xq5hx" Nov 01 21:26:37.875977 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:37.875948 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a7273a2-100d-4424-8c9b-406b797a38bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7249f4-xq5hx\" (UID: \"5a7273a2-100d-4424-8c9b-406b797a38bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7249f4-xq5hx" Nov 01 21:26:37.930753 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:37.930715 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjvcf\" (UniqueName: \"kubernetes.io/projected/5a7273a2-100d-4424-8c9b-406b797a38bb-kube-api-access-jjvcf\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7249f4-xq5hx\" (UID: \"5a7273a2-100d-4424-8c9b-406b797a38bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7249f4-xq5hx" Nov 01 21:26:38.208684 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:38.208598 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7249f4-xq5hx" Nov 01 21:26:38.579267 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:38.579238 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7249f4-xq5hx"] Nov 01 21:26:38.580410 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:26:38.580385 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a7273a2_100d_4424_8c9b_406b797a38bb.slice/crio-55a30d01856714b83ab6815275ed825c5a0a9e29bbf68248b0f438d9ae211e02 WatchSource:0}: Error finding container 55a30d01856714b83ab6815275ed825c5a0a9e29bbf68248b0f438d9ae211e02: Status 404 returned error can't find the container with id 55a30d01856714b83ab6815275ed825c5a0a9e29bbf68248b0f438d9ae211e02 Nov 01 21:26:39.162628 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:39.162582 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7249f4-xq5hx" event={"ID":"5a7273a2-100d-4424-8c9b-406b797a38bb","Type":"ContainerStarted","Data":"d2374531fe047fe69f08f32dcdb5d41fe6ac042c3da02fb96f7d3ea239aa3908"} Nov 01 21:26:39.162790 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:39.162655 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7249f4-xq5hx" event={"ID":"5a7273a2-100d-4424-8c9b-406b797a38bb","Type":"ContainerStarted","Data":"55a30d01856714b83ab6815275ed825c5a0a9e29bbf68248b0f438d9ae211e02"} Nov 01 21:26:39.277034 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:39.276963 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7249f4-xq5hx" podStartSLOduration=2.276948997 podStartE2EDuration="2.276948997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:26:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:26:39.276376626 +0000 UTC m=+3350.054016761" watchObservedRunningTime="2026-11-01 21:26:39.276948997 +0000 UTC m=+3350.054589153" Nov 01 21:26:40.168907 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:40.168866 4484 generic.go:356] "Generic (PLEG): container finished" podID="5a7273a2-100d-4424-8c9b-406b797a38bb" containerID="d2374531fe047fe69f08f32dcdb5d41fe6ac042c3da02fb96f7d3ea239aa3908" exitCode=0 Nov 01 21:26:40.169138 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:40.168951 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7249f4-xq5hx" event={"ID":"5a7273a2-100d-4424-8c9b-406b797a38bb","Type":"ContainerDied","Data":"d2374531fe047fe69f08f32dcdb5d41fe6ac042c3da02fb96f7d3ea239aa3908"} Nov 01 21:26:41.123234 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:41.123137 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:26:41.392150 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:41.392076 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7249f4-xq5hx" Nov 01 21:26:41.443887 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:41.443854 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a7273a2-100d-4424-8c9b-406b797a38bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a7273a2-100d-4424-8c9b-406b797a38bb-maas-api-xks-ca\") pod \"5a7273a2-100d-4424-8c9b-406b797a38bb\" (UID: \"5a7273a2-100d-4424-8c9b-406b797a38bb\") " Nov 01 21:26:41.444125 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:41.443943 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a7273a2-100d-4424-8c9b-406b797a38bb-kube-api-access-jjvcf\" (UniqueName: \"kubernetes.io/projected/5a7273a2-100d-4424-8c9b-406b797a38bb-kube-api-access-jjvcf\") pod \"5a7273a2-100d-4424-8c9b-406b797a38bb\" (UID: \"5a7273a2-100d-4424-8c9b-406b797a38bb\") " Nov 01 21:26:41.444125 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:41.443978 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a7273a2-100d-4424-8c9b-406b797a38bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a7273a2-100d-4424-8c9b-406b797a38bb-maas-api-openshift-ca\") pod \"5a7273a2-100d-4424-8c9b-406b797a38bb\" (UID: \"5a7273a2-100d-4424-8c9b-406b797a38bb\") " Nov 01 21:26:41.444450 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:41.444415 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a7273a2-100d-4424-8c9b-406b797a38bb-maas-api-openshift-ca" pod "5a7273a2-100d-4424-8c9b-406b797a38bb" (UID: "5a7273a2-100d-4424-8c9b-406b797a38bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:26:41.446261 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:41.446232 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a7273a2-100d-4424-8c9b-406b797a38bb-maas-api-xks-ca" pod "5a7273a2-100d-4424-8c9b-406b797a38bb" (UID: "5a7273a2-100d-4424-8c9b-406b797a38bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:26:41.446512 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:41.446490 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a7273a2-100d-4424-8c9b-406b797a38bb-kube-api-access-jjvcf" pod "5a7273a2-100d-4424-8c9b-406b797a38bb" (UID: "5a7273a2-100d-4424-8c9b-406b797a38bb"). InnerVolumeSpecName "kube-api-access-jjvcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:26:41.545459 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:41.545416 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a7273a2-100d-4424-8c9b-406b797a38bb-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:26:41.545459 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:41.545459 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjvcf\" (UniqueName: \"kubernetes.io/projected/5a7273a2-100d-4424-8c9b-406b797a38bb-kube-api-access-jjvcf\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:26:41.545680 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:41.545476 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a7273a2-100d-4424-8c9b-406b797a38bb-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:26:42.184895 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:42.184853 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:26:42.190724 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:42.190695 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7249f4-xq5hx" event={"ID":"5a7273a2-100d-4424-8c9b-406b797a38bb","Type":"ContainerDied","Data":"55a30d01856714b83ab6815275ed825c5a0a9e29bbf68248b0f438d9ae211e02"} Nov 01 21:26:42.190885 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:42.190731 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7249f4-xq5hx" Nov 01 21:26:42.190885 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:42.190745 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55a30d01856714b83ab6815275ed825c5a0a9e29bbf68248b0f438d9ae211e02" Nov 01 21:26:45.046024 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:45.045973 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bfs4p"] Nov 01 21:26:45.098060 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:45.098021 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bfs4p"] Nov 01 21:26:45.370470 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:45.370394 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a646bf7-f90c-4ea9-8056-9968f5f4b957" path="/var/lib/kubelet/pods/1a646bf7-f90c-4ea9-8056-9968f5f4b957/volumes" Nov 01 21:26:45.766046 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:26:45.766006 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:26:46.059541 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:46.059448 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:26:49.203532 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:49.203501 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7249f4-544589757t"] Nov 01 21:26:49.481282 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:49.481205 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:26:49.631112 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:49.631049 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:26:50.882194 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:50.882157 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:26:52.082944 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:52.082911 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:26:54.204384 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:54.204338 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7249f4-544589757t" podUID="3a8b983c-41b2-4020-bf69-f7641dbcca4d" containerName="payload-pre-processing" containerID="cri-o://6ebd9f85ef6b8e9626790d475a85a6876cc481f6b85dc1c49bb2451f0fc2c2b8" gracePeriod=25 Nov 01 21:26:55.025420 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:55.025379 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:26:56.603253 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:56.603215 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:26:56.865785 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:56.865702 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:26:57.234638 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:57.234526 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7249f4-xq5hx"] Nov 01 21:26:57.285659 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:57.285621 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7249f4-xq5hx"] Nov 01 21:26:57.370458 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:57.370427 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a7273a2-100d-4424-8c9b-406b797a38bb" path="/var/lib/kubelet/pods/5a7273a2-100d-4424-8c9b-406b797a38bb/volumes" Nov 01 21:26:57.597736 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:57.597687 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:26:58.693968 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:26:58.693925 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7249f4-544589757t" podUID="3a8b983c-41b2-4020-bf69-f7641dbcca4d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 21:27:02.444651 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:02.444576 4484 scope.go:122] "RemoveContainer" containerID="43509f35fba6724b6922ccd12f9f8df646362f41df75bb8b46ac6ceb88824629" Nov 01 21:27:02.453596 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:02.453570 4484 scope.go:122] "RemoveContainer" containerID="2a72172308db3f7ffa01deb33fce07c9c352d34fa30ffe7a0a760b0f7b404de9" Nov 01 21:27:03.400244 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:03.400210 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:27:03.561225 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:03.561192 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:27:03.695030 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:03.694927 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n"] Nov 01 21:27:03.695204 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:03.695164 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n" podUID="acd7c6cc-09ae-4036-b815-64b107627c63" containerName="istio-proxy" containerID="cri-o://4efbb9775c502815b64a18674c1bbcbc4ccbdee756fa60673d3c4dd2b4ce8f2b" gracePeriod=30 Nov 01 21:27:03.723902 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:03.723867 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:27:05.172335 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:05.172297 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kpr7v"] Nov 01 21:27:05.223243 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:05.223205 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kpr7v"] Nov 01 21:27:05.292607 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:05.292574 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g49bd"] Nov 01 21:27:05.293026 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:05.293013 4484 cpu_manager.go:414] "containerMap: removing container" podUID="5a7273a2-100d-4424-8c9b-406b797a38bb" containerName="revoke-keys" Nov 01 21:27:05.293089 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:05.293027 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a7273a2-100d-4424-8c9b-406b797a38bb" containerName="revoke-keys" Nov 01 21:27:05.293134 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:05.293093 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a7273a2-100d-4424-8c9b-406b797a38bb" containerName="revoke-keys" Nov 01 21:27:05.297794 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:05.297773 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g49bd" Nov 01 21:27:05.351338 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:05.351306 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:27:05.351540 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:05.351460 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:27:05.351613 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:05.351541 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:27:05.352041 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:05.351737 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:27:05.352041 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:05.352005 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g49bd"] Nov 01 21:27:05.371877 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:05.371842 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3c4a858-de7c-47af-9718-ef53f733df59" path="/var/lib/kubelet/pods/c3c4a858-de7c-47af-9718-ef53f733df59/volumes" Nov 01 21:27:05.419811 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:05.419778 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96462\" (UniqueName: \"kubernetes.io/projected/c88c6fad-5b4f-491c-aaa5-27d33b283112-kube-api-access-96462\") pod \"maas-api-revoke-sub-3d4156f1-g49bd\" (UID: \"c88c6fad-5b4f-491c-aaa5-27d33b283112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g49bd" Nov 01 21:27:05.420042 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:05.419843 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c88c6fad-5b4f-491c-aaa5-27d33b283112-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-g49bd\" (UID: \"c88c6fad-5b4f-491c-aaa5-27d33b283112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g49bd" Nov 01 21:27:05.420042 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:05.419933 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c88c6fad-5b4f-491c-aaa5-27d33b283112-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-g49bd\" (UID: \"c88c6fad-5b4f-491c-aaa5-27d33b283112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g49bd" Nov 01 21:27:05.521598 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:05.521560 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96462\" (UniqueName: \"kubernetes.io/projected/c88c6fad-5b4f-491c-aaa5-27d33b283112-kube-api-access-96462\") pod \"maas-api-revoke-sub-3d4156f1-g49bd\" (UID: \"c88c6fad-5b4f-491c-aaa5-27d33b283112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g49bd" Nov 01 21:27:05.521807 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:05.521625 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c88c6fad-5b4f-491c-aaa5-27d33b283112-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-g49bd\" (UID: \"c88c6fad-5b4f-491c-aaa5-27d33b283112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g49bd" Nov 01 21:27:05.521807 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:05.521656 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c88c6fad-5b4f-491c-aaa5-27d33b283112-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-g49bd\" (UID: \"c88c6fad-5b4f-491c-aaa5-27d33b283112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g49bd" Nov 01 21:27:05.522309 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:05.522286 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c88c6fad-5b4f-491c-aaa5-27d33b283112-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-g49bd\" (UID: \"c88c6fad-5b4f-491c-aaa5-27d33b283112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g49bd" Nov 01 21:27:05.523984 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:05.523959 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c88c6fad-5b4f-491c-aaa5-27d33b283112-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-g49bd\" (UID: \"c88c6fad-5b4f-491c-aaa5-27d33b283112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g49bd" Nov 01 21:27:05.579399 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:05.579368 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96462\" (UniqueName: \"kubernetes.io/projected/c88c6fad-5b4f-491c-aaa5-27d33b283112-kube-api-access-96462\") pod \"maas-api-revoke-sub-3d4156f1-g49bd\" (UID: \"c88c6fad-5b4f-491c-aaa5-27d33b283112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g49bd" Nov 01 21:27:05.611072 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:05.611038 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g49bd" Nov 01 21:27:05.987883 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:05.987853 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g49bd"] Nov 01 21:27:05.989259 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:27:05.989227 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc88c6fad_5b4f_491c_aaa5_27d33b283112.slice/crio-d53a8ae3b2c84c97841884ab6b923c8768123d3730ce78b94d102ef18c51dbea WatchSource:0}: Error finding container d53a8ae3b2c84c97841884ab6b923c8768123d3730ce78b94d102ef18c51dbea: Status 404 returned error can't find the container with id d53a8ae3b2c84c97841884ab6b923c8768123d3730ce78b94d102ef18c51dbea Nov 01 21:27:06.327669 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:06.327636 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g49bd" event={"ID":"c88c6fad-5b4f-491c-aaa5-27d33b283112","Type":"ContainerStarted","Data":"8093b107fa282245d9d037392ff0a5603d8d9493186b9320fbccf9af6ce25e4a"} Nov 01 21:27:06.328080 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:06.327688 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g49bd" event={"ID":"c88c6fad-5b4f-491c-aaa5-27d33b283112","Type":"ContainerStarted","Data":"d53a8ae3b2c84c97841884ab6b923c8768123d3730ce78b94d102ef18c51dbea"} Nov 01 21:27:06.443914 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:06.443860 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g49bd" podStartSLOduration=1.443844849 podStartE2EDuration="1.443844849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:27:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:27:06.441395534 +0000 UTC m=+3377.219035668" watchObservedRunningTime="2026-11-01 21:27:06.443844849 +0000 UTC m=+3377.221484983" Nov 01 21:27:07.337040 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:07.336983 4484 generic.go:356] "Generic (PLEG): container finished" podID="c88c6fad-5b4f-491c-aaa5-27d33b283112" containerID="8093b107fa282245d9d037392ff0a5603d8d9493186b9320fbccf9af6ce25e4a" exitCode=0 Nov 01 21:27:07.337450 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:07.337072 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g49bd" event={"ID":"c88c6fad-5b4f-491c-aaa5-27d33b283112","Type":"ContainerDied","Data":"8093b107fa282245d9d037392ff0a5603d8d9493186b9320fbccf9af6ce25e4a"} Nov 01 21:27:08.044581 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:08.044543 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zkx4k"] Nov 01 21:27:08.094955 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:08.094906 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zkx4k"] Nov 01 21:27:08.481150 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:08.481124 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g49bd" Nov 01 21:27:08.650073 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:08.649982 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c88c6fad-5b4f-491c-aaa5-27d33b283112-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c88c6fad-5b4f-491c-aaa5-27d33b283112-maas-api-xks-ca\") pod \"c88c6fad-5b4f-491c-aaa5-27d33b283112\" (UID: \"c88c6fad-5b4f-491c-aaa5-27d33b283112\") " Nov 01 21:27:08.650073 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:08.650033 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c88c6fad-5b4f-491c-aaa5-27d33b283112-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c88c6fad-5b4f-491c-aaa5-27d33b283112-maas-api-openshift-ca\") pod \"c88c6fad-5b4f-491c-aaa5-27d33b283112\" (UID: \"c88c6fad-5b4f-491c-aaa5-27d33b283112\") " Nov 01 21:27:08.650286 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:08.650261 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c88c6fad-5b4f-491c-aaa5-27d33b283112-kube-api-access-96462\" (UniqueName: \"kubernetes.io/projected/c88c6fad-5b4f-491c-aaa5-27d33b283112-kube-api-access-96462\") pod \"c88c6fad-5b4f-491c-aaa5-27d33b283112\" (UID: \"c88c6fad-5b4f-491c-aaa5-27d33b283112\") " Nov 01 21:27:08.650424 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:08.650402 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c88c6fad-5b4f-491c-aaa5-27d33b283112-maas-api-openshift-ca" pod "c88c6fad-5b4f-491c-aaa5-27d33b283112" (UID: "c88c6fad-5b4f-491c-aaa5-27d33b283112"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:27:08.650589 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:08.650570 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c88c6fad-5b4f-491c-aaa5-27d33b283112-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:27:08.652256 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:08.652224 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c88c6fad-5b4f-491c-aaa5-27d33b283112-maas-api-xks-ca" pod "c88c6fad-5b4f-491c-aaa5-27d33b283112" (UID: "c88c6fad-5b4f-491c-aaa5-27d33b283112"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:27:08.652364 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:08.652293 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c88c6fad-5b4f-491c-aaa5-27d33b283112-kube-api-access-96462" pod "c88c6fad-5b4f-491c-aaa5-27d33b283112" (UID: "c88c6fad-5b4f-491c-aaa5-27d33b283112"). InnerVolumeSpecName "kube-api-access-96462". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:27:08.693924 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:08.693884 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7249f4-544589757t" podUID="3a8b983c-41b2-4020-bf69-f7641dbcca4d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 21:27:08.751294 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:08.751258 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96462\" (UniqueName: \"kubernetes.io/projected/c88c6fad-5b4f-491c-aaa5-27d33b283112-kube-api-access-96462\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:27:08.751294 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:08.751288 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c88c6fad-5b4f-491c-aaa5-27d33b283112-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:27:08.932727 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:08.932700 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n" Nov 01 21:27:09.054777 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:09.054743 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acd7c6cc-09ae-4036-b815-64b107627c63-kube-api-access-zmq97\" (UniqueName: \"kubernetes.io/projected/acd7c6cc-09ae-4036-b815-64b107627c63-kube-api-access-zmq97\") pod \"acd7c6cc-09ae-4036-b815-64b107627c63\" (UID: \"acd7c6cc-09ae-4036-b815-64b107627c63\") " Nov 01 21:27:09.054990 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:09.054797 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acd7c6cc-09ae-4036-b815-64b107627c63-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acd7c6cc-09ae-4036-b815-64b107627c63-istiod-ca-cert\") pod \"acd7c6cc-09ae-4036-b815-64b107627c63\" (UID: \"acd7c6cc-09ae-4036-b815-64b107627c63\") " Nov 01 21:27:09.054990 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:09.054843 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acd7c6cc-09ae-4036-b815-64b107627c63-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acd7c6cc-09ae-4036-b815-64b107627c63-istio-data\") pod \"acd7c6cc-09ae-4036-b815-64b107627c63\" (UID: \"acd7c6cc-09ae-4036-b815-64b107627c63\") " Nov 01 21:27:09.054990 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:09.054892 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acd7c6cc-09ae-4036-b815-64b107627c63-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acd7c6cc-09ae-4036-b815-64b107627c63-workload-socket\") pod \"acd7c6cc-09ae-4036-b815-64b107627c63\" (UID: \"acd7c6cc-09ae-4036-b815-64b107627c63\") " Nov 01 21:27:09.054990 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:09.054930 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acd7c6cc-09ae-4036-b815-64b107627c63-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acd7c6cc-09ae-4036-b815-64b107627c63-workload-certs\") pod \"acd7c6cc-09ae-4036-b815-64b107627c63\" (UID: \"acd7c6cc-09ae-4036-b815-64b107627c63\") " Nov 01 21:27:09.055288 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:09.055013 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/acd7c6cc-09ae-4036-b815-64b107627c63-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acd7c6cc-09ae-4036-b815-64b107627c63-istio-podinfo\") pod \"acd7c6cc-09ae-4036-b815-64b107627c63\" (UID: \"acd7c6cc-09ae-4036-b815-64b107627c63\") " Nov 01 21:27:09.055288 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:09.055065 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acd7c6cc-09ae-4036-b815-64b107627c63-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acd7c6cc-09ae-4036-b815-64b107627c63-istio-envoy\") pod \"acd7c6cc-09ae-4036-b815-64b107627c63\" (UID: \"acd7c6cc-09ae-4036-b815-64b107627c63\") " Nov 01 21:27:09.055288 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:09.055121 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acd7c6cc-09ae-4036-b815-64b107627c63-istio-token\" (UniqueName: \"kubernetes.io/projected/acd7c6cc-09ae-4036-b815-64b107627c63-istio-token\") pod \"acd7c6cc-09ae-4036-b815-64b107627c63\" (UID: \"acd7c6cc-09ae-4036-b815-64b107627c63\") " Nov 01 21:27:09.055288 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:09.055173 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acd7c6cc-09ae-4036-b815-64b107627c63-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acd7c6cc-09ae-4036-b815-64b107627c63-credential-socket\") pod \"acd7c6cc-09ae-4036-b815-64b107627c63\" (UID: \"acd7c6cc-09ae-4036-b815-64b107627c63\") " Nov 01 21:27:09.055288 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:09.055235 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acd7c6cc-09ae-4036-b815-64b107627c63-istio-data" pod "acd7c6cc-09ae-4036-b815-64b107627c63" (UID: "acd7c6cc-09ae-4036-b815-64b107627c63"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:27:09.055577 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:09.055286 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acd7c6cc-09ae-4036-b815-64b107627c63-workload-socket" pod "acd7c6cc-09ae-4036-b815-64b107627c63" (UID: "acd7c6cc-09ae-4036-b815-64b107627c63"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:27:09.055850 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:09.055825 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acd7c6cc-09ae-4036-b815-64b107627c63-istiod-ca-cert" pod "acd7c6cc-09ae-4036-b815-64b107627c63" (UID: "acd7c6cc-09ae-4036-b815-64b107627c63"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:27:09.055934 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:09.055888 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acd7c6cc-09ae-4036-b815-64b107627c63-credential-socket" pod "acd7c6cc-09ae-4036-b815-64b107627c63" (UID: "acd7c6cc-09ae-4036-b815-64b107627c63"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:27:09.056102 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:09.056082 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acd7c6cc-09ae-4036-b815-64b107627c63-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:27:09.056191 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:09.056108 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acd7c6cc-09ae-4036-b815-64b107627c63-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:27:09.056191 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:09.056124 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acd7c6cc-09ae-4036-b815-64b107627c63-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:27:09.056191 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:09.056138 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acd7c6cc-09ae-4036-b815-64b107627c63-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:27:09.056372 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:09.056201 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acd7c6cc-09ae-4036-b815-64b107627c63-workload-certs" pod "acd7c6cc-09ae-4036-b815-64b107627c63" (UID: "acd7c6cc-09ae-4036-b815-64b107627c63"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:27:09.057683 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:09.057647 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acd7c6cc-09ae-4036-b815-64b107627c63-kube-api-access-zmq97" pod "acd7c6cc-09ae-4036-b815-64b107627c63" (UID: "acd7c6cc-09ae-4036-b815-64b107627c63"). InnerVolumeSpecName "kube-api-access-zmq97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:27:09.058023 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:09.057969 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acd7c6cc-09ae-4036-b815-64b107627c63-istio-token" pod "acd7c6cc-09ae-4036-b815-64b107627c63" (UID: "acd7c6cc-09ae-4036-b815-64b107627c63"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:27:09.058023 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:09.057981 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acd7c6cc-09ae-4036-b815-64b107627c63-istio-envoy" pod "acd7c6cc-09ae-4036-b815-64b107627c63" (UID: "acd7c6cc-09ae-4036-b815-64b107627c63"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:27:09.058247 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:09.058227 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/acd7c6cc-09ae-4036-b815-64b107627c63-istio-podinfo" pod "acd7c6cc-09ae-4036-b815-64b107627c63" (UID: "acd7c6cc-09ae-4036-b815-64b107627c63"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:27:09.156859 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:09.156822 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acd7c6cc-09ae-4036-b815-64b107627c63-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:27:09.156859 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:09.156852 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acd7c6cc-09ae-4036-b815-64b107627c63-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:27:09.156859 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:09.156861 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acd7c6cc-09ae-4036-b815-64b107627c63-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:27:09.157149 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:09.156869 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmq97\" (UniqueName: \"kubernetes.io/projected/acd7c6cc-09ae-4036-b815-64b107627c63-kube-api-access-zmq97\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:27:09.157149 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:09.156880 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acd7c6cc-09ae-4036-b815-64b107627c63-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:27:09.350214 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:09.350179 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g49bd" Nov 01 21:27:09.350214 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:09.350204 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g49bd" event={"ID":"c88c6fad-5b4f-491c-aaa5-27d33b283112","Type":"ContainerDied","Data":"d53a8ae3b2c84c97841884ab6b923c8768123d3730ce78b94d102ef18c51dbea"} Nov 01 21:27:09.350456 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:09.350255 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d53a8ae3b2c84c97841884ab6b923c8768123d3730ce78b94d102ef18c51dbea" Nov 01 21:27:09.352057 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:09.352028 4484 generic.go:356] "Generic (PLEG): container finished" podID="acd7c6cc-09ae-4036-b815-64b107627c63" containerID="4efbb9775c502815b64a18674c1bbcbc4ccbdee756fa60673d3c4dd2b4ce8f2b" exitCode=0 Nov 01 21:27:09.352195 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:09.352091 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n" Nov 01 21:27:09.352195 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:09.352106 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n" event={"ID":"acd7c6cc-09ae-4036-b815-64b107627c63","Type":"ContainerDied","Data":"4efbb9775c502815b64a18674c1bbcbc4ccbdee756fa60673d3c4dd2b4ce8f2b"} Nov 01 21:27:09.352195 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:09.352149 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n" event={"ID":"acd7c6cc-09ae-4036-b815-64b107627c63","Type":"ContainerDied","Data":"ef81e4296bdcaf80e5abc4ee2bf2e563b8f2ae90fff691fada2a0174bb43c137"} Nov 01 21:27:09.352195 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:09.352173 4484 scope.go:122] "RemoveContainer" containerID="4efbb9775c502815b64a18674c1bbcbc4ccbdee756fa60673d3c4dd2b4ce8f2b" Nov 01 21:27:09.361755 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:09.361733 4484 scope.go:122] "RemoveContainer" containerID="a3d5f2005bd09da34404ff2c334541c8c568a7ff9da7bb2a3c3b7abbaceeba39" Nov 01 21:27:09.373190 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:09.373162 4484 scope.go:122] "RemoveContainer" containerID="4efbb9775c502815b64a18674c1bbcbc4ccbdee756fa60673d3c4dd2b4ce8f2b" Nov 01 21:27:09.373607 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:27:09.373579 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4efbb9775c502815b64a18674c1bbcbc4ccbdee756fa60673d3c4dd2b4ce8f2b\": container with ID starting with 4efbb9775c502815b64a18674c1bbcbc4ccbdee756fa60673d3c4dd2b4ce8f2b not found: ID does not exist" containerID="4efbb9775c502815b64a18674c1bbcbc4ccbdee756fa60673d3c4dd2b4ce8f2b" Nov 01 21:27:09.373694 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:09.373619 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4efbb9775c502815b64a18674c1bbcbc4ccbdee756fa60673d3c4dd2b4ce8f2b"} err="failed to get container status \"4efbb9775c502815b64a18674c1bbcbc4ccbdee756fa60673d3c4dd2b4ce8f2b\": rpc error: code = NotFound desc = could not find container \"4efbb9775c502815b64a18674c1bbcbc4ccbdee756fa60673d3c4dd2b4ce8f2b\": container with ID starting with 4efbb9775c502815b64a18674c1bbcbc4ccbdee756fa60673d3c4dd2b4ce8f2b not found: ID does not exist" Nov 01 21:27:09.373694 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:09.373649 4484 scope.go:122] "RemoveContainer" containerID="a3d5f2005bd09da34404ff2c334541c8c568a7ff9da7bb2a3c3b7abbaceeba39" Nov 01 21:27:09.373973 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:27:09.373950 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3d5f2005bd09da34404ff2c334541c8c568a7ff9da7bb2a3c3b7abbaceeba39\": container with ID starting with a3d5f2005bd09da34404ff2c334541c8c568a7ff9da7bb2a3c3b7abbaceeba39 not found: ID does not exist" containerID="a3d5f2005bd09da34404ff2c334541c8c568a7ff9da7bb2a3c3b7abbaceeba39" Nov 01 21:27:09.374064 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:09.373981 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3d5f2005bd09da34404ff2c334541c8c568a7ff9da7bb2a3c3b7abbaceeba39"} err="failed to get container status \"a3d5f2005bd09da34404ff2c334541c8c568a7ff9da7bb2a3c3b7abbaceeba39\": rpc error: code = NotFound desc = could not find container \"a3d5f2005bd09da34404ff2c334541c8c568a7ff9da7bb2a3c3b7abbaceeba39\": container with ID starting with a3d5f2005bd09da34404ff2c334541c8c568a7ff9da7bb2a3c3b7abbaceeba39 not found: ID does not exist" Nov 01 21:27:09.375809 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:09.375783 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8cbbf27-4eda-4002-921e-e1d090b7d983" path="/var/lib/kubelet/pods/b8cbbf27-4eda-4002-921e-e1d090b7d983/volumes" Nov 01 21:27:09.519224 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:09.519187 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n"] Nov 01 21:27:09.520885 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:09.520864 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-7249f4-openshift-default-5d6946dbbf95j6n"] Nov 01 21:27:11.369535 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:11.369502 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acd7c6cc-09ae-4036-b815-64b107627c63" path="/var/lib/kubelet/pods/acd7c6cc-09ae-4036-b815-64b107627c63/volumes" Nov 01 21:27:14.383219 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:14.383186 4484 generic.go:356] "Generic (PLEG): container finished" podID="3a8b983c-41b2-4020-bf69-f7641dbcca4d" containerID="6ebd9f85ef6b8e9626790d475a85a6876cc481f6b85dc1c49bb2451f0fc2c2b8" exitCode=0 Nov 01 21:27:14.383662 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:14.383256 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7249f4-544589757t" event={"ID":"3a8b983c-41b2-4020-bf69-f7641dbcca4d","Type":"ContainerDied","Data":"6ebd9f85ef6b8e9626790d475a85a6876cc481f6b85dc1c49bb2451f0fc2c2b8"} Nov 01 21:27:14.460984 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:14.460959 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7249f4-544589757t" Nov 01 21:27:14.505772 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:14.505738 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a8b983c-41b2-4020-bf69-f7641dbcca4d-kube-api-access-gctf5\" (UniqueName: \"kubernetes.io/projected/3a8b983c-41b2-4020-bf69-f7641dbcca4d-kube-api-access-gctf5\") pod \"3a8b983c-41b2-4020-bf69-f7641dbcca4d\" (UID: \"3a8b983c-41b2-4020-bf69-f7641dbcca4d\") " Nov 01 21:27:14.505954 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:14.505791 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a8b983c-41b2-4020-bf69-f7641dbcca4d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a8b983c-41b2-4020-bf69-f7641dbcca4d-plugins-config-volume\") pod \"3a8b983c-41b2-4020-bf69-f7641dbcca4d\" (UID: \"3a8b983c-41b2-4020-bf69-f7641dbcca4d\") " Nov 01 21:27:14.506218 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:14.506195 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a8b983c-41b2-4020-bf69-f7641dbcca4d-plugins-config-volume" pod "3a8b983c-41b2-4020-bf69-f7641dbcca4d" (UID: "3a8b983c-41b2-4020-bf69-f7641dbcca4d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:27:14.507832 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:14.507813 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a8b983c-41b2-4020-bf69-f7641dbcca4d-kube-api-access-gctf5" pod "3a8b983c-41b2-4020-bf69-f7641dbcca4d" (UID: "3a8b983c-41b2-4020-bf69-f7641dbcca4d"). InnerVolumeSpecName "kube-api-access-gctf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:27:14.607426 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:14.607381 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gctf5\" (UniqueName: \"kubernetes.io/projected/3a8b983c-41b2-4020-bf69-f7641dbcca4d-kube-api-access-gctf5\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:27:14.607426 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:14.607425 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a8b983c-41b2-4020-bf69-f7641dbcca4d-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:27:15.389534 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:15.389501 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7249f4-544589757t" Nov 01 21:27:15.389534 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:15.389513 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7249f4-544589757t" event={"ID":"3a8b983c-41b2-4020-bf69-f7641dbcca4d","Type":"ContainerDied","Data":"0a48f7794215ce5f7df741b599d7e2d874521ab3376148df68f51bbc1899d426"} Nov 01 21:27:15.390016 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:15.389595 4484 scope.go:122] "RemoveContainer" containerID="6ebd9f85ef6b8e9626790d475a85a6876cc481f6b85dc1c49bb2451f0fc2c2b8" Nov 01 21:27:15.554863 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:15.554829 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7249f4-544589757t"] Nov 01 21:27:15.605537 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:15.605494 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7249f4-544589757t"] Nov 01 21:27:15.766694 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:27:15.766648 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:27:17.370442 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:17.370407 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a8b983c-41b2-4020-bf69-f7641dbcca4d" path="/var/lib/kubelet/pods/3a8b983c-41b2-4020-bf69-f7641dbcca4d/volumes" Nov 01 21:27:18.541793 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:18.541761 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:27:19.290446 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:19.290405 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:27:24.781146 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:24.781088 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:27:45.765841 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:27:45.765802 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:27:51.315057 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:51.315018 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:27:52.432474 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:27:52.432440 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:28:02.479936 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:28:02.479900 4484 scope.go:122] "RemoveContainer" containerID="98992f0381ed70d93404ab8cd60ec97bef1da25246966ec9ff749490b155af5f" Nov 01 21:28:02.488987 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:28:02.488964 4484 scope.go:122] "RemoveContainer" containerID="0000aa5a0fdf30882cadbe444029c3cdb56e62ec95f5902f06661a6078c60446" Nov 01 21:28:15.767399 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:28:15.767361 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:28:43.047249 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:28:43.047172 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6str6"] Nov 01 21:28:43.098119 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:28:43.098085 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6str6"] Nov 01 21:28:43.370123 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:28:43.370046 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ff52970-facf-4ef5-b12a-c9cbc3bed639" path="/var/lib/kubelet/pods/0ff52970-facf-4ef5-b12a-c9cbc3bed639/volumes" Nov 01 21:28:45.765951 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:28:45.765916 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:28:52.776398 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:28:52.776362 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:28:54.044504 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:28:54.044464 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j7lwr"] Nov 01 21:28:54.095211 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:28:54.095174 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j7lwr"] Nov 01 21:28:55.371011 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:28:55.370963 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="153c5475-eb19-47b8-bba6-f9ce73945692" path="/var/lib/kubelet/pods/153c5475-eb19-47b8-bba6-f9ce73945692/volumes" Nov 01 21:29:02.519757 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:02.519726 4484 scope.go:122] "RemoveContainer" containerID="b5436ac4e60510cb4a593ddef5a850bd5f8cb85d667f8f96c1d57c805f1c4b8a" Nov 01 21:29:02.528287 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:02.528265 4484 scope.go:122] "RemoveContainer" containerID="2a23753baaaa56a57641399efbee8788319ceac974ad596f11e53aebc43d80be" Nov 01 21:29:02.536353 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:02.536329 4484 scope.go:122] "RemoveContainer" containerID="8b6bafdebdf490aa0ab63270eb60ffa73944f852f4e0800f846eb0bb4cb218e3" Nov 01 21:29:13.025755 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:13.025722 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rxxx4"] Nov 01 21:29:13.026268 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:13.026133 4484 cpu_manager.go:414] "containerMap: removing container" podUID="acd7c6cc-09ae-4036-b815-64b107627c63" containerName="istio-proxy" Nov 01 21:29:13.026268 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:13.026144 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acd7c6cc-09ae-4036-b815-64b107627c63" containerName="istio-proxy" Nov 01 21:29:13.026268 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:13.026155 4484 cpu_manager.go:414] "containerMap: removing container" podUID="c88c6fad-5b4f-491c-aaa5-27d33b283112" containerName="revoke-keys" Nov 01 21:29:13.026268 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:13.026160 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c88c6fad-5b4f-491c-aaa5-27d33b283112" containerName="revoke-keys" Nov 01 21:29:13.026268 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:13.026179 4484 cpu_manager.go:414] "containerMap: removing container" podUID="3a8b983c-41b2-4020-bf69-f7641dbcca4d" containerName="payload-pre-processing" Nov 01 21:29:13.026268 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:13.026185 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a8b983c-41b2-4020-bf69-f7641dbcca4d" containerName="payload-pre-processing" Nov 01 21:29:13.026268 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:13.026256 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="acd7c6cc-09ae-4036-b815-64b107627c63" containerName="istio-proxy" Nov 01 21:29:13.026268 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:13.026263 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="acd7c6cc-09ae-4036-b815-64b107627c63" containerName="istio-proxy" Nov 01 21:29:13.026268 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:13.026271 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="c88c6fad-5b4f-491c-aaa5-27d33b283112" containerName="revoke-keys" Nov 01 21:29:13.026597 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:13.026280 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a8b983c-41b2-4020-bf69-f7641dbcca4d" containerName="payload-pre-processing" Nov 01 21:29:13.031263 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:13.031245 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rxxx4" Nov 01 21:29:13.068220 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:13.068182 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64dada2e-ccc9-4ab8-a9e6-c75d6c0e0611-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-rxxx4\" (UID: \"64dada2e-ccc9-4ab8-a9e6-c75d6c0e0611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rxxx4" Nov 01 21:29:13.068410 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:13.068230 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64dada2e-ccc9-4ab8-a9e6-c75d6c0e0611-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-rxxx4\" (UID: \"64dada2e-ccc9-4ab8-a9e6-c75d6c0e0611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rxxx4" Nov 01 21:29:13.068410 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:13.068305 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkrf6\" (UniqueName: \"kubernetes.io/projected/64dada2e-ccc9-4ab8-a9e6-c75d6c0e0611-kube-api-access-zkrf6\") pod \"maas-api-revoke-sub-d576e68b-rxxx4\" (UID: \"64dada2e-ccc9-4ab8-a9e6-c75d6c0e0611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rxxx4" Nov 01 21:29:13.086059 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:13.086021 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rxxx4"] Nov 01 21:29:13.134479 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:13.134443 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:29:13.134642 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:13.134496 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:29:13.135669 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:13.135639 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:29:13.135793 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:13.135670 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:29:13.169753 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:13.169721 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64dada2e-ccc9-4ab8-a9e6-c75d6c0e0611-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-rxxx4\" (UID: \"64dada2e-ccc9-4ab8-a9e6-c75d6c0e0611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rxxx4" Nov 01 21:29:13.169968 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:13.169764 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkrf6\" (UniqueName: \"kubernetes.io/projected/64dada2e-ccc9-4ab8-a9e6-c75d6c0e0611-kube-api-access-zkrf6\") pod \"maas-api-revoke-sub-d576e68b-rxxx4\" (UID: \"64dada2e-ccc9-4ab8-a9e6-c75d6c0e0611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rxxx4" Nov 01 21:29:13.169968 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:13.169855 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64dada2e-ccc9-4ab8-a9e6-c75d6c0e0611-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-rxxx4\" (UID: \"64dada2e-ccc9-4ab8-a9e6-c75d6c0e0611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rxxx4" Nov 01 21:29:13.170528 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:13.170505 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64dada2e-ccc9-4ab8-a9e6-c75d6c0e0611-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-rxxx4\" (UID: \"64dada2e-ccc9-4ab8-a9e6-c75d6c0e0611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rxxx4" Nov 01 21:29:13.172286 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:13.172263 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64dada2e-ccc9-4ab8-a9e6-c75d6c0e0611-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-rxxx4\" (UID: \"64dada2e-ccc9-4ab8-a9e6-c75d6c0e0611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rxxx4" Nov 01 21:29:13.227438 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:13.227400 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkrf6\" (UniqueName: \"kubernetes.io/projected/64dada2e-ccc9-4ab8-a9e6-c75d6c0e0611-kube-api-access-zkrf6\") pod \"maas-api-revoke-sub-d576e68b-rxxx4\" (UID: \"64dada2e-ccc9-4ab8-a9e6-c75d6c0e0611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rxxx4" Nov 01 21:29:13.344386 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:13.344293 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rxxx4" Nov 01 21:29:13.717943 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:13.717905 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rxxx4"] Nov 01 21:29:13.721786 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:29:13.721751 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64dada2e_ccc9_4ab8_a9e6_c75d6c0e0611.slice/crio-0ba0286aabe98a49312bd8f196c8c0828e2c690b14972bce0ff810074b0647ef WatchSource:0}: Error finding container 0ba0286aabe98a49312bd8f196c8c0828e2c690b14972bce0ff810074b0647ef: Status 404 returned error can't find the container with id 0ba0286aabe98a49312bd8f196c8c0828e2c690b14972bce0ff810074b0647ef Nov 01 21:29:14.067954 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:14.067916 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rxxx4" event={"ID":"64dada2e-ccc9-4ab8-a9e6-c75d6c0e0611","Type":"ContainerStarted","Data":"c2599c03b32ee7fe8ad3568c917edba9a2ace4817c9f3288773628791507a411"} Nov 01 21:29:14.068392 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:14.067977 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rxxx4" event={"ID":"64dada2e-ccc9-4ab8-a9e6-c75d6c0e0611","Type":"ContainerStarted","Data":"0ba0286aabe98a49312bd8f196c8c0828e2c690b14972bce0ff810074b0647ef"} Nov 01 21:29:14.183600 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:14.183544 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rxxx4" podStartSLOduration=2.183528466 podStartE2EDuration="2.183528466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:29:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:29:14.181944204 +0000 UTC m=+3504.959584343" watchObservedRunningTime="2026-11-01 21:29:14.183528466 +0000 UTC m=+3504.961168624" Nov 01 21:29:15.074512 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:15.074478 4484 generic.go:356] "Generic (PLEG): container finished" podID="64dada2e-ccc9-4ab8-a9e6-c75d6c0e0611" containerID="c2599c03b32ee7fe8ad3568c917edba9a2ace4817c9f3288773628791507a411" exitCode=0 Nov 01 21:29:15.074930 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:15.074572 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rxxx4" event={"ID":"64dada2e-ccc9-4ab8-a9e6-c75d6c0e0611","Type":"ContainerDied","Data":"c2599c03b32ee7fe8ad3568c917edba9a2ace4817c9f3288773628791507a411"} Nov 01 21:29:15.767053 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:29:15.767020 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:29:16.224304 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:16.224274 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rxxx4" Nov 01 21:29:16.298475 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:16.298437 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64dada2e-ccc9-4ab8-a9e6-c75d6c0e0611-kube-api-access-zkrf6\" (UniqueName: \"kubernetes.io/projected/64dada2e-ccc9-4ab8-a9e6-c75d6c0e0611-kube-api-access-zkrf6\") pod \"64dada2e-ccc9-4ab8-a9e6-c75d6c0e0611\" (UID: \"64dada2e-ccc9-4ab8-a9e6-c75d6c0e0611\") " Nov 01 21:29:16.298692 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:16.298513 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64dada2e-ccc9-4ab8-a9e6-c75d6c0e0611-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64dada2e-ccc9-4ab8-a9e6-c75d6c0e0611-maas-api-xks-ca\") pod \"64dada2e-ccc9-4ab8-a9e6-c75d6c0e0611\" (UID: \"64dada2e-ccc9-4ab8-a9e6-c75d6c0e0611\") " Nov 01 21:29:16.298692 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:16.298624 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64dada2e-ccc9-4ab8-a9e6-c75d6c0e0611-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64dada2e-ccc9-4ab8-a9e6-c75d6c0e0611-maas-api-openshift-ca\") pod \"64dada2e-ccc9-4ab8-a9e6-c75d6c0e0611\" (UID: \"64dada2e-ccc9-4ab8-a9e6-c75d6c0e0611\") " Nov 01 21:29:16.298967 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:16.298942 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64dada2e-ccc9-4ab8-a9e6-c75d6c0e0611-maas-api-openshift-ca" pod "64dada2e-ccc9-4ab8-a9e6-c75d6c0e0611" (UID: "64dada2e-ccc9-4ab8-a9e6-c75d6c0e0611"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:29:16.300704 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:16.300672 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64dada2e-ccc9-4ab8-a9e6-c75d6c0e0611-maas-api-xks-ca" pod "64dada2e-ccc9-4ab8-a9e6-c75d6c0e0611" (UID: "64dada2e-ccc9-4ab8-a9e6-c75d6c0e0611"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:29:16.300801 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:16.300714 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64dada2e-ccc9-4ab8-a9e6-c75d6c0e0611-kube-api-access-zkrf6" pod "64dada2e-ccc9-4ab8-a9e6-c75d6c0e0611" (UID: "64dada2e-ccc9-4ab8-a9e6-c75d6c0e0611"). InnerVolumeSpecName "kube-api-access-zkrf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:29:16.399762 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:16.399664 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64dada2e-ccc9-4ab8-a9e6-c75d6c0e0611-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:29:16.399762 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:16.399699 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkrf6\" (UniqueName: \"kubernetes.io/projected/64dada2e-ccc9-4ab8-a9e6-c75d6c0e0611-kube-api-access-zkrf6\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:29:16.399762 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:16.399716 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64dada2e-ccc9-4ab8-a9e6-c75d6c0e0611-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:29:17.087546 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:17.087501 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rxxx4" event={"ID":"64dada2e-ccc9-4ab8-a9e6-c75d6c0e0611","Type":"ContainerDied","Data":"0ba0286aabe98a49312bd8f196c8c0828e2c690b14972bce0ff810074b0647ef"} Nov 01 21:29:17.087733 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:17.087553 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ba0286aabe98a49312bd8f196c8c0828e2c690b14972bce0ff810074b0647ef" Nov 01 21:29:17.087733 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:17.087515 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rxxx4" Nov 01 21:29:41.783686 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:41.783655 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7hj4r"] Nov 01 21:29:41.784142 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:41.784087 4484 cpu_manager.go:414] "containerMap: removing container" podUID="acd7c6cc-09ae-4036-b815-64b107627c63" containerName="istio-proxy" Nov 01 21:29:41.784142 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:41.784100 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acd7c6cc-09ae-4036-b815-64b107627c63" containerName="istio-proxy" Nov 01 21:29:41.784142 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:41.784132 4484 cpu_manager.go:414] "containerMap: removing container" podUID="64dada2e-ccc9-4ab8-a9e6-c75d6c0e0611" containerName="revoke-keys" Nov 01 21:29:41.784142 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:41.784137 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64dada2e-ccc9-4ab8-a9e6-c75d6c0e0611" containerName="revoke-keys" Nov 01 21:29:41.784289 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:41.784223 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="64dada2e-ccc9-4ab8-a9e6-c75d6c0e0611" containerName="revoke-keys" Nov 01 21:29:41.787297 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:41.787277 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7hj4r" Nov 01 21:29:41.841781 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:41.841737 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:29:41.842068 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:41.842041 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:29:41.843065 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:41.843031 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7hj4r"] Nov 01 21:29:41.891454 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:41.891411 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:29:41.891642 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:41.891480 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:29:41.895885 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:41.895864 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08fca9d9-f3ac-49b2-bc69-8e13be477f5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-7hj4r\" (UID: \"08fca9d9-f3ac-49b2-bc69-8e13be477f5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7hj4r" Nov 01 21:29:41.895959 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:41.895937 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvdnv\" (UniqueName: \"kubernetes.io/projected/08fca9d9-f3ac-49b2-bc69-8e13be477f5a-kube-api-access-kvdnv\") pod \"maas-api-revoke-sub-d2bc93bc-7hj4r\" (UID: \"08fca9d9-f3ac-49b2-bc69-8e13be477f5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7hj4r" Nov 01 21:29:41.896093 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:41.896075 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08fca9d9-f3ac-49b2-bc69-8e13be477f5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-7hj4r\" (UID: \"08fca9d9-f3ac-49b2-bc69-8e13be477f5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7hj4r" Nov 01 21:29:41.997269 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:41.997228 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08fca9d9-f3ac-49b2-bc69-8e13be477f5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-7hj4r\" (UID: \"08fca9d9-f3ac-49b2-bc69-8e13be477f5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7hj4r" Nov 01 21:29:41.997498 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:41.997336 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvdnv\" (UniqueName: \"kubernetes.io/projected/08fca9d9-f3ac-49b2-bc69-8e13be477f5a-kube-api-access-kvdnv\") pod \"maas-api-revoke-sub-d2bc93bc-7hj4r\" (UID: \"08fca9d9-f3ac-49b2-bc69-8e13be477f5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7hj4r" Nov 01 21:29:41.997498 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:41.997396 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08fca9d9-f3ac-49b2-bc69-8e13be477f5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-7hj4r\" (UID: \"08fca9d9-f3ac-49b2-bc69-8e13be477f5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7hj4r" Nov 01 21:29:41.997960 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:41.997934 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08fca9d9-f3ac-49b2-bc69-8e13be477f5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-7hj4r\" (UID: \"08fca9d9-f3ac-49b2-bc69-8e13be477f5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7hj4r" Nov 01 21:29:41.999733 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:41.999710 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08fca9d9-f3ac-49b2-bc69-8e13be477f5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-7hj4r\" (UID: \"08fca9d9-f3ac-49b2-bc69-8e13be477f5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7hj4r" Nov 01 21:29:42.056086 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:42.055991 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvdnv\" (UniqueName: \"kubernetes.io/projected/08fca9d9-f3ac-49b2-bc69-8e13be477f5a-kube-api-access-kvdnv\") pod \"maas-api-revoke-sub-d2bc93bc-7hj4r\" (UID: \"08fca9d9-f3ac-49b2-bc69-8e13be477f5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7hj4r" Nov 01 21:29:42.101090 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:42.101054 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7hj4r" Nov 01 21:29:42.471916 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:42.471894 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7hj4r"] Nov 01 21:29:42.473769 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:29:42.473744 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08fca9d9_f3ac_49b2_bc69_8e13be477f5a.slice/crio-5324fcadccc236170c1dc448a3b8d208e7930ad0472f2ab8373abd1cd630222c WatchSource:0}: Error finding container 5324fcadccc236170c1dc448a3b8d208e7930ad0472f2ab8373abd1cd630222c: Status 404 returned error can't find the container with id 5324fcadccc236170c1dc448a3b8d208e7930ad0472f2ab8373abd1cd630222c Nov 01 21:29:43.235504 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:43.235466 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7hj4r" event={"ID":"08fca9d9-f3ac-49b2-bc69-8e13be477f5a","Type":"ContainerStarted","Data":"178435720ae42dbe0b9dc84687db2e054a7994008617a61ff7042129603cfbe9"} Nov 01 21:29:43.235918 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:43.235529 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7hj4r" event={"ID":"08fca9d9-f3ac-49b2-bc69-8e13be477f5a","Type":"ContainerStarted","Data":"5324fcadccc236170c1dc448a3b8d208e7930ad0472f2ab8373abd1cd630222c"} Nov 01 21:29:43.351713 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:43.351658 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7hj4r" podStartSLOduration=2.351643763 podStartE2EDuration="2.351643763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:29:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:29:43.350174528 +0000 UTC m=+3534.127814675" watchObservedRunningTime="2026-11-01 21:29:43.351643763 +0000 UTC m=+3534.129283897" Nov 01 21:29:44.243254 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:44.243211 4484 generic.go:356] "Generic (PLEG): container finished" podID="08fca9d9-f3ac-49b2-bc69-8e13be477f5a" containerID="178435720ae42dbe0b9dc84687db2e054a7994008617a61ff7042129603cfbe9" exitCode=0 Nov 01 21:29:44.243757 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:44.243290 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7hj4r" event={"ID":"08fca9d9-f3ac-49b2-bc69-8e13be477f5a","Type":"ContainerDied","Data":"178435720ae42dbe0b9dc84687db2e054a7994008617a61ff7042129603cfbe9"} Nov 01 21:29:45.388261 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:45.388232 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7hj4r" Nov 01 21:29:45.533731 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:45.533635 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08fca9d9-f3ac-49b2-bc69-8e13be477f5a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08fca9d9-f3ac-49b2-bc69-8e13be477f5a-maas-api-openshift-ca\") pod \"08fca9d9-f3ac-49b2-bc69-8e13be477f5a\" (UID: \"08fca9d9-f3ac-49b2-bc69-8e13be477f5a\") " Nov 01 21:29:45.533731 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:45.533684 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08fca9d9-f3ac-49b2-bc69-8e13be477f5a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08fca9d9-f3ac-49b2-bc69-8e13be477f5a-maas-api-xks-ca\") pod \"08fca9d9-f3ac-49b2-bc69-8e13be477f5a\" (UID: \"08fca9d9-f3ac-49b2-bc69-8e13be477f5a\") " Nov 01 21:29:45.533990 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:45.533744 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08fca9d9-f3ac-49b2-bc69-8e13be477f5a-kube-api-access-kvdnv\" (UniqueName: \"kubernetes.io/projected/08fca9d9-f3ac-49b2-bc69-8e13be477f5a-kube-api-access-kvdnv\") pod \"08fca9d9-f3ac-49b2-bc69-8e13be477f5a\" (UID: \"08fca9d9-f3ac-49b2-bc69-8e13be477f5a\") " Nov 01 21:29:45.534092 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:45.534024 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08fca9d9-f3ac-49b2-bc69-8e13be477f5a-maas-api-openshift-ca" pod "08fca9d9-f3ac-49b2-bc69-8e13be477f5a" (UID: "08fca9d9-f3ac-49b2-bc69-8e13be477f5a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:29:45.534203 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:45.534184 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08fca9d9-f3ac-49b2-bc69-8e13be477f5a-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:29:45.535871 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:45.535848 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08fca9d9-f3ac-49b2-bc69-8e13be477f5a-kube-api-access-kvdnv" pod "08fca9d9-f3ac-49b2-bc69-8e13be477f5a" (UID: "08fca9d9-f3ac-49b2-bc69-8e13be477f5a"). InnerVolumeSpecName "kube-api-access-kvdnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:29:45.536007 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:45.535902 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08fca9d9-f3ac-49b2-bc69-8e13be477f5a-maas-api-xks-ca" pod "08fca9d9-f3ac-49b2-bc69-8e13be477f5a" (UID: "08fca9d9-f3ac-49b2-bc69-8e13be477f5a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:29:45.635043 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:45.634982 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08fca9d9-f3ac-49b2-bc69-8e13be477f5a-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:29:45.635043 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:45.635044 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvdnv\" (UniqueName: \"kubernetes.io/projected/08fca9d9-f3ac-49b2-bc69-8e13be477f5a-kube-api-access-kvdnv\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:29:45.766139 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:29:45.766105 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:29:46.255290 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:46.255258 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7hj4r" Nov 01 21:29:46.255505 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:46.255290 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7hj4r" event={"ID":"08fca9d9-f3ac-49b2-bc69-8e13be477f5a","Type":"ContainerDied","Data":"5324fcadccc236170c1dc448a3b8d208e7930ad0472f2ab8373abd1cd630222c"} Nov 01 21:29:46.255505 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:46.255334 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5324fcadccc236170c1dc448a3b8d208e7930ad0472f2ab8373abd1cd630222c" Nov 01 21:29:53.133937 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:53.133903 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:29:54.070134 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:54.070099 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:29:54.528873 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:54.528833 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk"] Nov 01 21:29:54.529365 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:54.529273 4484 cpu_manager.go:414] "containerMap: removing container" podUID="08fca9d9-f3ac-49b2-bc69-8e13be477f5a" containerName="revoke-keys" Nov 01 21:29:54.529365 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:54.529287 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08fca9d9-f3ac-49b2-bc69-8e13be477f5a" containerName="revoke-keys" Nov 01 21:29:54.529365 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:54.529354 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="08fca9d9-f3ac-49b2-bc69-8e13be477f5a" containerName="revoke-keys" Nov 01 21:29:54.532448 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:54.532430 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk" Nov 01 21:29:54.611186 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:54.611147 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb507845-2edf-4ffa-b057-da162cebddee-credential-socket\") pod \"e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk\" (UID: \"fb507845-2edf-4ffa-b057-da162cebddee\") " pod="openshift-ingress/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk" Nov 01 21:29:54.611403 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:54.611228 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb507845-2edf-4ffa-b057-da162cebddee-istiod-ca-cert\") pod \"e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk\" (UID: \"fb507845-2edf-4ffa-b057-da162cebddee\") " pod="openshift-ingress/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk" Nov 01 21:29:54.611403 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:54.611277 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98rwj\" (UniqueName: \"kubernetes.io/projected/fb507845-2edf-4ffa-b057-da162cebddee-kube-api-access-98rwj\") pod \"e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk\" (UID: \"fb507845-2edf-4ffa-b057-da162cebddee\") " pod="openshift-ingress/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk" Nov 01 21:29:54.611403 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:54.611298 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb507845-2edf-4ffa-b057-da162cebddee-istio-podinfo\") pod \"e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk\" (UID: \"fb507845-2edf-4ffa-b057-da162cebddee\") " pod="openshift-ingress/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk" Nov 01 21:29:54.611403 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:54.611336 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb507845-2edf-4ffa-b057-da162cebddee-workload-certs\") pod \"e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk\" (UID: \"fb507845-2edf-4ffa-b057-da162cebddee\") " pod="openshift-ingress/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk" Nov 01 21:29:54.611403 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:54.611368 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb507845-2edf-4ffa-b057-da162cebddee-istio-envoy\") pod \"e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk\" (UID: \"fb507845-2edf-4ffa-b057-da162cebddee\") " pod="openshift-ingress/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk" Nov 01 21:29:54.611403 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:54.611400 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb507845-2edf-4ffa-b057-da162cebddee-workload-socket\") pod \"e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk\" (UID: \"fb507845-2edf-4ffa-b057-da162cebddee\") " pod="openshift-ingress/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk" Nov 01 21:29:54.611781 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:54.611441 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb507845-2edf-4ffa-b057-da162cebddee-istio-token\") pod \"e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk\" (UID: \"fb507845-2edf-4ffa-b057-da162cebddee\") " pod="openshift-ingress/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk" Nov 01 21:29:54.611781 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:54.611458 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb507845-2edf-4ffa-b057-da162cebddee-istio-data\") pod \"e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk\" (UID: \"fb507845-2edf-4ffa-b057-da162cebddee\") " pod="openshift-ingress/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk" Nov 01 21:29:54.635555 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:54.635527 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-236f0f-openshift-default-dockercfg-fzxwt\"" Nov 01 21:29:54.641799 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:54.641767 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk"] Nov 01 21:29:54.658279 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:54.658248 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:29:54.712951 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:54.712914 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98rwj\" (UniqueName: \"kubernetes.io/projected/fb507845-2edf-4ffa-b057-da162cebddee-kube-api-access-98rwj\") pod \"e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk\" (UID: \"fb507845-2edf-4ffa-b057-da162cebddee\") " pod="openshift-ingress/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk" Nov 01 21:29:54.713199 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:54.712965 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb507845-2edf-4ffa-b057-da162cebddee-istio-podinfo\") pod \"e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk\" (UID: \"fb507845-2edf-4ffa-b057-da162cebddee\") " pod="openshift-ingress/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk" Nov 01 21:29:54.713199 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:54.713038 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb507845-2edf-4ffa-b057-da162cebddee-workload-certs\") pod \"e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk\" (UID: \"fb507845-2edf-4ffa-b057-da162cebddee\") " pod="openshift-ingress/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk" Nov 01 21:29:54.713199 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:54.713078 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb507845-2edf-4ffa-b057-da162cebddee-istio-envoy\") pod \"e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk\" (UID: \"fb507845-2edf-4ffa-b057-da162cebddee\") " pod="openshift-ingress/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk" Nov 01 21:29:54.714059 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:54.714027 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb507845-2edf-4ffa-b057-da162cebddee-workload-socket\") pod \"e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk\" (UID: \"fb507845-2edf-4ffa-b057-da162cebddee\") " pod="openshift-ingress/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk" Nov 01 21:29:54.714226 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:54.714100 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb507845-2edf-4ffa-b057-da162cebddee-istio-token\") pod \"e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk\" (UID: \"fb507845-2edf-4ffa-b057-da162cebddee\") " pod="openshift-ingress/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk" Nov 01 21:29:54.714226 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:54.714125 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb507845-2edf-4ffa-b057-da162cebddee-istio-data\") pod \"e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk\" (UID: \"fb507845-2edf-4ffa-b057-da162cebddee\") " pod="openshift-ingress/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk" Nov 01 21:29:54.714226 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:54.714192 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb507845-2edf-4ffa-b057-da162cebddee-credential-socket\") pod \"e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk\" (UID: \"fb507845-2edf-4ffa-b057-da162cebddee\") " pod="openshift-ingress/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk" Nov 01 21:29:54.714226 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:54.714218 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb507845-2edf-4ffa-b057-da162cebddee-istiod-ca-cert\") pod \"e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk\" (UID: \"fb507845-2edf-4ffa-b057-da162cebddee\") " pod="openshift-ingress/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk" Nov 01 21:29:54.714586 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:54.714462 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb507845-2edf-4ffa-b057-da162cebddee-workload-socket\") pod \"e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk\" (UID: \"fb507845-2edf-4ffa-b057-da162cebddee\") " pod="openshift-ingress/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk" Nov 01 21:29:54.714844 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:54.714813 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb507845-2edf-4ffa-b057-da162cebddee-workload-certs\") pod \"e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk\" (UID: \"fb507845-2edf-4ffa-b057-da162cebddee\") " pod="openshift-ingress/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk" Nov 01 21:29:54.716111 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:54.715117 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb507845-2edf-4ffa-b057-da162cebddee-istiod-ca-cert\") pod \"e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk\" (UID: \"fb507845-2edf-4ffa-b057-da162cebddee\") " pod="openshift-ingress/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk" Nov 01 21:29:54.716111 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:54.716024 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb507845-2edf-4ffa-b057-da162cebddee-istio-data\") pod \"e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk\" (UID: \"fb507845-2edf-4ffa-b057-da162cebddee\") " pod="openshift-ingress/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk" Nov 01 21:29:54.716276 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:54.716247 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb507845-2edf-4ffa-b057-da162cebddee-credential-socket\") pod \"e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk\" (UID: \"fb507845-2edf-4ffa-b057-da162cebddee\") " pod="openshift-ingress/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk" Nov 01 21:29:54.716818 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:54.716797 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb507845-2edf-4ffa-b057-da162cebddee-istio-envoy\") pod \"e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk\" (UID: \"fb507845-2edf-4ffa-b057-da162cebddee\") " pod="openshift-ingress/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk" Nov 01 21:29:54.718516 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:54.718498 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb507845-2edf-4ffa-b057-da162cebddee-istio-podinfo\") pod \"e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk\" (UID: \"fb507845-2edf-4ffa-b057-da162cebddee\") " pod="openshift-ingress/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk" Nov 01 21:29:54.771309 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:54.771274 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb507845-2edf-4ffa-b057-da162cebddee-istio-token\") pod \"e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk\" (UID: \"fb507845-2edf-4ffa-b057-da162cebddee\") " pod="openshift-ingress/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk" Nov 01 21:29:54.771558 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:54.771532 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98rwj\" (UniqueName: \"kubernetes.io/projected/fb507845-2edf-4ffa-b057-da162cebddee-kube-api-access-98rwj\") pod \"e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk\" (UID: \"fb507845-2edf-4ffa-b057-da162cebddee\") " pod="openshift-ingress/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk" Nov 01 21:29:54.844907 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:54.844824 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk" Nov 01 21:29:55.098291 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:55.098200 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:29:55.338274 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:55.338246 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk"] Nov 01 21:29:55.339449 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:29:55.339421 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb507845_2edf_4ffa_b057_da162cebddee.slice/crio-a801dc068233497fe889c9727e5f88f9ac2b9c1c18f1be5e8315b82d46bd1be0 WatchSource:0}: Error finding container a801dc068233497fe889c9727e5f88f9ac2b9c1c18f1be5e8315b82d46bd1be0: Status 404 returned error can't find the container with id a801dc068233497fe889c9727e5f88f9ac2b9c1c18f1be5e8315b82d46bd1be0 Nov 01 21:29:55.341592 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:55.341556 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:29:55.341669 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:55.341633 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:29:55.341708 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:55.341667 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:29:56.305561 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:56.305527 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk" event={"ID":"fb507845-2edf-4ffa-b057-da162cebddee","Type":"ContainerStarted","Data":"801b18ee563f72bb54d212dd5ade7744675943cfe32f71fa896d984eae85b073"} Nov 01 21:29:56.306142 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:56.305577 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk" event={"ID":"fb507845-2edf-4ffa-b057-da162cebddee","Type":"ContainerStarted","Data":"a801dc068233497fe889c9727e5f88f9ac2b9c1c18f1be5e8315b82d46bd1be0"} Nov 01 21:29:56.424979 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:56.424925 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk" podStartSLOduration=2.424910947 podStartE2EDuration="2.424910947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:29:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:29:56.424580263 +0000 UTC m=+3547.202220397" watchObservedRunningTime="2026-11-01 21:29:56.424910947 +0000 UTC m=+3547.202551081" Nov 01 21:29:56.846014 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:56.845970 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk" Nov 01 21:29:56.850837 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:56.850811 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk" Nov 01 21:29:57.310322 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:57.310288 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk" Nov 01 21:29:57.311454 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:57.311430 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk" Nov 01 21:29:59.288313 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:59.288275 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:29:59.463185 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:59.463152 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:29:59.636269 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:29:59.636176 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:30:00.160007 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:00.159966 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892810-gkxdl"] Nov 01 21:30:00.163739 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:00.163713 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892810-gkxdl" Nov 01 21:30:00.217401 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:00.217366 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:30:00.220162 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:00.220131 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892810-gkxdl"] Nov 01 21:30:00.268425 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:00.268389 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:30:00.268614 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:00.268441 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:30:00.275936 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:00.275909 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d0e82d9-685b-4e15-81a9-a7c87ee13eb8-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892810-gkxdl\" (UID: \"3d0e82d9-685b-4e15-81a9-a7c87ee13eb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892810-gkxdl" Nov 01 21:30:00.276081 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:00.276059 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d46c\" (UniqueName: \"kubernetes.io/projected/3d0e82d9-685b-4e15-81a9-a7c87ee13eb8-kube-api-access-9d46c\") pod \"maas-api-key-cleanup-29892810-gkxdl\" (UID: \"3d0e82d9-685b-4e15-81a9-a7c87ee13eb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892810-gkxdl" Nov 01 21:30:00.377452 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:00.377416 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d0e82d9-685b-4e15-81a9-a7c87ee13eb8-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892810-gkxdl\" (UID: \"3d0e82d9-685b-4e15-81a9-a7c87ee13eb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892810-gkxdl" Nov 01 21:30:00.377885 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:00.377495 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d46c\" (UniqueName: \"kubernetes.io/projected/3d0e82d9-685b-4e15-81a9-a7c87ee13eb8-kube-api-access-9d46c\") pod \"maas-api-key-cleanup-29892810-gkxdl\" (UID: \"3d0e82d9-685b-4e15-81a9-a7c87ee13eb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892810-gkxdl" Nov 01 21:30:00.378103 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:00.378080 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d0e82d9-685b-4e15-81a9-a7c87ee13eb8-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892810-gkxdl\" (UID: \"3d0e82d9-685b-4e15-81a9-a7c87ee13eb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892810-gkxdl" Nov 01 21:30:00.435015 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:00.434939 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d46c\" (UniqueName: \"kubernetes.io/projected/3d0e82d9-685b-4e15-81a9-a7c87ee13eb8-kube-api-access-9d46c\") pod \"maas-api-key-cleanup-29892810-gkxdl\" (UID: \"3d0e82d9-685b-4e15-81a9-a7c87ee13eb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892810-gkxdl" Nov 01 21:30:00.478208 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:00.478172 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892810-gkxdl" Nov 01 21:30:00.847414 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:00.847384 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892810-gkxdl"] Nov 01 21:30:00.849337 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:30:00.849313 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d0e82d9_685b_4e15_81a9_a7c87ee13eb8.slice/crio-aa41de1cbae929d9a2766dae47650ca1331c3187d17143c39e11db190f77af01 WatchSource:0}: Error finding container aa41de1cbae929d9a2766dae47650ca1331c3187d17143c39e11db190f77af01: Status 404 returned error can't find the container with id aa41de1cbae929d9a2766dae47650ca1331c3187d17143c39e11db190f77af01 Nov 01 21:30:01.331691 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:01.331655 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892810-gkxdl" event={"ID":"3d0e82d9-685b-4e15-81a9-a7c87ee13eb8","Type":"ContainerStarted","Data":"cfa135e568ac2b2c0cbf8f404286afd9729d78f88f596c65bddc7512d59a52a3"} Nov 01 21:30:01.331875 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:01.331717 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892810-gkxdl" event={"ID":"3d0e82d9-685b-4e15-81a9-a7c87ee13eb8","Type":"ContainerStarted","Data":"aa41de1cbae929d9a2766dae47650ca1331c3187d17143c39e11db190f77af01"} Nov 01 21:30:01.447386 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:01.447289 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892810-gkxdl" podStartSLOduration=1.44727041 podStartE2EDuration="1.44727041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:30:01.445445467 +0000 UTC m=+3552.223085603" watchObservedRunningTime="2026-11-01 21:30:01.44727041 +0000 UTC m=+3552.224910539" Nov 01 21:30:02.338673 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:02.338631 4484 generic.go:356] "Generic (PLEG): container finished" podID="3d0e82d9-685b-4e15-81a9-a7c87ee13eb8" containerID="cfa135e568ac2b2c0cbf8f404286afd9729d78f88f596c65bddc7512d59a52a3" exitCode=60 Nov 01 21:30:02.338883 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:02.338722 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892810-gkxdl" event={"ID":"3d0e82d9-685b-4e15-81a9-a7c87ee13eb8","Type":"ContainerDied","Data":"cfa135e568ac2b2c0cbf8f404286afd9729d78f88f596c65bddc7512d59a52a3"} Nov 01 21:30:02.339029 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:02.339012 4484 scope.go:122] "RemoveContainer" containerID="cfa135e568ac2b2c0cbf8f404286afd9729d78f88f596c65bddc7512d59a52a3" Nov 01 21:30:02.487852 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:02.487810 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:30:02.564222 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:02.564190 4484 scope.go:122] "RemoveContainer" containerID="af7ac39d3c6a7474cf841ef458cf14abb8ac2cd62dec91c8e3f467481717cc1e" Nov 01 21:30:02.600673 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:02.600642 4484 scope.go:122] "RemoveContainer" containerID="7b709062397a2ed5118d6d9f047f8b38b74debf8cb31bee32912c1a9fff87716" Nov 01 21:30:02.667874 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:02.667841 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:30:03.200617 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:03.200582 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-236f0f-59cf6t5j8h"] Nov 01 21:30:03.204363 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:03.204326 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-236f0f-59cf6t5j8h" Nov 01 21:30:03.260358 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:03.260322 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-236f0f-59cf6t5j8h"] Nov 01 21:30:03.308188 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:03.308160 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-9ce98c44-dockercfg-v5pkt\"" Nov 01 21:30:03.308355 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:03.308207 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-236f0f\"" Nov 01 21:30:03.311705 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:03.311644 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56vnd\" (UniqueName: \"kubernetes.io/projected/24119262-952c-4224-985e-a52341194eb3-kube-api-access-56vnd\") pod \"payload-pre-processing-e2e-worker-single-main-236f0f-59cf6t5j8h\" (UID: \"24119262-952c-4224-985e-a52341194eb3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-236f0f-59cf6t5j8h" Nov 01 21:30:03.311860 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:03.311769 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24119262-952c-4224-985e-a52341194eb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-236f0f-59cf6t5j8h\" (UID: \"24119262-952c-4224-985e-a52341194eb3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-236f0f-59cf6t5j8h" Nov 01 21:30:03.345470 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:03.345435 4484 generic.go:356] "Generic (PLEG): container finished" podID="3d0e82d9-685b-4e15-81a9-a7c87ee13eb8" containerID="5cb876bd5c5bc97448d77ffdcd230c9a13d9c8ed7b999dc552651ac331316e5a" exitCode=60 Nov 01 21:30:03.345686 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:03.345493 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892810-gkxdl" event={"ID":"3d0e82d9-685b-4e15-81a9-a7c87ee13eb8","Type":"ContainerDied","Data":"5cb876bd5c5bc97448d77ffdcd230c9a13d9c8ed7b999dc552651ac331316e5a"} Nov 01 21:30:03.345686 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:03.345549 4484 scope.go:122] "RemoveContainer" containerID="cfa135e568ac2b2c0cbf8f404286afd9729d78f88f596c65bddc7512d59a52a3" Nov 01 21:30:03.345808 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:03.345792 4484 scope.go:122] "RemoveContainer" containerID="5cb876bd5c5bc97448d77ffdcd230c9a13d9c8ed7b999dc552651ac331316e5a" Nov 01 21:30:03.346151 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:30:03.346118 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892810-gkxdl_odh-ai-gateway-infra(3d0e82d9-685b-4e15-81a9-a7c87ee13eb8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892810-gkxdl" podUID="3d0e82d9-685b-4e15-81a9-a7c87ee13eb8" Nov 01 21:30:03.412727 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:03.412686 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24119262-952c-4224-985e-a52341194eb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-236f0f-59cf6t5j8h\" (UID: \"24119262-952c-4224-985e-a52341194eb3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-236f0f-59cf6t5j8h" Nov 01 21:30:03.412922 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:03.412811 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56vnd\" (UniqueName: \"kubernetes.io/projected/24119262-952c-4224-985e-a52341194eb3-kube-api-access-56vnd\") pod \"payload-pre-processing-e2e-worker-single-main-236f0f-59cf6t5j8h\" (UID: \"24119262-952c-4224-985e-a52341194eb3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-236f0f-59cf6t5j8h" Nov 01 21:30:03.413456 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:03.413433 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24119262-952c-4224-985e-a52341194eb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-236f0f-59cf6t5j8h\" (UID: \"24119262-952c-4224-985e-a52341194eb3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-236f0f-59cf6t5j8h" Nov 01 21:30:03.470594 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:03.470514 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56vnd\" (UniqueName: \"kubernetes.io/projected/24119262-952c-4224-985e-a52341194eb3-kube-api-access-56vnd\") pod \"payload-pre-processing-e2e-worker-single-main-236f0f-59cf6t5j8h\" (UID: \"24119262-952c-4224-985e-a52341194eb3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-236f0f-59cf6t5j8h" Nov 01 21:30:03.518461 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:03.518418 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-236f0f-59cf6t5j8h" Nov 01 21:30:03.898509 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:03.898479 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-236f0f-59cf6t5j8h"] Nov 01 21:30:03.899570 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:30:03.899542 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24119262_952c_4224_985e_a52341194eb3.slice/crio-e9c0019dfcce9dcbdd55042c18bf7a3b13f2c7049211222ebc8fba1b40b66890 WatchSource:0}: Error finding container e9c0019dfcce9dcbdd55042c18bf7a3b13f2c7049211222ebc8fba1b40b66890: Status 404 returned error can't find the container with id e9c0019dfcce9dcbdd55042c18bf7a3b13f2c7049211222ebc8fba1b40b66890 Nov 01 21:30:04.352263 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:04.352236 4484 scope.go:122] "RemoveContainer" containerID="5cb876bd5c5bc97448d77ffdcd230c9a13d9c8ed7b999dc552651ac331316e5a" Nov 01 21:30:04.352530 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:30:04.352509 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892810-gkxdl_odh-ai-gateway-infra(3d0e82d9-685b-4e15-81a9-a7c87ee13eb8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892810-gkxdl" podUID="3d0e82d9-685b-4e15-81a9-a7c87ee13eb8" Nov 01 21:30:04.353588 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:04.353563 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-236f0f-59cf6t5j8h" event={"ID":"24119262-952c-4224-985e-a52341194eb3","Type":"ContainerStarted","Data":"01781757d8447abf7d2573ce1090afbf9445d7f47b3439151c7162aa1cb26e1e"} Nov 01 21:30:04.353692 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:04.353606 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-236f0f-59cf6t5j8h" event={"ID":"24119262-952c-4224-985e-a52341194eb3","Type":"ContainerStarted","Data":"e9c0019dfcce9dcbdd55042c18bf7a3b13f2c7049211222ebc8fba1b40b66890"} Nov 01 21:30:04.353734 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:04.353692 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-236f0f-59cf6t5j8h" Nov 01 21:30:04.581536 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:04.581455 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-236f0f-59cf6t5j8h" podStartSLOduration=1.581441709 podStartE2EDuration="1.581441709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:30:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:30:04.580418274 +0000 UTC m=+3555.358058411" watchObservedRunningTime="2026-11-01 21:30:04.581441709 +0000 UTC m=+3555.359081843" Nov 01 21:30:05.475288 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:05.475256 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:30:05.632767 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:05.632728 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:30:05.805988 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:05.805952 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:30:10.423913 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:10.423873 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:30:15.382506 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:15.376776 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:30:15.383811 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:15.383788 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-236f0f-59cf6t5j8h" Nov 01 21:30:15.384448 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:15.384424 4484 scope.go:122] "RemoveContainer" containerID="5cb876bd5c5bc97448d77ffdcd230c9a13d9c8ed7b999dc552651ac331316e5a" Nov 01 21:30:15.766178 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:30:15.766141 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:30:16.438552 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:16.438516 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892810-gkxdl" event={"ID":"3d0e82d9-685b-4e15-81a9-a7c87ee13eb8","Type":"ContainerStarted","Data":"49a83dedd7489ffdf98b67f6e490cf632c6f8fa7ed6686f62ae2e2c0df9b798f"} Nov 01 21:30:17.447440 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:17.447397 4484 generic.go:356] "Generic (PLEG): container finished" podID="3d0e82d9-685b-4e15-81a9-a7c87ee13eb8" containerID="49a83dedd7489ffdf98b67f6e490cf632c6f8fa7ed6686f62ae2e2c0df9b798f" exitCode=60 Nov 01 21:30:17.447945 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:17.447458 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892810-gkxdl" event={"ID":"3d0e82d9-685b-4e15-81a9-a7c87ee13eb8","Type":"ContainerDied","Data":"49a83dedd7489ffdf98b67f6e490cf632c6f8fa7ed6686f62ae2e2c0df9b798f"} Nov 01 21:30:17.447945 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:17.447537 4484 scope.go:122] "RemoveContainer" containerID="5cb876bd5c5bc97448d77ffdcd230c9a13d9c8ed7b999dc552651ac331316e5a" Nov 01 21:30:17.447945 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:17.447915 4484 scope.go:122] "RemoveContainer" containerID="49a83dedd7489ffdf98b67f6e490cf632c6f8fa7ed6686f62ae2e2c0df9b798f" Nov 01 21:30:17.448261 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:30:17.448236 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29892810-gkxdl_odh-ai-gateway-infra(3d0e82d9-685b-4e15-81a9-a7c87ee13eb8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892810-gkxdl" podUID="3d0e82d9-685b-4e15-81a9-a7c87ee13eb8" Nov 01 21:30:17.565624 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:17.565523 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892810-gkxdl"] Nov 01 21:30:18.601222 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:18.601195 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892810-gkxdl" Nov 01 21:30:18.701763 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:18.701726 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d0e82d9-685b-4e15-81a9-a7c87ee13eb8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d0e82d9-685b-4e15-81a9-a7c87ee13eb8-maas-api-service-ca\") pod \"3d0e82d9-685b-4e15-81a9-a7c87ee13eb8\" (UID: \"3d0e82d9-685b-4e15-81a9-a7c87ee13eb8\") " Nov 01 21:30:18.701980 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:18.701801 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d0e82d9-685b-4e15-81a9-a7c87ee13eb8-kube-api-access-9d46c\" (UniqueName: \"kubernetes.io/projected/3d0e82d9-685b-4e15-81a9-a7c87ee13eb8-kube-api-access-9d46c\") pod \"3d0e82d9-685b-4e15-81a9-a7c87ee13eb8\" (UID: \"3d0e82d9-685b-4e15-81a9-a7c87ee13eb8\") " Nov 01 21:30:18.702222 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:18.702191 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d0e82d9-685b-4e15-81a9-a7c87ee13eb8-maas-api-service-ca" pod "3d0e82d9-685b-4e15-81a9-a7c87ee13eb8" (UID: "3d0e82d9-685b-4e15-81a9-a7c87ee13eb8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:30:18.704105 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:18.703968 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d0e82d9-685b-4e15-81a9-a7c87ee13eb8-kube-api-access-9d46c" pod "3d0e82d9-685b-4e15-81a9-a7c87ee13eb8" (UID: "3d0e82d9-685b-4e15-81a9-a7c87ee13eb8"). InnerVolumeSpecName "kube-api-access-9d46c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:30:18.803448 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:18.803406 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d0e82d9-685b-4e15-81a9-a7c87ee13eb8-maas-api-service-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:30:18.803448 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:18.803446 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d46c\" (UniqueName: \"kubernetes.io/projected/3d0e82d9-685b-4e15-81a9-a7c87ee13eb8-kube-api-access-9d46c\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:30:19.460336 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:19.460307 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892810-gkxdl" Nov 01 21:30:19.460521 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:19.460309 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892810-gkxdl" event={"ID":"3d0e82d9-685b-4e15-81a9-a7c87ee13eb8","Type":"ContainerDied","Data":"aa41de1cbae929d9a2766dae47650ca1331c3187d17143c39e11db190f77af01"} Nov 01 21:30:19.460521 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:19.460439 4484 scope.go:122] "RemoveContainer" containerID="49a83dedd7489ffdf98b67f6e490cf632c6f8fa7ed6686f62ae2e2c0df9b798f" Nov 01 21:30:19.625306 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:19.625269 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892810-gkxdl"] Nov 01 21:30:19.676646 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:19.676571 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892810-gkxdl"] Nov 01 21:30:20.143683 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:20.143650 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:30:21.371109 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:21.371075 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d0e82d9-685b-4e15-81a9-a7c87ee13eb8" path="/var/lib/kubelet/pods/3d0e82d9-685b-4e15-81a9-a7c87ee13eb8/volumes" Nov 01 21:30:21.474369 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:21.474336 4484 generic.go:356] "Generic (PLEG): container finished" podID="fb507845-2edf-4ffa-b057-da162cebddee" containerID="801b18ee563f72bb54d212dd5ade7744675943cfe32f71fa896d984eae85b073" exitCode=0 Nov 01 21:30:21.474576 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:21.474414 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk" event={"ID":"fb507845-2edf-4ffa-b057-da162cebddee","Type":"ContainerDied","Data":"801b18ee563f72bb54d212dd5ade7744675943cfe32f71fa896d984eae85b073"} Nov 01 21:30:21.474880 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:21.474862 4484 scope.go:122] "RemoveContainer" containerID="801b18ee563f72bb54d212dd5ade7744675943cfe32f71fa896d984eae85b073" Nov 01 21:30:21.476174 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:21.476144 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:30:21.476264 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:21.476220 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:30:21.476264 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:21.476254 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:30:22.480892 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:22.480859 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk" event={"ID":"fb507845-2edf-4ffa-b057-da162cebddee","Type":"ContainerStarted","Data":"9b5f6404562c55f40ef93c828810d27880ee9df174902fffd18157ce30c56e93"} Nov 01 21:30:22.845386 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:22.845344 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk" Nov 01 21:30:22.846666 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:22.846640 4484 patch_prober.go:29] interesting pod/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused" start-of-body= Nov 01 21:30:22.846802 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:22.846688 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk" podUID="fb507845-2edf-4ffa-b057-da162cebddee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused" Nov 01 21:30:23.845334 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:23.845294 4484 patch_prober.go:29] interesting pod/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused" start-of-body= Nov 01 21:30:23.845768 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:23.845348 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk" podUID="fb507845-2edf-4ffa-b057-da162cebddee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused" Nov 01 21:30:23.936675 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:23.936637 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:30:24.488547 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:24.488504 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:30:24.664161 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:24.664113 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:30:24.845594 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:24.845513 4484 patch_prober.go:29] interesting pod/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused" start-of-body= Nov 01 21:30:24.845594 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:24.845562 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk" podUID="fb507845-2edf-4ffa-b057-da162cebddee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused" Nov 01 21:30:25.368639 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:25.368604 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:30:25.845823 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:25.845790 4484 patch_prober.go:29] interesting pod/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused" start-of-body= Nov 01 21:30:25.846268 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:25.845842 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk" podUID="fb507845-2edf-4ffa-b057-da162cebddee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused" Nov 01 21:30:26.845367 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:26.845325 4484 patch_prober.go:29] interesting pod/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused" start-of-body= Nov 01 21:30:26.845576 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:26.845381 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk" podUID="fb507845-2edf-4ffa-b057-da162cebddee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused" Nov 01 21:30:27.328133 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:27.328095 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk" Nov 01 21:30:27.737571 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:27.737483 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:30:27.850009 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:27.849960 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk" Nov 01 21:30:27.851242 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:27.851220 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk" Nov 01 21:30:30.129395 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:30.129364 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:30:34.932657 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:34.932619 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:30:39.722482 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:39.722444 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:30:44.531727 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:44.531686 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:30:45.766430 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:30:45.766397 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:30:49.328231 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:49.328192 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:30:53.449546 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:53.449513 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:30:56.145675 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:56.145638 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:30:58.936702 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:30:58.936666 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:31:02.636169 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:02.636136 4484 scope.go:122] "RemoveContainer" containerID="48db40f94ffc4b64e30aaf555a59b0eda483a0f7ae32efc831cb5fdb3d29be32" Nov 01 21:31:03.748204 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:03.748170 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:31:08.530629 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:08.530585 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:31:12.171035 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:12.170982 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71e16860-jwj5n"] Nov 01 21:31:12.171433 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:12.171408 4484 cpu_manager.go:414] "containerMap: removing container" podUID="3d0e82d9-685b-4e15-81a9-a7c87ee13eb8" containerName="cleanup" Nov 01 21:31:12.171433 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:12.171418 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d0e82d9-685b-4e15-81a9-a7c87ee13eb8" containerName="cleanup" Nov 01 21:31:12.171433 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:12.171432 4484 cpu_manager.go:414] "containerMap: removing container" podUID="3d0e82d9-685b-4e15-81a9-a7c87ee13eb8" containerName="cleanup" Nov 01 21:31:12.171545 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:12.171437 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d0e82d9-685b-4e15-81a9-a7c87ee13eb8" containerName="cleanup" Nov 01 21:31:12.171545 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:12.171466 4484 cpu_manager.go:414] "containerMap: removing container" podUID="3d0e82d9-685b-4e15-81a9-a7c87ee13eb8" containerName="cleanup" Nov 01 21:31:12.171545 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:12.171471 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d0e82d9-685b-4e15-81a9-a7c87ee13eb8" containerName="cleanup" Nov 01 21:31:12.171545 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:12.171528 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d0e82d9-685b-4e15-81a9-a7c87ee13eb8" containerName="cleanup" Nov 01 21:31:12.171545 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:12.171537 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d0e82d9-685b-4e15-81a9-a7c87ee13eb8" containerName="cleanup" Nov 01 21:31:12.171545 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:12.171544 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d0e82d9-685b-4e15-81a9-a7c87ee13eb8" containerName="cleanup" Nov 01 21:31:12.176005 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:12.175978 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e16860-jwj5n" Nov 01 21:31:12.230772 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:12.230735 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71e16860-jwj5n"] Nov 01 21:31:12.263485 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:12.263446 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cbdb6cf-6bf6-461b-b658-6adff40437df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71e16860-jwj5n\" (UID: \"8cbdb6cf-6bf6-461b-b658-6adff40437df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e16860-jwj5n" Nov 01 21:31:12.263681 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:12.263495 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k87wl\" (UniqueName: \"kubernetes.io/projected/8cbdb6cf-6bf6-461b-b658-6adff40437df-kube-api-access-k87wl\") pod \"maas-api-revoke-sub-71e16860-jwj5n\" (UID: \"8cbdb6cf-6bf6-461b-b658-6adff40437df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e16860-jwj5n" Nov 01 21:31:12.263681 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:12.263580 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cbdb6cf-6bf6-461b-b658-6adff40437df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71e16860-jwj5n\" (UID: \"8cbdb6cf-6bf6-461b-b658-6adff40437df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e16860-jwj5n" Nov 01 21:31:12.278891 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:12.278853 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:31:12.278891 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:12.278884 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:31:12.279111 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:12.278883 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:31:12.279111 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:12.278913 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:31:12.364883 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:12.364841 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cbdb6cf-6bf6-461b-b658-6adff40437df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71e16860-jwj5n\" (UID: \"8cbdb6cf-6bf6-461b-b658-6adff40437df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e16860-jwj5n" Nov 01 21:31:12.365116 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:12.364895 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k87wl\" (UniqueName: \"kubernetes.io/projected/8cbdb6cf-6bf6-461b-b658-6adff40437df-kube-api-access-k87wl\") pod \"maas-api-revoke-sub-71e16860-jwj5n\" (UID: \"8cbdb6cf-6bf6-461b-b658-6adff40437df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e16860-jwj5n" Nov 01 21:31:12.365116 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:12.364921 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cbdb6cf-6bf6-461b-b658-6adff40437df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71e16860-jwj5n\" (UID: \"8cbdb6cf-6bf6-461b-b658-6adff40437df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e16860-jwj5n" Nov 01 21:31:12.365584 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:12.365560 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cbdb6cf-6bf6-461b-b658-6adff40437df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71e16860-jwj5n\" (UID: \"8cbdb6cf-6bf6-461b-b658-6adff40437df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e16860-jwj5n" Nov 01 21:31:12.367292 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:12.367267 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cbdb6cf-6bf6-461b-b658-6adff40437df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71e16860-jwj5n\" (UID: \"8cbdb6cf-6bf6-461b-b658-6adff40437df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e16860-jwj5n" Nov 01 21:31:12.422469 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:12.422399 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k87wl\" (UniqueName: \"kubernetes.io/projected/8cbdb6cf-6bf6-461b-b658-6adff40437df-kube-api-access-k87wl\") pod \"maas-api-revoke-sub-71e16860-jwj5n\" (UID: \"8cbdb6cf-6bf6-461b-b658-6adff40437df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e16860-jwj5n" Nov 01 21:31:12.488390 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:12.488359 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e16860-jwj5n" Nov 01 21:31:12.860376 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:12.860353 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71e16860-jwj5n"] Nov 01 21:31:12.862420 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:31:12.862393 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cbdb6cf_6bf6_461b_b658_6adff40437df.slice/crio-b49221350f0bb8ca020776564149ddf640d4aa59b9bf286a6013a34e277e1719 WatchSource:0}: Error finding container b49221350f0bb8ca020776564149ddf640d4aa59b9bf286a6013a34e277e1719: Status 404 returned error can't find the container with id b49221350f0bb8ca020776564149ddf640d4aa59b9bf286a6013a34e277e1719 Nov 01 21:31:13.330687 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:13.330638 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:31:13.740636 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:13.740603 4484 generic.go:356] "Generic (PLEG): container finished" podID="8cbdb6cf-6bf6-461b-b658-6adff40437df" containerID="d2f2b098a7ac1c3981e4475df29591fda74ad03a7318c38d9fbd41eb4a6bc4a6" exitCode=0 Nov 01 21:31:13.740837 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:13.740674 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e16860-jwj5n" event={"ID":"8cbdb6cf-6bf6-461b-b658-6adff40437df","Type":"ContainerDied","Data":"d2f2b098a7ac1c3981e4475df29591fda74ad03a7318c38d9fbd41eb4a6bc4a6"} Nov 01 21:31:13.740837 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:13.740709 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e16860-jwj5n" event={"ID":"8cbdb6cf-6bf6-461b-b658-6adff40437df","Type":"ContainerStarted","Data":"b49221350f0bb8ca020776564149ddf640d4aa59b9bf286a6013a34e277e1719"} Nov 01 21:31:14.948556 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:14.948214 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e16860-jwj5n" Nov 01 21:31:15.112416 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:15.112312 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cbdb6cf-6bf6-461b-b658-6adff40437df-kube-api-access-k87wl\" (UniqueName: \"kubernetes.io/projected/8cbdb6cf-6bf6-461b-b658-6adff40437df-kube-api-access-k87wl\") pod \"8cbdb6cf-6bf6-461b-b658-6adff40437df\" (UID: \"8cbdb6cf-6bf6-461b-b658-6adff40437df\") " Nov 01 21:31:15.112609 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:15.112461 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cbdb6cf-6bf6-461b-b658-6adff40437df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cbdb6cf-6bf6-461b-b658-6adff40437df-maas-api-xks-ca\") pod \"8cbdb6cf-6bf6-461b-b658-6adff40437df\" (UID: \"8cbdb6cf-6bf6-461b-b658-6adff40437df\") " Nov 01 21:31:15.112609 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:15.112505 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cbdb6cf-6bf6-461b-b658-6adff40437df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cbdb6cf-6bf6-461b-b658-6adff40437df-maas-api-openshift-ca\") pod \"8cbdb6cf-6bf6-461b-b658-6adff40437df\" (UID: \"8cbdb6cf-6bf6-461b-b658-6adff40437df\") " Nov 01 21:31:15.113096 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:15.113064 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cbdb6cf-6bf6-461b-b658-6adff40437df-maas-api-openshift-ca" pod "8cbdb6cf-6bf6-461b-b658-6adff40437df" (UID: "8cbdb6cf-6bf6-461b-b658-6adff40437df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:31:15.114622 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:15.114584 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cbdb6cf-6bf6-461b-b658-6adff40437df-kube-api-access-k87wl" pod "8cbdb6cf-6bf6-461b-b658-6adff40437df" (UID: "8cbdb6cf-6bf6-461b-b658-6adff40437df"). InnerVolumeSpecName "kube-api-access-k87wl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:31:15.115047 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:15.114986 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cbdb6cf-6bf6-461b-b658-6adff40437df-maas-api-xks-ca" pod "8cbdb6cf-6bf6-461b-b658-6adff40437df" (UID: "8cbdb6cf-6bf6-461b-b658-6adff40437df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:31:15.213560 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:15.213518 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cbdb6cf-6bf6-461b-b658-6adff40437df-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:31:15.213560 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:15.213554 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cbdb6cf-6bf6-461b-b658-6adff40437df-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:31:15.213560 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:15.213565 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k87wl\" (UniqueName: \"kubernetes.io/projected/8cbdb6cf-6bf6-461b-b658-6adff40437df-kube-api-access-k87wl\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:31:15.751893 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:15.751847 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e16860-jwj5n" event={"ID":"8cbdb6cf-6bf6-461b-b658-6adff40437df","Type":"ContainerDied","Data":"b49221350f0bb8ca020776564149ddf640d4aa59b9bf286a6013a34e277e1719"} Nov 01 21:31:15.752091 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:15.751919 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b49221350f0bb8ca020776564149ddf640d4aa59b9bf286a6013a34e277e1719" Nov 01 21:31:15.752091 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:15.751867 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e16860-jwj5n" Nov 01 21:31:15.767548 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:31:15.767510 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:31:18.135465 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:18.135433 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:31:22.943959 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:22.943916 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:31:23.253644 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:23.253617 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:31:23.745762 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:23.745724 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-236f0f-7szg8"] Nov 01 21:31:23.746199 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:23.746185 4484 cpu_manager.go:414] "containerMap: removing container" podUID="8cbdb6cf-6bf6-461b-b658-6adff40437df" containerName="revoke-keys" Nov 01 21:31:23.746258 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:23.746200 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cbdb6cf-6bf6-461b-b658-6adff40437df" containerName="revoke-keys" Nov 01 21:31:23.746296 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:23.746264 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cbdb6cf-6bf6-461b-b658-6adff40437df" containerName="revoke-keys" Nov 01 21:31:23.749659 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:23.749640 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-236f0f-7szg8" Nov 01 21:31:23.805460 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:23.805415 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-236f0f-7szg8"] Nov 01 21:31:23.853362 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:23.853306 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:31:23.853362 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:23.853317 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:31:23.853580 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:23.853369 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:31:23.853943 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:23.853925 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:31:23.902212 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:23.902172 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46656c20-b77c-4a12-91cf-4cf3919687cd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-236f0f-7szg8\" (UID: \"46656c20-b77c-4a12-91cf-4cf3919687cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-236f0f-7szg8" Nov 01 21:31:23.902393 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:23.902231 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d886f\" (UniqueName: \"kubernetes.io/projected/46656c20-b77c-4a12-91cf-4cf3919687cd-kube-api-access-d886f\") pod \"maas-api-revoke-keys-e2e-worker-single-main-236f0f-7szg8\" (UID: \"46656c20-b77c-4a12-91cf-4cf3919687cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-236f0f-7szg8" Nov 01 21:31:23.902393 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:23.902378 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46656c20-b77c-4a12-91cf-4cf3919687cd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-236f0f-7szg8\" (UID: \"46656c20-b77c-4a12-91cf-4cf3919687cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-236f0f-7szg8" Nov 01 21:31:24.003350 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:24.003258 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46656c20-b77c-4a12-91cf-4cf3919687cd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-236f0f-7szg8\" (UID: \"46656c20-b77c-4a12-91cf-4cf3919687cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-236f0f-7szg8" Nov 01 21:31:24.003350 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:24.003329 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46656c20-b77c-4a12-91cf-4cf3919687cd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-236f0f-7szg8\" (UID: \"46656c20-b77c-4a12-91cf-4cf3919687cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-236f0f-7szg8" Nov 01 21:31:24.003773 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:24.003402 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d886f\" (UniqueName: \"kubernetes.io/projected/46656c20-b77c-4a12-91cf-4cf3919687cd-kube-api-access-d886f\") pod \"maas-api-revoke-keys-e2e-worker-single-main-236f0f-7szg8\" (UID: \"46656c20-b77c-4a12-91cf-4cf3919687cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-236f0f-7szg8" Nov 01 21:31:24.003929 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:24.003907 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46656c20-b77c-4a12-91cf-4cf3919687cd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-236f0f-7szg8\" (UID: \"46656c20-b77c-4a12-91cf-4cf3919687cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-236f0f-7szg8" Nov 01 21:31:24.005883 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:24.005857 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46656c20-b77c-4a12-91cf-4cf3919687cd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-236f0f-7szg8\" (UID: \"46656c20-b77c-4a12-91cf-4cf3919687cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-236f0f-7szg8" Nov 01 21:31:24.060932 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:24.060901 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d886f\" (UniqueName: \"kubernetes.io/projected/46656c20-b77c-4a12-91cf-4cf3919687cd-kube-api-access-d886f\") pod \"maas-api-revoke-keys-e2e-worker-single-main-236f0f-7szg8\" (UID: \"46656c20-b77c-4a12-91cf-4cf3919687cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-236f0f-7szg8" Nov 01 21:31:24.062723 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:24.062706 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-236f0f-7szg8" Nov 01 21:31:24.436808 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:24.436774 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-236f0f-7szg8"] Nov 01 21:31:24.438912 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:31:24.438881 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46656c20_b77c_4a12_91cf_4cf3919687cd.slice/crio-738de03c2c20c583891fdbd5e9285add56f8c081037111a297e9fb8df2ca5f0a WatchSource:0}: Error finding container 738de03c2c20c583891fdbd5e9285add56f8c081037111a297e9fb8df2ca5f0a: Status 404 returned error can't find the container with id 738de03c2c20c583891fdbd5e9285add56f8c081037111a297e9fb8df2ca5f0a Nov 01 21:31:24.796947 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:24.796914 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-236f0f-7szg8" event={"ID":"46656c20-b77c-4a12-91cf-4cf3919687cd","Type":"ContainerStarted","Data":"c8370589ce0af86c4c8f729814d1ed5fe5e04b47f8c7e8ff8143693e21fb2415"} Nov 01 21:31:24.797145 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:24.796966 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-236f0f-7szg8" event={"ID":"46656c20-b77c-4a12-91cf-4cf3919687cd","Type":"ContainerStarted","Data":"738de03c2c20c583891fdbd5e9285add56f8c081037111a297e9fb8df2ca5f0a"} Nov 01 21:31:24.911876 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:24.911821 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-236f0f-7szg8" podStartSLOduration=1.911808024 podStartE2EDuration="1.911808024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:31:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:31:24.910650564 +0000 UTC m=+3635.688290699" watchObservedRunningTime="2026-11-01 21:31:24.911808024 +0000 UTC m=+3635.689448159" Nov 01 21:31:25.803569 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:25.803537 4484 generic.go:356] "Generic (PLEG): container finished" podID="46656c20-b77c-4a12-91cf-4cf3919687cd" containerID="c8370589ce0af86c4c8f729814d1ed5fe5e04b47f8c7e8ff8143693e21fb2415" exitCode=0 Nov 01 21:31:25.804062 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:25.803622 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-236f0f-7szg8" event={"ID":"46656c20-b77c-4a12-91cf-4cf3919687cd","Type":"ContainerDied","Data":"c8370589ce0af86c4c8f729814d1ed5fe5e04b47f8c7e8ff8143693e21fb2415"} Nov 01 21:31:26.943828 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:26.943802 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-236f0f-7szg8" Nov 01 21:31:27.040915 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:27.040874 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46656c20-b77c-4a12-91cf-4cf3919687cd-kube-api-access-d886f\" (UniqueName: \"kubernetes.io/projected/46656c20-b77c-4a12-91cf-4cf3919687cd-kube-api-access-d886f\") pod \"46656c20-b77c-4a12-91cf-4cf3919687cd\" (UID: \"46656c20-b77c-4a12-91cf-4cf3919687cd\") " Nov 01 21:31:27.041135 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:27.040958 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46656c20-b77c-4a12-91cf-4cf3919687cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46656c20-b77c-4a12-91cf-4cf3919687cd-maas-api-xks-ca\") pod \"46656c20-b77c-4a12-91cf-4cf3919687cd\" (UID: \"46656c20-b77c-4a12-91cf-4cf3919687cd\") " Nov 01 21:31:27.041135 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:27.041053 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46656c20-b77c-4a12-91cf-4cf3919687cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46656c20-b77c-4a12-91cf-4cf3919687cd-maas-api-openshift-ca\") pod \"46656c20-b77c-4a12-91cf-4cf3919687cd\" (UID: \"46656c20-b77c-4a12-91cf-4cf3919687cd\") " Nov 01 21:31:27.041459 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:27.041428 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46656c20-b77c-4a12-91cf-4cf3919687cd-maas-api-openshift-ca" pod "46656c20-b77c-4a12-91cf-4cf3919687cd" (UID: "46656c20-b77c-4a12-91cf-4cf3919687cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:31:27.043227 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:27.043202 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46656c20-b77c-4a12-91cf-4cf3919687cd-kube-api-access-d886f" pod "46656c20-b77c-4a12-91cf-4cf3919687cd" (UID: "46656c20-b77c-4a12-91cf-4cf3919687cd"). InnerVolumeSpecName "kube-api-access-d886f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:31:27.043306 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:27.043222 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46656c20-b77c-4a12-91cf-4cf3919687cd-maas-api-xks-ca" pod "46656c20-b77c-4a12-91cf-4cf3919687cd" (UID: "46656c20-b77c-4a12-91cf-4cf3919687cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:31:27.142165 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:27.142053 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d886f\" (UniqueName: \"kubernetes.io/projected/46656c20-b77c-4a12-91cf-4cf3919687cd-kube-api-access-d886f\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:31:27.142165 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:27.142103 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46656c20-b77c-4a12-91cf-4cf3919687cd-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:31:27.142165 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:27.142117 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46656c20-b77c-4a12-91cf-4cf3919687cd-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:31:27.814093 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:27.814059 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-236f0f-7szg8" Nov 01 21:31:27.814273 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:27.814067 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-236f0f-7szg8" event={"ID":"46656c20-b77c-4a12-91cf-4cf3919687cd","Type":"ContainerDied","Data":"738de03c2c20c583891fdbd5e9285add56f8c081037111a297e9fb8df2ca5f0a"} Nov 01 21:31:27.814273 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:27.814185 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="738de03c2c20c583891fdbd5e9285add56f8c081037111a297e9fb8df2ca5f0a" Nov 01 21:31:28.122433 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:28.122261 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:31:28.292654 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:28.292623 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:31:29.045218 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:29.045187 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd00b00d-bbfdh"] Nov 01 21:31:29.095027 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:29.094978 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd00b00d-bbfdh"] Nov 01 21:31:29.370575 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:29.370497 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ded6c12-5f9a-4279-ae09-014bde385042" path="/var/lib/kubelet/pods/7ded6c12-5f9a-4279-ae09-014bde385042/volumes" Nov 01 21:31:33.063708 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:33.063571 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:31:35.352594 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:35.352558 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-236f0f-59cf6t5j8h"] Nov 01 21:31:35.654310 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:35.654224 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:31:35.847413 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:35.847375 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:31:37.730057 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:37.730023 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:31:40.353104 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:40.353060 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-236f0f-59cf6t5j8h" podUID="24119262-952c-4224-985e-a52341194eb3" containerName="payload-pre-processing" containerID="cri-o://01781757d8447abf7d2573ce1090afbf9445d7f47b3439151c7162aa1cb26e1e" gracePeriod=25 Nov 01 21:31:41.933161 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:41.933130 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:31:43.443365 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:43.443324 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:31:43.712648 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:43.712553 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:31:44.132836 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:44.132801 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-236f0f-7szg8"] Nov 01 21:31:44.134613 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:44.134585 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-236f0f-7szg8"] Nov 01 21:31:44.447886 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:44.447802 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:31:45.360087 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:45.360049 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-236f0f-59cf6t5j8h" podUID="24119262-952c-4224-985e-a52341194eb3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 21:31:45.370673 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:45.370646 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46656c20-b77c-4a12-91cf-4cf3919687cd" path="/var/lib/kubelet/pods/46656c20-b77c-4a12-91cf-4cf3919687cd/volumes" Nov 01 21:31:45.767457 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:31:45.767417 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:31:50.053489 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:50.053457 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:31:50.209380 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:50.209347 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:31:50.357015 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:50.356919 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk"] Nov 01 21:31:50.357183 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:50.357136 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk" podUID="fb507845-2edf-4ffa-b057-da162cebddee" containerName="istio-proxy" containerID="cri-o://9b5f6404562c55f40ef93c828810d27880ee9df174902fffd18157ce30c56e93" gracePeriod=30 Nov 01 21:31:50.393235 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:50.393196 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:31:53.836790 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:53.836756 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:31:55.360730 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:55.360692 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-236f0f-59cf6t5j8h" podUID="24119262-952c-4224-985e-a52341194eb3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 21:31:55.605805 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:55.605775 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk" Nov 01 21:31:55.744872 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:55.744843 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb507845-2edf-4ffa-b057-da162cebddee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb507845-2edf-4ffa-b057-da162cebddee-credential-socket\") pod \"fb507845-2edf-4ffa-b057-da162cebddee\" (UID: \"fb507845-2edf-4ffa-b057-da162cebddee\") " Nov 01 21:31:55.745120 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:55.744892 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb507845-2edf-4ffa-b057-da162cebddee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb507845-2edf-4ffa-b057-da162cebddee-istio-data\") pod \"fb507845-2edf-4ffa-b057-da162cebddee\" (UID: \"fb507845-2edf-4ffa-b057-da162cebddee\") " Nov 01 21:31:55.745120 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:55.744925 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb507845-2edf-4ffa-b057-da162cebddee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb507845-2edf-4ffa-b057-da162cebddee-workload-certs\") pod \"fb507845-2edf-4ffa-b057-da162cebddee\" (UID: \"fb507845-2edf-4ffa-b057-da162cebddee\") " Nov 01 21:31:55.745120 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:55.744984 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb507845-2edf-4ffa-b057-da162cebddee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb507845-2edf-4ffa-b057-da162cebddee-istiod-ca-cert\") pod \"fb507845-2edf-4ffa-b057-da162cebddee\" (UID: \"fb507845-2edf-4ffa-b057-da162cebddee\") " Nov 01 21:31:55.745120 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:55.745038 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb507845-2edf-4ffa-b057-da162cebddee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb507845-2edf-4ffa-b057-da162cebddee-workload-socket\") pod \"fb507845-2edf-4ffa-b057-da162cebddee\" (UID: \"fb507845-2edf-4ffa-b057-da162cebddee\") " Nov 01 21:31:55.745120 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:55.745077 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb507845-2edf-4ffa-b057-da162cebddee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb507845-2edf-4ffa-b057-da162cebddee-istio-podinfo\") pod \"fb507845-2edf-4ffa-b057-da162cebddee\" (UID: \"fb507845-2edf-4ffa-b057-da162cebddee\") " Nov 01 21:31:55.745120 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:55.745103 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb507845-2edf-4ffa-b057-da162cebddee-kube-api-access-98rwj\" (UniqueName: \"kubernetes.io/projected/fb507845-2edf-4ffa-b057-da162cebddee-kube-api-access-98rwj\") pod \"fb507845-2edf-4ffa-b057-da162cebddee\" (UID: \"fb507845-2edf-4ffa-b057-da162cebddee\") " Nov 01 21:31:55.745464 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:55.745134 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb507845-2edf-4ffa-b057-da162cebddee-istio-token\" (UniqueName: \"kubernetes.io/projected/fb507845-2edf-4ffa-b057-da162cebddee-istio-token\") pod \"fb507845-2edf-4ffa-b057-da162cebddee\" (UID: \"fb507845-2edf-4ffa-b057-da162cebddee\") " Nov 01 21:31:55.745464 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:55.745207 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb507845-2edf-4ffa-b057-da162cebddee-credential-socket" pod "fb507845-2edf-4ffa-b057-da162cebddee" (UID: "fb507845-2edf-4ffa-b057-da162cebddee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:31:55.745464 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:55.745280 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb507845-2edf-4ffa-b057-da162cebddee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb507845-2edf-4ffa-b057-da162cebddee-istio-envoy\") pod \"fb507845-2edf-4ffa-b057-da162cebddee\" (UID: \"fb507845-2edf-4ffa-b057-da162cebddee\") " Nov 01 21:31:55.745464 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:55.745342 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb507845-2edf-4ffa-b057-da162cebddee-istio-data" pod "fb507845-2edf-4ffa-b057-da162cebddee" (UID: "fb507845-2edf-4ffa-b057-da162cebddee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:31:55.745464 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:55.745393 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb507845-2edf-4ffa-b057-da162cebddee-workload-certs" pod "fb507845-2edf-4ffa-b057-da162cebddee" (UID: "fb507845-2edf-4ffa-b057-da162cebddee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:31:55.745464 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:55.745417 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb507845-2edf-4ffa-b057-da162cebddee-workload-socket" pod "fb507845-2edf-4ffa-b057-da162cebddee" (UID: "fb507845-2edf-4ffa-b057-da162cebddee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:31:55.745464 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:55.745455 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb507845-2edf-4ffa-b057-da162cebddee-istiod-ca-cert" pod "fb507845-2edf-4ffa-b057-da162cebddee" (UID: "fb507845-2edf-4ffa-b057-da162cebddee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:31:55.745818 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:55.745766 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb507845-2edf-4ffa-b057-da162cebddee-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:31:55.745818 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:55.745788 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb507845-2edf-4ffa-b057-da162cebddee-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:31:55.745818 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:55.745802 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb507845-2edf-4ffa-b057-da162cebddee-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:31:55.745818 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:55.745815 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb507845-2edf-4ffa-b057-da162cebddee-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:31:55.745989 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:55.745828 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb507845-2edf-4ffa-b057-da162cebddee-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:31:55.747474 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:55.747452 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb507845-2edf-4ffa-b057-da162cebddee-istio-envoy" pod "fb507845-2edf-4ffa-b057-da162cebddee" (UID: "fb507845-2edf-4ffa-b057-da162cebddee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:31:55.747574 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:55.747529 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb507845-2edf-4ffa-b057-da162cebddee-kube-api-access-98rwj" pod "fb507845-2edf-4ffa-b057-da162cebddee" (UID: "fb507845-2edf-4ffa-b057-da162cebddee"). InnerVolumeSpecName "kube-api-access-98rwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:31:55.747825 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:55.747802 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb507845-2edf-4ffa-b057-da162cebddee-istio-podinfo" pod "fb507845-2edf-4ffa-b057-da162cebddee" (UID: "fb507845-2edf-4ffa-b057-da162cebddee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:31:55.747900 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:55.747809 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb507845-2edf-4ffa-b057-da162cebddee-istio-token" pod "fb507845-2edf-4ffa-b057-da162cebddee" (UID: "fb507845-2edf-4ffa-b057-da162cebddee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:31:55.847411 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:55.847373 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb507845-2edf-4ffa-b057-da162cebddee-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:31:55.847411 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:55.847406 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98rwj\" (UniqueName: \"kubernetes.io/projected/fb507845-2edf-4ffa-b057-da162cebddee-kube-api-access-98rwj\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:31:55.847411 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:55.847417 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb507845-2edf-4ffa-b057-da162cebddee-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:31:55.847672 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:55.847426 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb507845-2edf-4ffa-b057-da162cebddee-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:31:55.966764 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:55.966723 4484 generic.go:356] "Generic (PLEG): container finished" podID="fb507845-2edf-4ffa-b057-da162cebddee" containerID="9b5f6404562c55f40ef93c828810d27880ee9df174902fffd18157ce30c56e93" exitCode=0 Nov 01 21:31:55.966969 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:55.966775 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk" Nov 01 21:31:55.966969 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:55.966776 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk" event={"ID":"fb507845-2edf-4ffa-b057-da162cebddee","Type":"ContainerDied","Data":"9b5f6404562c55f40ef93c828810d27880ee9df174902fffd18157ce30c56e93"} Nov 01 21:31:55.966969 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:55.966904 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk" event={"ID":"fb507845-2edf-4ffa-b057-da162cebddee","Type":"ContainerDied","Data":"a801dc068233497fe889c9727e5f88f9ac2b9c1c18f1be5e8315b82d46bd1be0"} Nov 01 21:31:55.966969 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:55.966930 4484 scope.go:122] "RemoveContainer" containerID="9b5f6404562c55f40ef93c828810d27880ee9df174902fffd18157ce30c56e93" Nov 01 21:31:55.976704 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:55.976683 4484 scope.go:122] "RemoveContainer" containerID="801b18ee563f72bb54d212dd5ade7744675943cfe32f71fa896d984eae85b073" Nov 01 21:31:55.984986 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:55.984960 4484 scope.go:122] "RemoveContainer" containerID="9b5f6404562c55f40ef93c828810d27880ee9df174902fffd18157ce30c56e93" Nov 01 21:31:55.985240 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:31:55.985220 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b5f6404562c55f40ef93c828810d27880ee9df174902fffd18157ce30c56e93\": container with ID starting with 9b5f6404562c55f40ef93c828810d27880ee9df174902fffd18157ce30c56e93 not found: ID does not exist" containerID="9b5f6404562c55f40ef93c828810d27880ee9df174902fffd18157ce30c56e93" Nov 01 21:31:55.985296 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:55.985249 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b5f6404562c55f40ef93c828810d27880ee9df174902fffd18157ce30c56e93"} err="failed to get container status \"9b5f6404562c55f40ef93c828810d27880ee9df174902fffd18157ce30c56e93\": rpc error: code = NotFound desc = could not find container \"9b5f6404562c55f40ef93c828810d27880ee9df174902fffd18157ce30c56e93\": container with ID starting with 9b5f6404562c55f40ef93c828810d27880ee9df174902fffd18157ce30c56e93 not found: ID does not exist" Nov 01 21:31:55.985296 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:55.985267 4484 scope.go:122] "RemoveContainer" containerID="801b18ee563f72bb54d212dd5ade7744675943cfe32f71fa896d984eae85b073" Nov 01 21:31:55.985490 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:31:55.985471 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"801b18ee563f72bb54d212dd5ade7744675943cfe32f71fa896d984eae85b073\": container with ID starting with 801b18ee563f72bb54d212dd5ade7744675943cfe32f71fa896d984eae85b073 not found: ID does not exist" containerID="801b18ee563f72bb54d212dd5ade7744675943cfe32f71fa896d984eae85b073" Nov 01 21:31:55.985530 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:55.985497 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"801b18ee563f72bb54d212dd5ade7744675943cfe32f71fa896d984eae85b073"} err="failed to get container status \"801b18ee563f72bb54d212dd5ade7744675943cfe32f71fa896d984eae85b073\": rpc error: code = NotFound desc = could not find container \"801b18ee563f72bb54d212dd5ade7744675943cfe32f71fa896d984eae85b073\": container with ID starting with 801b18ee563f72bb54d212dd5ade7744675943cfe32f71fa896d984eae85b073 not found: ID does not exist" Nov 01 21:31:56.133288 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:56.133194 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk"] Nov 01 21:31:56.138865 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:56.138839 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-236f0f-openshift-default-689d7dd8c7jnbkk"] Nov 01 21:31:56.484201 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:56.484127 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:31:57.370505 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:31:57.370475 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb507845-2edf-4ffa-b057-da162cebddee" path="/var/lib/kubelet/pods/fb507845-2edf-4ffa-b057-da162cebddee/volumes" Nov 01 21:32:00.600800 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:32:00.600769 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-236f0f-59cf6t5j8h" Nov 01 21:32:00.697687 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:32:00.697595 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24119262-952c-4224-985e-a52341194eb3-kube-api-access-56vnd\" (UniqueName: \"kubernetes.io/projected/24119262-952c-4224-985e-a52341194eb3-kube-api-access-56vnd\") pod \"24119262-952c-4224-985e-a52341194eb3\" (UID: \"24119262-952c-4224-985e-a52341194eb3\") " Nov 01 21:32:00.697880 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:32:00.697703 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24119262-952c-4224-985e-a52341194eb3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24119262-952c-4224-985e-a52341194eb3-plugins-config-volume\") pod \"24119262-952c-4224-985e-a52341194eb3\" (UID: \"24119262-952c-4224-985e-a52341194eb3\") " Nov 01 21:32:00.698163 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:32:00.698132 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24119262-952c-4224-985e-a52341194eb3-plugins-config-volume" pod "24119262-952c-4224-985e-a52341194eb3" (UID: "24119262-952c-4224-985e-a52341194eb3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:32:00.700006 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:32:00.699969 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24119262-952c-4224-985e-a52341194eb3-kube-api-access-56vnd" pod "24119262-952c-4224-985e-a52341194eb3" (UID: "24119262-952c-4224-985e-a52341194eb3"). InnerVolumeSpecName "kube-api-access-56vnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:32:00.798734 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:32:00.798697 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24119262-952c-4224-985e-a52341194eb3-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:32:00.798734 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:32:00.798728 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56vnd\" (UniqueName: \"kubernetes.io/projected/24119262-952c-4224-985e-a52341194eb3-kube-api-access-56vnd\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:32:00.995244 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:32:00.995210 4484 generic.go:356] "Generic (PLEG): container finished" podID="24119262-952c-4224-985e-a52341194eb3" containerID="01781757d8447abf7d2573ce1090afbf9445d7f47b3439151c7162aa1cb26e1e" exitCode=0 Nov 01 21:32:00.995453 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:32:00.995282 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-236f0f-59cf6t5j8h" Nov 01 21:32:00.995453 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:32:00.995299 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-236f0f-59cf6t5j8h" event={"ID":"24119262-952c-4224-985e-a52341194eb3","Type":"ContainerDied","Data":"01781757d8447abf7d2573ce1090afbf9445d7f47b3439151c7162aa1cb26e1e"} Nov 01 21:32:00.995453 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:32:00.995365 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-236f0f-59cf6t5j8h" event={"ID":"24119262-952c-4224-985e-a52341194eb3","Type":"ContainerDied","Data":"e9c0019dfcce9dcbdd55042c18bf7a3b13f2c7049211222ebc8fba1b40b66890"} Nov 01 21:32:00.995453 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:32:00.995395 4484 scope.go:122] "RemoveContainer" containerID="01781757d8447abf7d2573ce1090afbf9445d7f47b3439151c7162aa1cb26e1e" Nov 01 21:32:01.005032 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:32:01.004986 4484 scope.go:122] "RemoveContainer" containerID="01781757d8447abf7d2573ce1090afbf9445d7f47b3439151c7162aa1cb26e1e" Nov 01 21:32:01.005370 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:32:01.005351 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01781757d8447abf7d2573ce1090afbf9445d7f47b3439151c7162aa1cb26e1e\": container with ID starting with 01781757d8447abf7d2573ce1090afbf9445d7f47b3439151c7162aa1cb26e1e not found: ID does not exist" containerID="01781757d8447abf7d2573ce1090afbf9445d7f47b3439151c7162aa1cb26e1e" Nov 01 21:32:01.005433 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:32:01.005378 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01781757d8447abf7d2573ce1090afbf9445d7f47b3439151c7162aa1cb26e1e"} err="failed to get container status \"01781757d8447abf7d2573ce1090afbf9445d7f47b3439151c7162aa1cb26e1e\": rpc error: code = NotFound desc = could not find container \"01781757d8447abf7d2573ce1090afbf9445d7f47b3439151c7162aa1cb26e1e\": container with ID starting with 01781757d8447abf7d2573ce1090afbf9445d7f47b3439151c7162aa1cb26e1e not found: ID does not exist" Nov 01 21:32:01.172534 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:32:01.172493 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-236f0f-59cf6t5j8h"] Nov 01 21:32:01.223034 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:32:01.222990 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-236f0f-59cf6t5j8h"] Nov 01 21:32:01.411257 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:32:01.411162 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24119262-952c-4224-985e-a52341194eb3" path="/var/lib/kubelet/pods/24119262-952c-4224-985e-a52341194eb3/volumes" Nov 01 21:32:04.328020 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:32:04.327962 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:32:05.057441 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:32:05.057408 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:32:09.044011 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:32:09.043970 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g49bd"] Nov 01 21:32:09.094821 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:32:09.094787 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g49bd"] Nov 01 21:32:09.370059 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:32:09.369976 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c88c6fad-5b4f-491c-aaa5-27d33b283112" path="/var/lib/kubelet/pods/c88c6fad-5b4f-491c-aaa5-27d33b283112/volumes" Nov 01 21:32:15.767765 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:32:15.767729 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:32:27.474979 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:32:27.474927 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:32:41.140012 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:32:41.139968 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:32:45.766184 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:32:45.766144 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:32:54.137465 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:32:54.137428 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:33:02.870301 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:02.870221 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nqq79"] Nov 01 21:33:02.870766 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:02.870750 4484 cpu_manager.go:414] "containerMap: removing container" podUID="46656c20-b77c-4a12-91cf-4cf3919687cd" containerName="revoke-keys" Nov 01 21:33:02.870821 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:02.870767 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46656c20-b77c-4a12-91cf-4cf3919687cd" containerName="revoke-keys" Nov 01 21:33:02.870821 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:02.870788 4484 cpu_manager.go:414] "containerMap: removing container" podUID="fb507845-2edf-4ffa-b057-da162cebddee" containerName="istio-proxy" Nov 01 21:33:02.870821 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:02.870794 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb507845-2edf-4ffa-b057-da162cebddee" containerName="istio-proxy" Nov 01 21:33:02.870821 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:02.870805 4484 cpu_manager.go:414] "containerMap: removing container" podUID="fb507845-2edf-4ffa-b057-da162cebddee" containerName="istio-proxy" Nov 01 21:33:02.870821 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:02.870810 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb507845-2edf-4ffa-b057-da162cebddee" containerName="istio-proxy" Nov 01 21:33:02.870821 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:02.870818 4484 cpu_manager.go:414] "containerMap: removing container" podUID="24119262-952c-4224-985e-a52341194eb3" containerName="payload-pre-processing" Nov 01 21:33:02.870821 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:02.870823 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24119262-952c-4224-985e-a52341194eb3" containerName="payload-pre-processing" Nov 01 21:33:02.871093 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:02.870909 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb507845-2edf-4ffa-b057-da162cebddee" containerName="istio-proxy" Nov 01 21:33:02.871093 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:02.870917 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="24119262-952c-4224-985e-a52341194eb3" containerName="payload-pre-processing" Nov 01 21:33:02.871093 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:02.870929 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="46656c20-b77c-4a12-91cf-4cf3919687cd" containerName="revoke-keys" Nov 01 21:33:02.873838 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:02.873816 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nqq79" Nov 01 21:33:02.911439 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:02.911399 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbjn4\" (UniqueName: \"kubernetes.io/projected/8fbeb785-f1ad-4f9a-8246-6e00398f3ffa-kube-api-access-fbjn4\") pod \"maas-api-revoke-sub-452b6d7b-nqq79\" (UID: \"8fbeb785-f1ad-4f9a-8246-6e00398f3ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nqq79" Nov 01 21:33:02.911635 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:02.911546 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fbeb785-f1ad-4f9a-8246-6e00398f3ffa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-nqq79\" (UID: \"8fbeb785-f1ad-4f9a-8246-6e00398f3ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nqq79" Nov 01 21:33:02.911686 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:02.911645 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fbeb785-f1ad-4f9a-8246-6e00398f3ffa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-nqq79\" (UID: \"8fbeb785-f1ad-4f9a-8246-6e00398f3ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nqq79" Nov 01 21:33:02.920900 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:02.920871 4484 scope.go:122] "RemoveContainer" containerID="d2374531fe047fe69f08f32dcdb5d41fe6ac042c3da02fb96f7d3ea239aa3908" Nov 01 21:33:02.930301 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:02.930265 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nqq79"] Nov 01 21:33:02.931269 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:02.931177 4484 scope.go:122] "RemoveContainer" containerID="f218ac21528d52f316b51affa0f892d1754d77df85ef052b7c5157c876742fb6" Nov 01 21:33:02.978685 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:02.978651 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:33:02.978685 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:02.978658 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:33:02.978911 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:02.978729 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:33:02.978911 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:02.978797 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:33:03.012867 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:03.012826 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fbeb785-f1ad-4f9a-8246-6e00398f3ffa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-nqq79\" (UID: \"8fbeb785-f1ad-4f9a-8246-6e00398f3ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nqq79" Nov 01 21:33:03.012867 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:03.012870 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbjn4\" (UniqueName: \"kubernetes.io/projected/8fbeb785-f1ad-4f9a-8246-6e00398f3ffa-kube-api-access-fbjn4\") pod \"maas-api-revoke-sub-452b6d7b-nqq79\" (UID: \"8fbeb785-f1ad-4f9a-8246-6e00398f3ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nqq79" Nov 01 21:33:03.013181 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:03.012943 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fbeb785-f1ad-4f9a-8246-6e00398f3ffa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-nqq79\" (UID: \"8fbeb785-f1ad-4f9a-8246-6e00398f3ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nqq79" Nov 01 21:33:03.013616 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:03.013593 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fbeb785-f1ad-4f9a-8246-6e00398f3ffa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-nqq79\" (UID: \"8fbeb785-f1ad-4f9a-8246-6e00398f3ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nqq79" Nov 01 21:33:03.015241 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:03.015221 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fbeb785-f1ad-4f9a-8246-6e00398f3ffa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-nqq79\" (UID: \"8fbeb785-f1ad-4f9a-8246-6e00398f3ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nqq79" Nov 01 21:33:03.070688 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:03.070651 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbjn4\" (UniqueName: \"kubernetes.io/projected/8fbeb785-f1ad-4f9a-8246-6e00398f3ffa-kube-api-access-fbjn4\") pod \"maas-api-revoke-sub-452b6d7b-nqq79\" (UID: \"8fbeb785-f1ad-4f9a-8246-6e00398f3ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nqq79" Nov 01 21:33:03.187474 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:03.187382 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nqq79" Nov 01 21:33:03.564080 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:03.564043 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nqq79"] Nov 01 21:33:03.566079 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:33:03.566047 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fbeb785_f1ad_4f9a_8246_6e00398f3ffa.slice/crio-cc9e63f1837c4f1d0a74867a58b9a0e61841a63efdf085153bfc52a4cdca546b WatchSource:0}: Error finding container cc9e63f1837c4f1d0a74867a58b9a0e61841a63efdf085153bfc52a4cdca546b: Status 404 returned error can't find the container with id cc9e63f1837c4f1d0a74867a58b9a0e61841a63efdf085153bfc52a4cdca546b Nov 01 21:33:04.361309 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:04.361275 4484 generic.go:356] "Generic (PLEG): container finished" podID="8fbeb785-f1ad-4f9a-8246-6e00398f3ffa" containerID="4ea1e931e7da2d0e64b6d0556453011e8432361d800cfd34b8cb2e618413ebdb" exitCode=0 Nov 01 21:33:04.361683 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:04.361326 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nqq79" event={"ID":"8fbeb785-f1ad-4f9a-8246-6e00398f3ffa","Type":"ContainerDied","Data":"4ea1e931e7da2d0e64b6d0556453011e8432361d800cfd34b8cb2e618413ebdb"} Nov 01 21:33:04.361683 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:04.361369 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nqq79" event={"ID":"8fbeb785-f1ad-4f9a-8246-6e00398f3ffa","Type":"ContainerStarted","Data":"cc9e63f1837c4f1d0a74867a58b9a0e61841a63efdf085153bfc52a4cdca546b"} Nov 01 21:33:05.573656 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:05.573620 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nqq79" Nov 01 21:33:05.641802 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:05.641707 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fbeb785-f1ad-4f9a-8246-6e00398f3ffa-kube-api-access-fbjn4\" (UniqueName: \"kubernetes.io/projected/8fbeb785-f1ad-4f9a-8246-6e00398f3ffa-kube-api-access-fbjn4\") pod \"8fbeb785-f1ad-4f9a-8246-6e00398f3ffa\" (UID: \"8fbeb785-f1ad-4f9a-8246-6e00398f3ffa\") " Nov 01 21:33:05.642009 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:05.641822 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fbeb785-f1ad-4f9a-8246-6e00398f3ffa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fbeb785-f1ad-4f9a-8246-6e00398f3ffa-maas-api-openshift-ca\") pod \"8fbeb785-f1ad-4f9a-8246-6e00398f3ffa\" (UID: \"8fbeb785-f1ad-4f9a-8246-6e00398f3ffa\") " Nov 01 21:33:05.642009 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:05.641899 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fbeb785-f1ad-4f9a-8246-6e00398f3ffa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fbeb785-f1ad-4f9a-8246-6e00398f3ffa-maas-api-xks-ca\") pod \"8fbeb785-f1ad-4f9a-8246-6e00398f3ffa\" (UID: \"8fbeb785-f1ad-4f9a-8246-6e00398f3ffa\") " Nov 01 21:33:05.642594 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:05.642403 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fbeb785-f1ad-4f9a-8246-6e00398f3ffa-maas-api-openshift-ca" pod "8fbeb785-f1ad-4f9a-8246-6e00398f3ffa" (UID: "8fbeb785-f1ad-4f9a-8246-6e00398f3ffa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:33:05.646496 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:05.646448 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fbeb785-f1ad-4f9a-8246-6e00398f3ffa-kube-api-access-fbjn4" pod "8fbeb785-f1ad-4f9a-8246-6e00398f3ffa" (UID: "8fbeb785-f1ad-4f9a-8246-6e00398f3ffa"). InnerVolumeSpecName "kube-api-access-fbjn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:33:05.646803 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:05.646765 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fbeb785-f1ad-4f9a-8246-6e00398f3ffa-maas-api-xks-ca" pod "8fbeb785-f1ad-4f9a-8246-6e00398f3ffa" (UID: "8fbeb785-f1ad-4f9a-8246-6e00398f3ffa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:33:05.743699 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:05.743646 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fbeb785-f1ad-4f9a-8246-6e00398f3ffa-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:33:05.743699 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:05.743693 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fbeb785-f1ad-4f9a-8246-6e00398f3ffa-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:33:05.743985 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:05.743712 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbjn4\" (UniqueName: \"kubernetes.io/projected/8fbeb785-f1ad-4f9a-8246-6e00398f3ffa-kube-api-access-fbjn4\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:33:06.402426 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:06.395084 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nqq79" event={"ID":"8fbeb785-f1ad-4f9a-8246-6e00398f3ffa","Type":"ContainerDied","Data":"cc9e63f1837c4f1d0a74867a58b9a0e61841a63efdf085153bfc52a4cdca546b"} Nov 01 21:33:06.402426 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:06.395152 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc9e63f1837c4f1d0a74867a58b9a0e61841a63efdf085153bfc52a4cdca546b" Nov 01 21:33:06.402426 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:06.395223 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nqq79" Nov 01 21:33:15.678548 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:15.678513 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:33:15.786497 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:33:15.786461 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:33:30.313303 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:30.313262 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:33:34.331655 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:34.331619 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xqsxd"] Nov 01 21:33:34.332109 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:34.332095 4484 cpu_manager.go:414] "containerMap: removing container" podUID="8fbeb785-f1ad-4f9a-8246-6e00398f3ffa" containerName="revoke-keys" Nov 01 21:33:34.332162 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:34.332110 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fbeb785-f1ad-4f9a-8246-6e00398f3ffa" containerName="revoke-keys" Nov 01 21:33:34.332206 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:34.332165 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb507845-2edf-4ffa-b057-da162cebddee" containerName="istio-proxy" Nov 01 21:33:34.332206 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:34.332180 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fbeb785-f1ad-4f9a-8246-6e00398f3ffa" containerName="revoke-keys" Nov 01 21:33:34.335201 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:34.335184 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xqsxd" Nov 01 21:33:34.389583 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:34.389549 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:33:34.389583 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:34.389560 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:33:34.392670 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:34.392639 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xqsxd"] Nov 01 21:33:34.439265 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:34.439225 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb2ed355-e8f4-425a-be5d-24ff5b99d9ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-xqsxd\" (UID: \"bb2ed355-e8f4-425a-be5d-24ff5b99d9ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xqsxd" Nov 01 21:33:34.439450 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:34.439338 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcdvz\" (UniqueName: \"kubernetes.io/projected/bb2ed355-e8f4-425a-be5d-24ff5b99d9ce-kube-api-access-xcdvz\") pod \"maas-api-revoke-sub-3e31a506-xqsxd\" (UID: \"bb2ed355-e8f4-425a-be5d-24ff5b99d9ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xqsxd" Nov 01 21:33:34.439450 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:34.439428 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb2ed355-e8f4-425a-be5d-24ff5b99d9ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-xqsxd\" (UID: \"bb2ed355-e8f4-425a-be5d-24ff5b99d9ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xqsxd" Nov 01 21:33:34.440317 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:34.440293 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:33:34.440441 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:34.440403 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:33:34.540423 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:34.540390 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb2ed355-e8f4-425a-be5d-24ff5b99d9ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-xqsxd\" (UID: \"bb2ed355-e8f4-425a-be5d-24ff5b99d9ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xqsxd" Nov 01 21:33:34.540626 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:34.540569 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb2ed355-e8f4-425a-be5d-24ff5b99d9ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-xqsxd\" (UID: \"bb2ed355-e8f4-425a-be5d-24ff5b99d9ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xqsxd" Nov 01 21:33:34.540626 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:34.540604 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcdvz\" (UniqueName: \"kubernetes.io/projected/bb2ed355-e8f4-425a-be5d-24ff5b99d9ce-kube-api-access-xcdvz\") pod \"maas-api-revoke-sub-3e31a506-xqsxd\" (UID: \"bb2ed355-e8f4-425a-be5d-24ff5b99d9ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xqsxd" Nov 01 21:33:34.541304 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:34.541281 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb2ed355-e8f4-425a-be5d-24ff5b99d9ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-xqsxd\" (UID: \"bb2ed355-e8f4-425a-be5d-24ff5b99d9ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xqsxd" Nov 01 21:33:34.542889 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:34.542858 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb2ed355-e8f4-425a-be5d-24ff5b99d9ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-xqsxd\" (UID: \"bb2ed355-e8f4-425a-be5d-24ff5b99d9ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xqsxd" Nov 01 21:33:34.598723 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:34.598650 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcdvz\" (UniqueName: \"kubernetes.io/projected/bb2ed355-e8f4-425a-be5d-24ff5b99d9ce-kube-api-access-xcdvz\") pod \"maas-api-revoke-sub-3e31a506-xqsxd\" (UID: \"bb2ed355-e8f4-425a-be5d-24ff5b99d9ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xqsxd" Nov 01 21:33:34.648477 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:34.648441 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xqsxd" Nov 01 21:33:35.028954 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:35.028918 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xqsxd"] Nov 01 21:33:35.574706 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:35.574594 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xqsxd" event={"ID":"bb2ed355-e8f4-425a-be5d-24ff5b99d9ce","Type":"ContainerStarted","Data":"6431f6a39f20d540eefda0beb35853cf8cbdc958f1ce880ce201ece67e31341e"} Nov 01 21:33:35.574706 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:35.574676 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xqsxd" event={"ID":"bb2ed355-e8f4-425a-be5d-24ff5b99d9ce","Type":"ContainerStarted","Data":"e8719406eb0e3a7cf37a311dce00ece3635dd9a128044ad322835eaf9cc67fb4"} Nov 01 21:33:35.691884 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:35.691805 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xqsxd" podStartSLOduration=1.6917703240000002 podStartE2EDuration="1.691770324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:33:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:33:35.690083788 +0000 UTC m=+3766.467723948" watchObservedRunningTime="2026-11-01 21:33:35.691770324 +0000 UTC m=+3766.469410460" Nov 01 21:33:36.592019 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:36.591294 4484 generic.go:356] "Generic (PLEG): container finished" podID="bb2ed355-e8f4-425a-be5d-24ff5b99d9ce" containerID="6431f6a39f20d540eefda0beb35853cf8cbdc958f1ce880ce201ece67e31341e" exitCode=0 Nov 01 21:33:36.592019 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:36.591472 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xqsxd" event={"ID":"bb2ed355-e8f4-425a-be5d-24ff5b99d9ce","Type":"ContainerDied","Data":"6431f6a39f20d540eefda0beb35853cf8cbdc958f1ce880ce201ece67e31341e"} Nov 01 21:33:37.847213 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:37.847183 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xqsxd" Nov 01 21:33:37.927650 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:37.927565 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb2ed355-e8f4-425a-be5d-24ff5b99d9ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb2ed355-e8f4-425a-be5d-24ff5b99d9ce-maas-api-xks-ca\") pod \"bb2ed355-e8f4-425a-be5d-24ff5b99d9ce\" (UID: \"bb2ed355-e8f4-425a-be5d-24ff5b99d9ce\") " Nov 01 21:33:37.927650 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:37.927609 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb2ed355-e8f4-425a-be5d-24ff5b99d9ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb2ed355-e8f4-425a-be5d-24ff5b99d9ce-maas-api-openshift-ca\") pod \"bb2ed355-e8f4-425a-be5d-24ff5b99d9ce\" (UID: \"bb2ed355-e8f4-425a-be5d-24ff5b99d9ce\") " Nov 01 21:33:37.927899 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:37.927680 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb2ed355-e8f4-425a-be5d-24ff5b99d9ce-kube-api-access-xcdvz\" (UniqueName: \"kubernetes.io/projected/bb2ed355-e8f4-425a-be5d-24ff5b99d9ce-kube-api-access-xcdvz\") pod \"bb2ed355-e8f4-425a-be5d-24ff5b99d9ce\" (UID: \"bb2ed355-e8f4-425a-be5d-24ff5b99d9ce\") " Nov 01 21:33:37.928072 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:37.928048 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb2ed355-e8f4-425a-be5d-24ff5b99d9ce-maas-api-openshift-ca" pod "bb2ed355-e8f4-425a-be5d-24ff5b99d9ce" (UID: "bb2ed355-e8f4-425a-be5d-24ff5b99d9ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:33:37.929669 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:37.929650 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb2ed355-e8f4-425a-be5d-24ff5b99d9ce-maas-api-xks-ca" pod "bb2ed355-e8f4-425a-be5d-24ff5b99d9ce" (UID: "bb2ed355-e8f4-425a-be5d-24ff5b99d9ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:33:37.929759 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:37.929733 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb2ed355-e8f4-425a-be5d-24ff5b99d9ce-kube-api-access-xcdvz" pod "bb2ed355-e8f4-425a-be5d-24ff5b99d9ce" (UID: "bb2ed355-e8f4-425a-be5d-24ff5b99d9ce"). InnerVolumeSpecName "kube-api-access-xcdvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:33:38.029368 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:38.029331 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb2ed355-e8f4-425a-be5d-24ff5b99d9ce-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:33:38.029368 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:38.029362 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb2ed355-e8f4-425a-be5d-24ff5b99d9ce-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:33:38.029368 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:38.029373 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcdvz\" (UniqueName: \"kubernetes.io/projected/bb2ed355-e8f4-425a-be5d-24ff5b99d9ce-kube-api-access-xcdvz\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:33:38.614713 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:38.614682 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xqsxd" Nov 01 21:33:38.614713 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:38.614703 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xqsxd" event={"ID":"bb2ed355-e8f4-425a-be5d-24ff5b99d9ce","Type":"ContainerDied","Data":"e8719406eb0e3a7cf37a311dce00ece3635dd9a128044ad322835eaf9cc67fb4"} Nov 01 21:33:38.614947 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:38.614748 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8719406eb0e3a7cf37a311dce00ece3635dd9a128044ad322835eaf9cc67fb4" Nov 01 21:33:45.765861 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:33:45.765822 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:33:54.531019 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:33:54.530684 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:34:02.960820 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:02.960786 4484 scope.go:122] "RemoveContainer" containerID="8093b107fa282245d9d037392ff0a5603d8d9493186b9320fbccf9af6ce25e4a" Nov 01 21:34:07.823644 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:07.823606 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8a7e60d-zzftn"] Nov 01 21:34:07.824259 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:07.824239 4484 cpu_manager.go:414] "containerMap: removing container" podUID="bb2ed355-e8f4-425a-be5d-24ff5b99d9ce" containerName="revoke-keys" Nov 01 21:34:07.824358 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:07.824262 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb2ed355-e8f4-425a-be5d-24ff5b99d9ce" containerName="revoke-keys" Nov 01 21:34:07.824358 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:07.824354 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb2ed355-e8f4-425a-be5d-24ff5b99d9ce" containerName="revoke-keys" Nov 01 21:34:07.829083 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:07.829059 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8a7e60d-zzftn" Nov 01 21:34:07.882824 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:07.882788 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:34:07.885282 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:07.885252 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8a7e60d-zzftn"] Nov 01 21:34:07.925255 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:07.925228 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bdb6494-4dfd-4ff8-9037-e65c81ccd574-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8a7e60d-zzftn\" (UID: \"9bdb6494-4dfd-4ff8-9037-e65c81ccd574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8a7e60d-zzftn" Nov 01 21:34:07.925442 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:07.925301 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bdb6494-4dfd-4ff8-9037-e65c81ccd574-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8a7e60d-zzftn\" (UID: \"9bdb6494-4dfd-4ff8-9037-e65c81ccd574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8a7e60d-zzftn" Nov 01 21:34:07.925442 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:07.925396 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nsbv\" (UniqueName: \"kubernetes.io/projected/9bdb6494-4dfd-4ff8-9037-e65c81ccd574-kube-api-access-4nsbv\") pod \"maas-api-revoke-sub-d8a7e60d-zzftn\" (UID: \"9bdb6494-4dfd-4ff8-9037-e65c81ccd574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8a7e60d-zzftn" Nov 01 21:34:07.933399 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:07.933382 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dhfs2\"" Nov 01 21:34:07.933470 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:07.933412 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:34:07.933470 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:07.933420 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:34:08.026071 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:08.025989 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bdb6494-4dfd-4ff8-9037-e65c81ccd574-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8a7e60d-zzftn\" (UID: \"9bdb6494-4dfd-4ff8-9037-e65c81ccd574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8a7e60d-zzftn" Nov 01 21:34:08.026309 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:08.026148 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bdb6494-4dfd-4ff8-9037-e65c81ccd574-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8a7e60d-zzftn\" (UID: \"9bdb6494-4dfd-4ff8-9037-e65c81ccd574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8a7e60d-zzftn" Nov 01 21:34:08.026309 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:08.026191 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nsbv\" (UniqueName: \"kubernetes.io/projected/9bdb6494-4dfd-4ff8-9037-e65c81ccd574-kube-api-access-4nsbv\") pod \"maas-api-revoke-sub-d8a7e60d-zzftn\" (UID: \"9bdb6494-4dfd-4ff8-9037-e65c81ccd574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8a7e60d-zzftn" Nov 01 21:34:08.026818 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:08.026793 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bdb6494-4dfd-4ff8-9037-e65c81ccd574-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8a7e60d-zzftn\" (UID: \"9bdb6494-4dfd-4ff8-9037-e65c81ccd574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8a7e60d-zzftn" Nov 01 21:34:08.028396 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:08.028377 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bdb6494-4dfd-4ff8-9037-e65c81ccd574-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8a7e60d-zzftn\" (UID: \"9bdb6494-4dfd-4ff8-9037-e65c81ccd574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8a7e60d-zzftn" Nov 01 21:34:08.084023 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:08.083929 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nsbv\" (UniqueName: \"kubernetes.io/projected/9bdb6494-4dfd-4ff8-9037-e65c81ccd574-kube-api-access-4nsbv\") pod \"maas-api-revoke-sub-d8a7e60d-zzftn\" (UID: \"9bdb6494-4dfd-4ff8-9037-e65c81ccd574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8a7e60d-zzftn" Nov 01 21:34:08.142188 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:08.142156 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8a7e60d-zzftn" Nov 01 21:34:08.512369 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:08.512342 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8a7e60d-zzftn"] Nov 01 21:34:08.514282 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:34:08.514249 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bdb6494_4dfd_4ff8_9037_e65c81ccd574.slice/crio-3bf7357cdbdf7d47f546a82dc2378975bda1b8455b921ceb22af0c90c07ec889 WatchSource:0}: Error finding container 3bf7357cdbdf7d47f546a82dc2378975bda1b8455b921ceb22af0c90c07ec889: Status 404 returned error can't find the container with id 3bf7357cdbdf7d47f546a82dc2378975bda1b8455b921ceb22af0c90c07ec889 Nov 01 21:34:08.865821 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:08.865774 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8a7e60d-zzftn" event={"ID":"9bdb6494-4dfd-4ff8-9037-e65c81ccd574","Type":"ContainerStarted","Data":"b57d51b7bfff36116d2c4b7822b5e14e4826334d58d86b14753f09042a8160c2"} Nov 01 21:34:08.865821 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:08.865826 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8a7e60d-zzftn" event={"ID":"9bdb6494-4dfd-4ff8-9037-e65c81ccd574","Type":"ContainerStarted","Data":"3bf7357cdbdf7d47f546a82dc2378975bda1b8455b921ceb22af0c90c07ec889"} Nov 01 21:34:08.982760 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:08.982690 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8a7e60d-zzftn" podStartSLOduration=1.9826750789999998 podStartE2EDuration="1.982675079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:34:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:34:08.980654081 +0000 UTC m=+3799.758294217" watchObservedRunningTime="2026-11-01 21:34:08.982675079 +0000 UTC m=+3799.760315214" Nov 01 21:34:09.872973 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:09.872939 4484 generic.go:356] "Generic (PLEG): container finished" podID="9bdb6494-4dfd-4ff8-9037-e65c81ccd574" containerID="b57d51b7bfff36116d2c4b7822b5e14e4826334d58d86b14753f09042a8160c2" exitCode=0 Nov 01 21:34:09.873403 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:09.873029 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8a7e60d-zzftn" event={"ID":"9bdb6494-4dfd-4ff8-9037-e65c81ccd574","Type":"ContainerDied","Data":"b57d51b7bfff36116d2c4b7822b5e14e4826334d58d86b14753f09042a8160c2"} Nov 01 21:34:11.016177 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:11.016150 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8a7e60d-zzftn" Nov 01 21:34:11.155185 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:11.155093 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bdb6494-4dfd-4ff8-9037-e65c81ccd574-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bdb6494-4dfd-4ff8-9037-e65c81ccd574-maas-api-xks-ca\") pod \"9bdb6494-4dfd-4ff8-9037-e65c81ccd574\" (UID: \"9bdb6494-4dfd-4ff8-9037-e65c81ccd574\") " Nov 01 21:34:11.155185 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:11.155132 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bdb6494-4dfd-4ff8-9037-e65c81ccd574-kube-api-access-4nsbv\" (UniqueName: \"kubernetes.io/projected/9bdb6494-4dfd-4ff8-9037-e65c81ccd574-kube-api-access-4nsbv\") pod \"9bdb6494-4dfd-4ff8-9037-e65c81ccd574\" (UID: \"9bdb6494-4dfd-4ff8-9037-e65c81ccd574\") " Nov 01 21:34:11.155449 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:11.155255 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bdb6494-4dfd-4ff8-9037-e65c81ccd574-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bdb6494-4dfd-4ff8-9037-e65c81ccd574-maas-api-openshift-ca\") pod \"9bdb6494-4dfd-4ff8-9037-e65c81ccd574\" (UID: \"9bdb6494-4dfd-4ff8-9037-e65c81ccd574\") " Nov 01 21:34:11.155643 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:11.155617 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bdb6494-4dfd-4ff8-9037-e65c81ccd574-maas-api-openshift-ca" pod "9bdb6494-4dfd-4ff8-9037-e65c81ccd574" (UID: "9bdb6494-4dfd-4ff8-9037-e65c81ccd574"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:34:11.157322 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:11.157285 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bdb6494-4dfd-4ff8-9037-e65c81ccd574-kube-api-access-4nsbv" pod "9bdb6494-4dfd-4ff8-9037-e65c81ccd574" (UID: "9bdb6494-4dfd-4ff8-9037-e65c81ccd574"). InnerVolumeSpecName "kube-api-access-4nsbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:34:11.157456 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:11.157342 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bdb6494-4dfd-4ff8-9037-e65c81ccd574-maas-api-xks-ca" pod "9bdb6494-4dfd-4ff8-9037-e65c81ccd574" (UID: "9bdb6494-4dfd-4ff8-9037-e65c81ccd574"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:34:11.256585 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:11.256544 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bdb6494-4dfd-4ff8-9037-e65c81ccd574-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:34:11.256585 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:11.256582 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bdb6494-4dfd-4ff8-9037-e65c81ccd574-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:34:11.256585 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:11.256593 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nsbv\" (UniqueName: \"kubernetes.io/projected/9bdb6494-4dfd-4ff8-9037-e65c81ccd574-kube-api-access-4nsbv\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:34:11.883525 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:11.883490 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8a7e60d-zzftn" Nov 01 21:34:11.883525 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:11.883508 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8a7e60d-zzftn" event={"ID":"9bdb6494-4dfd-4ff8-9037-e65c81ccd574","Type":"ContainerDied","Data":"3bf7357cdbdf7d47f546a82dc2378975bda1b8455b921ceb22af0c90c07ec889"} Nov 01 21:34:11.883756 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:11.883552 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3bf7357cdbdf7d47f546a82dc2378975bda1b8455b921ceb22af0c90c07ec889" Nov 01 21:34:15.766159 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:34:15.766123 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:34:17.043195 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:17.043155 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rxxx4"] Nov 01 21:34:17.093864 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:17.093827 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rxxx4"] Nov 01 21:34:17.370524 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:17.370422 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64dada2e-ccc9-4ab8-a9e6-c75d6c0e0611" path="/var/lib/kubelet/pods/64dada2e-ccc9-4ab8-a9e6-c75d6c0e0611/volumes" Nov 01 21:34:18.868668 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:18.868638 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:34:27.351144 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:27.351103 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:34:27.622617 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:27.622522 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:34:27.816857 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:27.816792 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8"] Nov 01 21:34:27.817359 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:27.817343 4484 cpu_manager.go:414] "containerMap: removing container" podUID="9bdb6494-4dfd-4ff8-9037-e65c81ccd574" containerName="revoke-keys" Nov 01 21:34:27.817425 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:27.817361 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bdb6494-4dfd-4ff8-9037-e65c81ccd574" containerName="revoke-keys" Nov 01 21:34:27.817464 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:27.817450 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bdb6494-4dfd-4ff8-9037-e65c81ccd574" containerName="revoke-keys" Nov 01 21:34:27.820588 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:27.820569 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8" Nov 01 21:34:27.889594 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:27.889507 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-49c82ebd-openshift-default-dockercfg-tx85g\"" Nov 01 21:34:27.920209 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:27.920174 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38ce6dc7-e34a-48da-a283-f140baa31047-istio-podinfo\") pod \"e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8\" (UID: \"38ce6dc7-e34a-48da-a283-f140baa31047\") " pod="openshift-ingress/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8" Nov 01 21:34:27.920380 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:27.920216 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38ce6dc7-e34a-48da-a283-f140baa31047-workload-certs\") pod \"e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8\" (UID: \"38ce6dc7-e34a-48da-a283-f140baa31047\") " pod="openshift-ingress/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8" Nov 01 21:34:27.920380 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:27.920244 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38ce6dc7-e34a-48da-a283-f140baa31047-istio-token\") pod \"e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8\" (UID: \"38ce6dc7-e34a-48da-a283-f140baa31047\") " pod="openshift-ingress/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8" Nov 01 21:34:27.920380 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:27.920345 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38ce6dc7-e34a-48da-a283-f140baa31047-istio-data\") pod \"e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8\" (UID: \"38ce6dc7-e34a-48da-a283-f140baa31047\") " pod="openshift-ingress/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8" Nov 01 21:34:27.920499 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:27.920448 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ce6dc7-e34a-48da-a283-f140baa31047-credential-socket\") pod \"e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8\" (UID: \"38ce6dc7-e34a-48da-a283-f140baa31047\") " pod="openshift-ingress/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8" Nov 01 21:34:27.920499 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:27.920493 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnhlw\" (UniqueName: \"kubernetes.io/projected/38ce6dc7-e34a-48da-a283-f140baa31047-kube-api-access-mnhlw\") pod \"e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8\" (UID: \"38ce6dc7-e34a-48da-a283-f140baa31047\") " pod="openshift-ingress/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8" Nov 01 21:34:27.920569 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:27.920524 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38ce6dc7-e34a-48da-a283-f140baa31047-istio-envoy\") pod \"e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8\" (UID: \"38ce6dc7-e34a-48da-a283-f140baa31047\") " pod="openshift-ingress/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8" Nov 01 21:34:27.920643 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:27.920623 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38ce6dc7-e34a-48da-a283-f140baa31047-istiod-ca-cert\") pod \"e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8\" (UID: \"38ce6dc7-e34a-48da-a283-f140baa31047\") " pod="openshift-ingress/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8" Nov 01 21:34:27.920678 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:27.920665 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ce6dc7-e34a-48da-a283-f140baa31047-workload-socket\") pod \"e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8\" (UID: \"38ce6dc7-e34a-48da-a283-f140baa31047\") " pod="openshift-ingress/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8" Nov 01 21:34:27.929571 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:27.929527 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8"] Nov 01 21:34:27.941689 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:27.941590 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:34:28.021784 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:28.021743 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38ce6dc7-e34a-48da-a283-f140baa31047-workload-certs\") pod \"e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8\" (UID: \"38ce6dc7-e34a-48da-a283-f140baa31047\") " pod="openshift-ingress/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8" Nov 01 21:34:28.022059 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:28.021792 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38ce6dc7-e34a-48da-a283-f140baa31047-istio-token\") pod \"e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8\" (UID: \"38ce6dc7-e34a-48da-a283-f140baa31047\") " pod="openshift-ingress/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8" Nov 01 21:34:28.022059 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:28.021819 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38ce6dc7-e34a-48da-a283-f140baa31047-istio-data\") pod \"e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8\" (UID: \"38ce6dc7-e34a-48da-a283-f140baa31047\") " pod="openshift-ingress/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8" Nov 01 21:34:28.022059 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:28.021903 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ce6dc7-e34a-48da-a283-f140baa31047-credential-socket\") pod \"e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8\" (UID: \"38ce6dc7-e34a-48da-a283-f140baa31047\") " pod="openshift-ingress/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8" Nov 01 21:34:28.022059 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:28.021957 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnhlw\" (UniqueName: \"kubernetes.io/projected/38ce6dc7-e34a-48da-a283-f140baa31047-kube-api-access-mnhlw\") pod \"e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8\" (UID: \"38ce6dc7-e34a-48da-a283-f140baa31047\") " pod="openshift-ingress/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8" Nov 01 21:34:28.022340 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:28.022061 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38ce6dc7-e34a-48da-a283-f140baa31047-istio-envoy\") pod \"e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8\" (UID: \"38ce6dc7-e34a-48da-a283-f140baa31047\") " pod="openshift-ingress/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8" Nov 01 21:34:28.022340 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:28.022118 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38ce6dc7-e34a-48da-a283-f140baa31047-istiod-ca-cert\") pod \"e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8\" (UID: \"38ce6dc7-e34a-48da-a283-f140baa31047\") " pod="openshift-ingress/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8" Nov 01 21:34:28.022340 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:28.022146 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ce6dc7-e34a-48da-a283-f140baa31047-workload-socket\") pod \"e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8\" (UID: \"38ce6dc7-e34a-48da-a283-f140baa31047\") " pod="openshift-ingress/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8" Nov 01 21:34:28.022340 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:28.022190 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38ce6dc7-e34a-48da-a283-f140baa31047-workload-certs\") pod \"e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8\" (UID: \"38ce6dc7-e34a-48da-a283-f140baa31047\") " pod="openshift-ingress/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8" Nov 01 21:34:28.022340 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:28.022199 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38ce6dc7-e34a-48da-a283-f140baa31047-istio-podinfo\") pod \"e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8\" (UID: \"38ce6dc7-e34a-48da-a283-f140baa31047\") " pod="openshift-ingress/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8" Nov 01 21:34:28.022340 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:28.022264 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38ce6dc7-e34a-48da-a283-f140baa31047-istio-data\") pod \"e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8\" (UID: \"38ce6dc7-e34a-48da-a283-f140baa31047\") " pod="openshift-ingress/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8" Nov 01 21:34:28.022340 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:28.022332 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ce6dc7-e34a-48da-a283-f140baa31047-credential-socket\") pod \"e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8\" (UID: \"38ce6dc7-e34a-48da-a283-f140baa31047\") " pod="openshift-ingress/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8" Nov 01 21:34:28.022702 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:28.022566 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ce6dc7-e34a-48da-a283-f140baa31047-workload-socket\") pod \"e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8\" (UID: \"38ce6dc7-e34a-48da-a283-f140baa31047\") " pod="openshift-ingress/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8" Nov 01 21:34:28.022702 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:28.022681 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38ce6dc7-e34a-48da-a283-f140baa31047-istiod-ca-cert\") pod \"e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8\" (UID: \"38ce6dc7-e34a-48da-a283-f140baa31047\") " pod="openshift-ingress/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8" Nov 01 21:34:28.024451 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:28.024430 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38ce6dc7-e34a-48da-a283-f140baa31047-istio-envoy\") pod \"e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8\" (UID: \"38ce6dc7-e34a-48da-a283-f140baa31047\") " pod="openshift-ingress/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8" Nov 01 21:34:28.024646 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:28.024627 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38ce6dc7-e34a-48da-a283-f140baa31047-istio-podinfo\") pod \"e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8\" (UID: \"38ce6dc7-e34a-48da-a283-f140baa31047\") " pod="openshift-ingress/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8" Nov 01 21:34:28.079601 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:28.079562 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnhlw\" (UniqueName: \"kubernetes.io/projected/38ce6dc7-e34a-48da-a283-f140baa31047-kube-api-access-mnhlw\") pod \"e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8\" (UID: \"38ce6dc7-e34a-48da-a283-f140baa31047\") " pod="openshift-ingress/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8" Nov 01 21:34:28.079939 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:28.079911 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38ce6dc7-e34a-48da-a283-f140baa31047-istio-token\") pod \"e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8\" (UID: \"38ce6dc7-e34a-48da-a283-f140baa31047\") " pod="openshift-ingress/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8" Nov 01 21:34:28.133885 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:28.133845 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8" Nov 01 21:34:28.142818 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:28.142704 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:34:28.558208 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:28.558181 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8"] Nov 01 21:34:28.560127 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:34:28.560097 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38ce6dc7_e34a_48da_a283_f140baa31047.slice/crio-c287e088ad16d52813d356ba9c954570525c386ff2df43815e0fe827a96d78b4 WatchSource:0}: Error finding container c287e088ad16d52813d356ba9c954570525c386ff2df43815e0fe827a96d78b4: Status 404 returned error can't find the container with id c287e088ad16d52813d356ba9c954570525c386ff2df43815e0fe827a96d78b4 Nov 01 21:34:28.562379 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:28.562349 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:34:28.562462 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:28.562415 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:34:28.562462 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:28.562445 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:34:28.987517 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:28.987431 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8" event={"ID":"38ce6dc7-e34a-48da-a283-f140baa31047","Type":"ContainerStarted","Data":"11eec086e09984b31ff6727bc5f8e31d136be7de9735d40d7efb09d36f8c4bad"} Nov 01 21:34:28.987517 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:28.987484 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8" event={"ID":"38ce6dc7-e34a-48da-a283-f140baa31047","Type":"ContainerStarted","Data":"c287e088ad16d52813d356ba9c954570525c386ff2df43815e0fe827a96d78b4"} Nov 01 21:34:29.106947 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:29.106894 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8" podStartSLOduration=2.106880315 podStartE2EDuration="2.106880315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:34:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:34:29.104922416 +0000 UTC m=+3819.882562556" watchObservedRunningTime="2026-11-01 21:34:29.106880315 +0000 UTC m=+3819.884520449" Nov 01 21:34:29.134902 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:29.134858 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8" Nov 01 21:34:29.140096 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:29.140072 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8" Nov 01 21:34:29.992275 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:29.992235 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8" Nov 01 21:34:29.993248 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:29.993228 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8" Nov 01 21:34:31.261971 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:31.261886 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:34:31.532061 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:31.531973 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:34:31.739575 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:31.739533 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:34:34.552262 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:34.552230 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:34:34.710470 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:34.710437 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:34:35.067677 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:35.067637 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-49c82ebd-59b45759bf-hjk4h"] Nov 01 21:34:35.071773 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:35.071741 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-49c82ebd-59b45759bf-hjk4h" Nov 01 21:34:35.125141 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:35.125114 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-49c82ebd\"" Nov 01 21:34:35.175956 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:35.175927 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-49c82ebd-dockercfg-8shbf\"" Nov 01 21:34:35.178474 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:35.178447 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-49c82ebd-59b45759bf-hjk4h"] Nov 01 21:34:35.207346 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:35.207307 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p86tb\" (UniqueName: \"kubernetes.io/projected/3d5b9979-482e-4416-b861-2028b0a5a2da-kube-api-access-p86tb\") pod \"payload-processing-e2e-ait-49c82ebd-59b45759bf-hjk4h\" (UID: \"3d5b9979-482e-4416-b861-2028b0a5a2da\") " pod="openshift-ingress/payload-processing-e2e-ait-49c82ebd-59b45759bf-hjk4h" Nov 01 21:34:35.207527 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:35.207366 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d5b9979-482e-4416-b861-2028b0a5a2da-plugins-config-volume\") pod \"payload-processing-e2e-ait-49c82ebd-59b45759bf-hjk4h\" (UID: \"3d5b9979-482e-4416-b861-2028b0a5a2da\") " pod="openshift-ingress/payload-processing-e2e-ait-49c82ebd-59b45759bf-hjk4h" Nov 01 21:34:35.229715 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:35.229681 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-49c82ebd-649fbdcc87-7rzw2"] Nov 01 21:34:35.233182 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:35.233167 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-49c82ebd-649fbdcc87-7rzw2" Nov 01 21:34:35.289402 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:35.289369 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-49c82ebd-649fbdcc87-7rzw2"] Nov 01 21:34:35.308222 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:35.308187 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p86tb\" (UniqueName: \"kubernetes.io/projected/3d5b9979-482e-4416-b861-2028b0a5a2da-kube-api-access-p86tb\") pod \"payload-processing-e2e-ait-49c82ebd-59b45759bf-hjk4h\" (UID: \"3d5b9979-482e-4416-b861-2028b0a5a2da\") " pod="openshift-ingress/payload-processing-e2e-ait-49c82ebd-59b45759bf-hjk4h" Nov 01 21:34:35.308410 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:35.308235 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhgxl\" (UniqueName: \"kubernetes.io/projected/628c73a8-34d3-409e-a38d-b97dcda63da7-kube-api-access-fhgxl\") pod \"payload-pre-processing-e2e-ait-49c82ebd-649fbdcc87-7rzw2\" (UID: \"628c73a8-34d3-409e-a38d-b97dcda63da7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-49c82ebd-649fbdcc87-7rzw2" Nov 01 21:34:35.308410 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:35.308284 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d5b9979-482e-4416-b861-2028b0a5a2da-plugins-config-volume\") pod \"payload-processing-e2e-ait-49c82ebd-59b45759bf-hjk4h\" (UID: \"3d5b9979-482e-4416-b861-2028b0a5a2da\") " pod="openshift-ingress/payload-processing-e2e-ait-49c82ebd-59b45759bf-hjk4h" Nov 01 21:34:35.308410 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:35.308336 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/628c73a8-34d3-409e-a38d-b97dcda63da7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-49c82ebd-649fbdcc87-7rzw2\" (UID: \"628c73a8-34d3-409e-a38d-b97dcda63da7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-49c82ebd-649fbdcc87-7rzw2" Nov 01 21:34:35.309090 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:35.309066 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d5b9979-482e-4416-b861-2028b0a5a2da-plugins-config-volume\") pod \"payload-processing-e2e-ait-49c82ebd-59b45759bf-hjk4h\" (UID: \"3d5b9979-482e-4416-b861-2028b0a5a2da\") " pod="openshift-ingress/payload-processing-e2e-ait-49c82ebd-59b45759bf-hjk4h" Nov 01 21:34:35.365591 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:35.365519 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p86tb\" (UniqueName: \"kubernetes.io/projected/3d5b9979-482e-4416-b861-2028b0a5a2da-kube-api-access-p86tb\") pod \"payload-processing-e2e-ait-49c82ebd-59b45759bf-hjk4h\" (UID: \"3d5b9979-482e-4416-b861-2028b0a5a2da\") " pod="openshift-ingress/payload-processing-e2e-ait-49c82ebd-59b45759bf-hjk4h" Nov 01 21:34:35.388243 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:35.388208 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-49c82ebd-59b45759bf-hjk4h" Nov 01 21:34:35.409681 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:35.409647 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhgxl\" (UniqueName: \"kubernetes.io/projected/628c73a8-34d3-409e-a38d-b97dcda63da7-kube-api-access-fhgxl\") pod \"payload-pre-processing-e2e-ait-49c82ebd-649fbdcc87-7rzw2\" (UID: \"628c73a8-34d3-409e-a38d-b97dcda63da7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-49c82ebd-649fbdcc87-7rzw2" Nov 01 21:34:35.409817 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:35.409732 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/628c73a8-34d3-409e-a38d-b97dcda63da7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-49c82ebd-649fbdcc87-7rzw2\" (UID: \"628c73a8-34d3-409e-a38d-b97dcda63da7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-49c82ebd-649fbdcc87-7rzw2" Nov 01 21:34:35.410681 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:35.410654 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/628c73a8-34d3-409e-a38d-b97dcda63da7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-49c82ebd-649fbdcc87-7rzw2\" (UID: \"628c73a8-34d3-409e-a38d-b97dcda63da7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-49c82ebd-649fbdcc87-7rzw2" Nov 01 21:34:35.468775 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:35.468741 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhgxl\" (UniqueName: \"kubernetes.io/projected/628c73a8-34d3-409e-a38d-b97dcda63da7-kube-api-access-fhgxl\") pod \"payload-pre-processing-e2e-ait-49c82ebd-649fbdcc87-7rzw2\" (UID: \"628c73a8-34d3-409e-a38d-b97dcda63da7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-49c82ebd-649fbdcc87-7rzw2" Nov 01 21:34:35.546355 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:35.546316 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-49c82ebd-649fbdcc87-7rzw2" Nov 01 21:34:35.761638 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:35.761608 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-49c82ebd-59b45759bf-hjk4h"] Nov 01 21:34:35.762783 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:34:35.762757 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d5b9979_482e_4416_b861_2028b0a5a2da.slice/crio-42a46932f734c71186cd126aceeebe339b11d6e17fa393829da48dc98133918d WatchSource:0}: Error finding container 42a46932f734c71186cd126aceeebe339b11d6e17fa393829da48dc98133918d: Status 404 returned error can't find the container with id 42a46932f734c71186cd126aceeebe339b11d6e17fa393829da48dc98133918d Nov 01 21:34:35.923444 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:34:35.923416 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod628c73a8_34d3_409e_a38d_b97dcda63da7.slice/crio-4c0960daf7f6b27225ded231c821f6200c12e388c38f557c09bc69f3414de858 WatchSource:0}: Error finding container 4c0960daf7f6b27225ded231c821f6200c12e388c38f557c09bc69f3414de858: Status 404 returned error can't find the container with id 4c0960daf7f6b27225ded231c821f6200c12e388c38f557c09bc69f3414de858 Nov 01 21:34:35.923592 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:35.923556 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-49c82ebd-649fbdcc87-7rzw2"] Nov 01 21:34:35.978933 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:35.978909 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:34:36.037385 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:36.037292 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-49c82ebd-649fbdcc87-7rzw2" event={"ID":"628c73a8-34d3-409e-a38d-b97dcda63da7","Type":"ContainerStarted","Data":"fef79252397e098825a961dc6435f17136ca394ec294662e2058ea36e824299e"} Nov 01 21:34:36.037385 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:36.037354 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-49c82ebd-649fbdcc87-7rzw2" event={"ID":"628c73a8-34d3-409e-a38d-b97dcda63da7","Type":"ContainerStarted","Data":"4c0960daf7f6b27225ded231c821f6200c12e388c38f557c09bc69f3414de858"} Nov 01 21:34:36.037624 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:36.037480 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-49c82ebd-649fbdcc87-7rzw2" Nov 01 21:34:36.040894 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:36.040864 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-49c82ebd-59b45759bf-hjk4h" event={"ID":"3d5b9979-482e-4416-b861-2028b0a5a2da","Type":"ContainerStarted","Data":"a7800305d33c61320c0175fa5a37f6d8d97186cc73d68b1b45de5b8ee591c7d1"} Nov 01 21:34:36.041174 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:36.041151 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-49c82ebd-59b45759bf-hjk4h" event={"ID":"3d5b9979-482e-4416-b861-2028b0a5a2da","Type":"ContainerStarted","Data":"42a46932f734c71186cd126aceeebe339b11d6e17fa393829da48dc98133918d"} Nov 01 21:34:36.041315 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:36.041193 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-49c82ebd-59b45759bf-hjk4h" Nov 01 21:34:36.153628 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:36.153572 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-49c82ebd-649fbdcc87-7rzw2" podStartSLOduration=1.15355774 podStartE2EDuration="1.15355774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:34:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:34:36.152364428 +0000 UTC m=+3826.930004566" watchObservedRunningTime="2026-11-01 21:34:36.15355774 +0000 UTC m=+3826.931197935" Nov 01 21:34:36.179717 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:36.179672 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:34:36.269719 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:36.269660 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-49c82ebd-59b45759bf-hjk4h" podStartSLOduration=1.2696445920000001 podStartE2EDuration="1.269644592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:34:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:34:36.267637725 +0000 UTC m=+3827.045277860" watchObservedRunningTime="2026-11-01 21:34:36.269644592 +0000 UTC m=+3827.047284726" Nov 01 21:34:36.392163 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:36.388744 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:34:41.119851 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:41.119822 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:34:45.768013 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:34:45.767966 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:34:45.943261 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:45.943220 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:34:46.044499 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:46.044414 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7hj4r"] Nov 01 21:34:46.095045 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:46.095007 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7hj4r"] Nov 01 21:34:46.529621 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:46.529586 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:34:46.686380 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:46.686346 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:34:47.048664 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:47.048635 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-49c82ebd-59b45759bf-hjk4h" Nov 01 21:34:47.049237 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:47.048702 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-49c82ebd-649fbdcc87-7rzw2" Nov 01 21:34:47.371032 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:47.370930 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08fca9d9-f3ac-49b2-bc69-8e13be477f5a" path="/var/lib/kubelet/pods/08fca9d9-f3ac-49b2-bc69-8e13be477f5a/volumes" Nov 01 21:34:50.743559 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:50.743522 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:34:53.149863 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:53.149788 4484 generic.go:356] "Generic (PLEG): container finished" podID="38ce6dc7-e34a-48da-a283-f140baa31047" containerID="11eec086e09984b31ff6727bc5f8e31d136be7de9735d40d7efb09d36f8c4bad" exitCode=0 Nov 01 21:34:53.150293 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:53.149863 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8" event={"ID":"38ce6dc7-e34a-48da-a283-f140baa31047","Type":"ContainerDied","Data":"11eec086e09984b31ff6727bc5f8e31d136be7de9735d40d7efb09d36f8c4bad"} Nov 01 21:34:53.150350 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:53.150333 4484 scope.go:122] "RemoveContainer" containerID="11eec086e09984b31ff6727bc5f8e31d136be7de9735d40d7efb09d36f8c4bad" Nov 01 21:34:53.151568 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:53.151534 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:34:53.151677 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:53.151608 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:34:53.151677 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:53.151644 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:34:54.157196 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:54.157160 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8" event={"ID":"38ce6dc7-e34a-48da-a283-f140baa31047","Type":"ContainerStarted","Data":"4b3f6dc14241ebd96f174cd2061e6455fec55c28b3c74591852d0e86de605283"} Nov 01 21:34:54.863906 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:54.863875 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:34:55.134339 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:55.134245 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8" Nov 01 21:34:55.134636 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:55.134601 4484 patch_prober.go:29] interesting pod/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.65:15021/healthz/ready\": dial tcp 10.134.1.65:15021: connect: connection refused" start-of-body= Nov 01 21:34:55.134766 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:55.134655 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8" podUID="38ce6dc7-e34a-48da-a283-f140baa31047" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.65:15021/healthz/ready\": dial tcp 10.134.1.65:15021: connect: connection refused" Nov 01 21:34:55.929490 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:55.929454 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:34:56.134801 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:56.134766 4484 patch_prober.go:29] interesting pod/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.65:15021/healthz/ready\": dial tcp 10.134.1.65:15021: connect: connection refused" start-of-body= Nov 01 21:34:56.135014 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:56.134819 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8" podUID="38ce6dc7-e34a-48da-a283-f140baa31047" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.65:15021/healthz/ready\": dial tcp 10.134.1.65:15021: connect: connection refused" Nov 01 21:34:57.134722 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:57.134686 4484 patch_prober.go:29] interesting pod/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.65:15021/healthz/ready\": dial tcp 10.134.1.65:15021: connect: connection refused" start-of-body= Nov 01 21:34:57.135169 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:57.134737 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8" podUID="38ce6dc7-e34a-48da-a283-f140baa31047" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.65:15021/healthz/ready\": dial tcp 10.134.1.65:15021: connect: connection refused" Nov 01 21:34:58.134530 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:58.134495 4484 patch_prober.go:29] interesting pod/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.65:15021/healthz/ready\": dial tcp 10.134.1.65:15021: connect: connection refused" start-of-body= Nov 01 21:34:58.134723 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:58.134550 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8" podUID="38ce6dc7-e34a-48da-a283-f140baa31047" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.65:15021/healthz/ready\": dial tcp 10.134.1.65:15021: connect: connection refused" Nov 01 21:34:59.134844 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:59.134801 4484 patch_prober.go:29] interesting pod/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.65:15021/healthz/ready\": dial tcp 10.134.1.65:15021: connect: connection refused" start-of-body= Nov 01 21:34:59.135311 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:59.134864 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8" podUID="38ce6dc7-e34a-48da-a283-f140baa31047" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.65:15021/healthz/ready\": dial tcp 10.134.1.65:15021: connect: connection refused" Nov 01 21:34:59.992292 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:34:59.992257 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8" Nov 01 21:35:00.139098 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:00.139064 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8" Nov 01 21:35:00.187890 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:00.187860 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8" Nov 01 21:35:00.986405 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:00.986362 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:35:05.947067 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:05.947027 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:35:07.510702 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:07.510663 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49c82ebd-hxrqv"] Nov 01 21:35:07.515399 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:07.515380 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49c82ebd-hxrqv" Nov 01 21:35:07.569651 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:07.569611 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4253b695-93ed-4cd7-bdb6-562401a387a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-49c82ebd-hxrqv\" (UID: \"4253b695-93ed-4cd7-bdb6-562401a387a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49c82ebd-hxrqv" Nov 01 21:35:07.569651 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:07.569679 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4253b695-93ed-4cd7-bdb6-562401a387a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-49c82ebd-hxrqv\" (UID: \"4253b695-93ed-4cd7-bdb6-562401a387a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49c82ebd-hxrqv" Nov 01 21:35:07.569968 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:07.569708 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqlmc\" (UniqueName: \"kubernetes.io/projected/4253b695-93ed-4cd7-bdb6-562401a387a2-kube-api-access-qqlmc\") pod \"maas-api-revoke-keys-e2e-ait-49c82ebd-hxrqv\" (UID: \"4253b695-93ed-4cd7-bdb6-562401a387a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49c82ebd-hxrqv" Nov 01 21:35:07.571296 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:07.571267 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49c82ebd-hxrqv"] Nov 01 21:35:07.619030 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:07.618983 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:35:07.619227 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:07.619049 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:35:07.619293 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:07.619224 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:35:07.620378 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:07.620363 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzdwq\"" Nov 01 21:35:07.670887 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:07.670849 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4253b695-93ed-4cd7-bdb6-562401a387a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-49c82ebd-hxrqv\" (UID: \"4253b695-93ed-4cd7-bdb6-562401a387a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49c82ebd-hxrqv" Nov 01 21:35:07.671120 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:07.670902 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4253b695-93ed-4cd7-bdb6-562401a387a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-49c82ebd-hxrqv\" (UID: \"4253b695-93ed-4cd7-bdb6-562401a387a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49c82ebd-hxrqv" Nov 01 21:35:07.671120 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:07.670922 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqlmc\" (UniqueName: \"kubernetes.io/projected/4253b695-93ed-4cd7-bdb6-562401a387a2-kube-api-access-qqlmc\") pod \"maas-api-revoke-keys-e2e-ait-49c82ebd-hxrqv\" (UID: \"4253b695-93ed-4cd7-bdb6-562401a387a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49c82ebd-hxrqv" Nov 01 21:35:07.671636 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:07.671610 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4253b695-93ed-4cd7-bdb6-562401a387a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-49c82ebd-hxrqv\" (UID: \"4253b695-93ed-4cd7-bdb6-562401a387a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49c82ebd-hxrqv" Nov 01 21:35:07.673477 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:07.673453 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4253b695-93ed-4cd7-bdb6-562401a387a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-49c82ebd-hxrqv\" (UID: \"4253b695-93ed-4cd7-bdb6-562401a387a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49c82ebd-hxrqv" Nov 01 21:35:07.728571 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:07.728537 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqlmc\" (UniqueName: \"kubernetes.io/projected/4253b695-93ed-4cd7-bdb6-562401a387a2-kube-api-access-qqlmc\") pod \"maas-api-revoke-keys-e2e-ait-49c82ebd-hxrqv\" (UID: \"4253b695-93ed-4cd7-bdb6-562401a387a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49c82ebd-hxrqv" Nov 01 21:35:07.828696 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:07.828604 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49c82ebd-hxrqv" Nov 01 21:35:08.197613 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:08.197580 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49c82ebd-hxrqv"] Nov 01 21:35:08.198910 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:35:08.198884 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4253b695_93ed_4cd7_bdb6_562401a387a2.slice/crio-7099708f7a2f40f6bd70a77e6125de12facb8deda20a86531afb89dbdfcf8776 WatchSource:0}: Error finding container 7099708f7a2f40f6bd70a77e6125de12facb8deda20a86531afb89dbdfcf8776: Status 404 returned error can't find the container with id 7099708f7a2f40f6bd70a77e6125de12facb8deda20a86531afb89dbdfcf8776 Nov 01 21:35:08.232780 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:08.232751 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49c82ebd-hxrqv" event={"ID":"4253b695-93ed-4cd7-bdb6-562401a387a2","Type":"ContainerStarted","Data":"7099708f7a2f40f6bd70a77e6125de12facb8deda20a86531afb89dbdfcf8776"} Nov 01 21:35:09.240503 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:09.240468 4484 generic.go:356] "Generic (PLEG): container finished" podID="4253b695-93ed-4cd7-bdb6-562401a387a2" containerID="065c3959e9c74cab2ff8b95e6dc49f9185d14aa46103aac785dc23919d344568" exitCode=0 Nov 01 21:35:09.240895 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:09.240556 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49c82ebd-hxrqv" event={"ID":"4253b695-93ed-4cd7-bdb6-562401a387a2","Type":"ContainerDied","Data":"065c3959e9c74cab2ff8b95e6dc49f9185d14aa46103aac785dc23919d344568"} Nov 01 21:35:10.376042 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:10.376016 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49c82ebd-hxrqv" Nov 01 21:35:10.498953 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:10.498861 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4253b695-93ed-4cd7-bdb6-562401a387a2-kube-api-access-qqlmc\" (UniqueName: \"kubernetes.io/projected/4253b695-93ed-4cd7-bdb6-562401a387a2-kube-api-access-qqlmc\") pod \"4253b695-93ed-4cd7-bdb6-562401a387a2\" (UID: \"4253b695-93ed-4cd7-bdb6-562401a387a2\") " Nov 01 21:35:10.498953 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:10.498923 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4253b695-93ed-4cd7-bdb6-562401a387a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4253b695-93ed-4cd7-bdb6-562401a387a2-maas-api-xks-ca\") pod \"4253b695-93ed-4cd7-bdb6-562401a387a2\" (UID: \"4253b695-93ed-4cd7-bdb6-562401a387a2\") " Nov 01 21:35:10.499224 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:10.499071 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4253b695-93ed-4cd7-bdb6-562401a387a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4253b695-93ed-4cd7-bdb6-562401a387a2-maas-api-openshift-ca\") pod \"4253b695-93ed-4cd7-bdb6-562401a387a2\" (UID: \"4253b695-93ed-4cd7-bdb6-562401a387a2\") " Nov 01 21:35:10.499452 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:10.499426 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4253b695-93ed-4cd7-bdb6-562401a387a2-maas-api-openshift-ca" pod "4253b695-93ed-4cd7-bdb6-562401a387a2" (UID: "4253b695-93ed-4cd7-bdb6-562401a387a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:35:10.501119 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:10.501088 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4253b695-93ed-4cd7-bdb6-562401a387a2-maas-api-xks-ca" pod "4253b695-93ed-4cd7-bdb6-562401a387a2" (UID: "4253b695-93ed-4cd7-bdb6-562401a387a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:35:10.501119 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:10.501100 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4253b695-93ed-4cd7-bdb6-562401a387a2-kube-api-access-qqlmc" pod "4253b695-93ed-4cd7-bdb6-562401a387a2" (UID: "4253b695-93ed-4cd7-bdb6-562401a387a2"). InnerVolumeSpecName "kube-api-access-qqlmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:35:10.600338 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:10.600300 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4253b695-93ed-4cd7-bdb6-562401a387a2-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:35:10.600338 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:10.600333 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqlmc\" (UniqueName: \"kubernetes.io/projected/4253b695-93ed-4cd7-bdb6-562401a387a2-kube-api-access-qqlmc\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:35:10.600338 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:10.600345 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4253b695-93ed-4cd7-bdb6-562401a387a2-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:35:10.739447 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:10.739413 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:35:11.252715 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:11.252687 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49c82ebd-hxrqv" Nov 01 21:35:11.252910 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:11.252717 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49c82ebd-hxrqv" event={"ID":"4253b695-93ed-4cd7-bdb6-562401a387a2","Type":"ContainerDied","Data":"7099708f7a2f40f6bd70a77e6125de12facb8deda20a86531afb89dbdfcf8776"} Nov 01 21:35:11.252910 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:11.252768 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7099708f7a2f40f6bd70a77e6125de12facb8deda20a86531afb89dbdfcf8776" Nov 01 21:35:15.536975 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:15.536940 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:35:15.767109 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:35:15.767071 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:35:19.008327 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:19.008287 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-49c82ebd-59b45759bf-hjk4h"] Nov 01 21:35:19.116189 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:19.116150 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-49c82ebd-649fbdcc87-7rzw2"] Nov 01 21:35:19.510885 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:19.510852 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:35:19.687358 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:19.687329 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:35:20.321448 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:20.321416 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:35:24.009019 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:24.008956 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-49c82ebd-59b45759bf-hjk4h" podUID="3d5b9979-482e-4416-b861-2028b0a5a2da" containerName="payload-processing" containerID="cri-o://a7800305d33c61320c0175fa5a37f6d8d97186cc73d68b1b45de5b8ee591c7d1" gracePeriod=25 Nov 01 21:35:24.117184 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:24.117137 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-49c82ebd-649fbdcc87-7rzw2" podUID="628c73a8-34d3-409e-a38d-b97dcda63da7" containerName="payload-pre-processing" containerID="cri-o://fef79252397e098825a961dc6435f17136ca394ec294662e2058ea36e824299e" gracePeriod=25 Nov 01 21:35:24.268356 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:24.268281 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-49c82ebd-59b45759bf-hjk4h" Nov 01 21:35:24.321313 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:24.321280 4484 generic.go:356] "Generic (PLEG): container finished" podID="3d5b9979-482e-4416-b861-2028b0a5a2da" containerID="a7800305d33c61320c0175fa5a37f6d8d97186cc73d68b1b45de5b8ee591c7d1" exitCode=0 Nov 01 21:35:24.321519 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:24.321354 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-49c82ebd-59b45759bf-hjk4h" Nov 01 21:35:24.321519 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:24.321362 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-49c82ebd-59b45759bf-hjk4h" event={"ID":"3d5b9979-482e-4416-b861-2028b0a5a2da","Type":"ContainerDied","Data":"a7800305d33c61320c0175fa5a37f6d8d97186cc73d68b1b45de5b8ee591c7d1"} Nov 01 21:35:24.321519 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:24.321431 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-49c82ebd-59b45759bf-hjk4h" event={"ID":"3d5b9979-482e-4416-b861-2028b0a5a2da","Type":"ContainerDied","Data":"42a46932f734c71186cd126aceeebe339b11d6e17fa393829da48dc98133918d"} Nov 01 21:35:24.321519 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:24.321465 4484 scope.go:122] "RemoveContainer" containerID="a7800305d33c61320c0175fa5a37f6d8d97186cc73d68b1b45de5b8ee591c7d1" Nov 01 21:35:24.323287 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:24.323266 4484 generic.go:356] "Generic (PLEG): container finished" podID="628c73a8-34d3-409e-a38d-b97dcda63da7" containerID="fef79252397e098825a961dc6435f17136ca394ec294662e2058ea36e824299e" exitCode=0 Nov 01 21:35:24.323437 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:24.323308 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-49c82ebd-649fbdcc87-7rzw2" event={"ID":"628c73a8-34d3-409e-a38d-b97dcda63da7","Type":"ContainerDied","Data":"fef79252397e098825a961dc6435f17136ca394ec294662e2058ea36e824299e"} Nov 01 21:35:24.332923 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:24.332899 4484 scope.go:122] "RemoveContainer" containerID="a7800305d33c61320c0175fa5a37f6d8d97186cc73d68b1b45de5b8ee591c7d1" Nov 01 21:35:24.333248 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:35:24.333223 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7800305d33c61320c0175fa5a37f6d8d97186cc73d68b1b45de5b8ee591c7d1\": container with ID starting with a7800305d33c61320c0175fa5a37f6d8d97186cc73d68b1b45de5b8ee591c7d1 not found: ID does not exist" containerID="a7800305d33c61320c0175fa5a37f6d8d97186cc73d68b1b45de5b8ee591c7d1" Nov 01 21:35:24.333363 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:24.333258 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7800305d33c61320c0175fa5a37f6d8d97186cc73d68b1b45de5b8ee591c7d1"} err="failed to get container status \"a7800305d33c61320c0175fa5a37f6d8d97186cc73d68b1b45de5b8ee591c7d1\": rpc error: code = NotFound desc = could not find container \"a7800305d33c61320c0175fa5a37f6d8d97186cc73d68b1b45de5b8ee591c7d1\": container with ID starting with a7800305d33c61320c0175fa5a37f6d8d97186cc73d68b1b45de5b8ee591c7d1 not found: ID does not exist" Nov 01 21:35:24.343557 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:24.343529 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d5b9979-482e-4416-b861-2028b0a5a2da-kube-api-access-p86tb\" (UniqueName: \"kubernetes.io/projected/3d5b9979-482e-4416-b861-2028b0a5a2da-kube-api-access-p86tb\") pod \"3d5b9979-482e-4416-b861-2028b0a5a2da\" (UID: \"3d5b9979-482e-4416-b861-2028b0a5a2da\") " Nov 01 21:35:24.343728 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:24.343682 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d5b9979-482e-4416-b861-2028b0a5a2da-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d5b9979-482e-4416-b861-2028b0a5a2da-plugins-config-volume\") pod \"3d5b9979-482e-4416-b861-2028b0a5a2da\" (UID: \"3d5b9979-482e-4416-b861-2028b0a5a2da\") " Nov 01 21:35:24.344144 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:24.344115 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d5b9979-482e-4416-b861-2028b0a5a2da-plugins-config-volume" pod "3d5b9979-482e-4416-b861-2028b0a5a2da" (UID: "3d5b9979-482e-4416-b861-2028b0a5a2da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:35:24.345791 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:24.345759 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d5b9979-482e-4416-b861-2028b0a5a2da-kube-api-access-p86tb" pod "3d5b9979-482e-4416-b861-2028b0a5a2da" (UID: "3d5b9979-482e-4416-b861-2028b0a5a2da"). InnerVolumeSpecName "kube-api-access-p86tb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:35:24.365145 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:24.365121 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-49c82ebd-649fbdcc87-7rzw2" Nov 01 21:35:24.445243 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:24.445209 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/628c73a8-34d3-409e-a38d-b97dcda63da7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/628c73a8-34d3-409e-a38d-b97dcda63da7-plugins-config-volume\") pod \"628c73a8-34d3-409e-a38d-b97dcda63da7\" (UID: \"628c73a8-34d3-409e-a38d-b97dcda63da7\") " Nov 01 21:35:24.445428 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:24.445253 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/628c73a8-34d3-409e-a38d-b97dcda63da7-kube-api-access-fhgxl\" (UniqueName: \"kubernetes.io/projected/628c73a8-34d3-409e-a38d-b97dcda63da7-kube-api-access-fhgxl\") pod \"628c73a8-34d3-409e-a38d-b97dcda63da7\" (UID: \"628c73a8-34d3-409e-a38d-b97dcda63da7\") " Nov 01 21:35:24.445646 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:24.445623 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/628c73a8-34d3-409e-a38d-b97dcda63da7-plugins-config-volume" pod "628c73a8-34d3-409e-a38d-b97dcda63da7" (UID: "628c73a8-34d3-409e-a38d-b97dcda63da7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:35:24.445753 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:24.445646 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d5b9979-482e-4416-b861-2028b0a5a2da-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:35:24.445753 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:24.445691 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p86tb\" (UniqueName: \"kubernetes.io/projected/3d5b9979-482e-4416-b861-2028b0a5a2da-kube-api-access-p86tb\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:35:24.448083 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:24.448057 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/628c73a8-34d3-409e-a38d-b97dcda63da7-kube-api-access-fhgxl" pod "628c73a8-34d3-409e-a38d-b97dcda63da7" (UID: "628c73a8-34d3-409e-a38d-b97dcda63da7"). InnerVolumeSpecName "kube-api-access-fhgxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:35:24.510812 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:24.510781 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:35:24.546636 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:24.546544 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/628c73a8-34d3-409e-a38d-b97dcda63da7-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:35:24.546636 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:24.546575 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhgxl\" (UniqueName: \"kubernetes.io/projected/628c73a8-34d3-409e-a38d-b97dcda63da7-kube-api-access-fhgxl\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:35:24.745148 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:24.745111 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-49c82ebd-59b45759bf-hjk4h"] Nov 01 21:35:24.751102 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:24.751075 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-49c82ebd-59b45759bf-hjk4h"] Nov 01 21:35:25.339536 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:25.339442 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-49c82ebd-649fbdcc87-7rzw2" event={"ID":"628c73a8-34d3-409e-a38d-b97dcda63da7","Type":"ContainerDied","Data":"4c0960daf7f6b27225ded231c821f6200c12e388c38f557c09bc69f3414de858"} Nov 01 21:35:25.339536 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:25.339497 4484 scope.go:122] "RemoveContainer" containerID="fef79252397e098825a961dc6435f17136ca394ec294662e2058ea36e824299e" Nov 01 21:35:25.339536 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:25.339582 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-49c82ebd-649fbdcc87-7rzw2" Nov 01 21:35:25.370537 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:25.370504 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d5b9979-482e-4416-b861-2028b0a5a2da" path="/var/lib/kubelet/pods/3d5b9979-482e-4416-b861-2028b0a5a2da/volumes" Nov 01 21:35:25.507329 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:25.507285 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-49c82ebd-649fbdcc87-7rzw2"] Nov 01 21:35:25.557240 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:25.557208 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-49c82ebd-649fbdcc87-7rzw2"] Nov 01 21:35:26.017385 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:26.017353 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:35:26.299893 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:26.299809 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:35:26.496941 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:26.496903 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49c82ebd-hxrqv"] Nov 01 21:35:26.547653 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:26.547620 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49c82ebd-hxrqv"] Nov 01 21:35:27.369680 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:27.369647 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4253b695-93ed-4cd7-bdb6-562401a387a2" path="/var/lib/kubelet/pods/4253b695-93ed-4cd7-bdb6-562401a387a2/volumes" Nov 01 21:35:27.370068 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:27.370054 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="628c73a8-34d3-409e-a38d-b97dcda63da7" path="/var/lib/kubelet/pods/628c73a8-34d3-409e-a38d-b97dcda63da7/volumes" Nov 01 21:35:28.389443 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:28.389407 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8"] Nov 01 21:35:28.389965 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:28.389602 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8" podUID="38ce6dc7-e34a-48da-a283-f140baa31047" containerName="istio-proxy" containerID="cri-o://4b3f6dc14241ebd96f174cd2061e6455fec55c28b3c74591852d0e86de605283" gracePeriod=30 Nov 01 21:35:28.490114 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:28.490048 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:35:29.567861 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:29.567792 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:35:30.073877 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:30.073834 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx"] Nov 01 21:35:30.074368 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:30.074349 4484 cpu_manager.go:414] "containerMap: removing container" podUID="3d5b9979-482e-4416-b861-2028b0a5a2da" containerName="payload-processing" Nov 01 21:35:30.074501 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:30.074369 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d5b9979-482e-4416-b861-2028b0a5a2da" containerName="payload-processing" Nov 01 21:35:30.074501 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:30.074408 4484 cpu_manager.go:414] "containerMap: removing container" podUID="628c73a8-34d3-409e-a38d-b97dcda63da7" containerName="payload-pre-processing" Nov 01 21:35:30.074501 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:30.074416 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="628c73a8-34d3-409e-a38d-b97dcda63da7" containerName="payload-pre-processing" Nov 01 21:35:30.074501 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:30.074444 4484 cpu_manager.go:414] "containerMap: removing container" podUID="4253b695-93ed-4cd7-bdb6-562401a387a2" containerName="revoke-keys" Nov 01 21:35:30.074501 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:30.074452 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4253b695-93ed-4cd7-bdb6-562401a387a2" containerName="revoke-keys" Nov 01 21:35:30.074781 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:30.074564 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="628c73a8-34d3-409e-a38d-b97dcda63da7" containerName="payload-pre-processing" Nov 01 21:35:30.074781 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:30.074579 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="4253b695-93ed-4cd7-bdb6-562401a387a2" containerName="revoke-keys" Nov 01 21:35:30.074781 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:30.074596 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d5b9979-482e-4416-b861-2028b0a5a2da" containerName="payload-processing" Nov 01 21:35:30.078064 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:30.078040 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx" Nov 01 21:35:30.131168 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:30.131125 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-e23c3b-openshift-default-dockercfg-xfk5l\"" Nov 01 21:35:30.177158 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:30.177124 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx"] Nov 01 21:35:30.189963 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:30.189927 4484 patch_prober.go:29] interesting pod/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 21:35:30.190158 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:30.189984 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8" podUID="38ce6dc7-e34a-48da-a283-f140baa31047" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 21:35:30.226477 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:30.226444 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd4411b-c555-4088-aa6f-52ff7244ad98-credential-socket\") pod \"e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx\" (UID: \"cfd4411b-c555-4088-aa6f-52ff7244ad98\") " pod="openshift-ingress/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx" Nov 01 21:35:30.226676 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:30.226484 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjtlt\" (UniqueName: \"kubernetes.io/projected/cfd4411b-c555-4088-aa6f-52ff7244ad98-kube-api-access-hjtlt\") pod \"e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx\" (UID: \"cfd4411b-c555-4088-aa6f-52ff7244ad98\") " pod="openshift-ingress/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx" Nov 01 21:35:30.226676 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:30.226505 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfd4411b-c555-4088-aa6f-52ff7244ad98-istio-token\") pod \"e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx\" (UID: \"cfd4411b-c555-4088-aa6f-52ff7244ad98\") " pod="openshift-ingress/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx" Nov 01 21:35:30.226676 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:30.226599 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfd4411b-c555-4088-aa6f-52ff7244ad98-istio-podinfo\") pod \"e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx\" (UID: \"cfd4411b-c555-4088-aa6f-52ff7244ad98\") " pod="openshift-ingress/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx" Nov 01 21:35:30.226676 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:30.226656 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfd4411b-c555-4088-aa6f-52ff7244ad98-istiod-ca-cert\") pod \"e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx\" (UID: \"cfd4411b-c555-4088-aa6f-52ff7244ad98\") " pod="openshift-ingress/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx" Nov 01 21:35:30.226876 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:30.226679 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfd4411b-c555-4088-aa6f-52ff7244ad98-istio-envoy\") pod \"e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx\" (UID: \"cfd4411b-c555-4088-aa6f-52ff7244ad98\") " pod="openshift-ingress/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx" Nov 01 21:35:30.226876 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:30.226761 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfd4411b-c555-4088-aa6f-52ff7244ad98-workload-certs\") pod \"e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx\" (UID: \"cfd4411b-c555-4088-aa6f-52ff7244ad98\") " pod="openshift-ingress/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx" Nov 01 21:35:30.226876 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:30.226787 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfd4411b-c555-4088-aa6f-52ff7244ad98-istio-data\") pod \"e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx\" (UID: \"cfd4411b-c555-4088-aa6f-52ff7244ad98\") " pod="openshift-ingress/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx" Nov 01 21:35:30.227031 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:30.226906 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd4411b-c555-4088-aa6f-52ff7244ad98-workload-socket\") pod \"e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx\" (UID: \"cfd4411b-c555-4088-aa6f-52ff7244ad98\") " pod="openshift-ingress/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx" Nov 01 21:35:30.233025 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:30.232985 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:35:30.331733 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:30.331627 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd4411b-c555-4088-aa6f-52ff7244ad98-credential-socket\") pod \"e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx\" (UID: \"cfd4411b-c555-4088-aa6f-52ff7244ad98\") " pod="openshift-ingress/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx" Nov 01 21:35:30.331733 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:30.331683 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjtlt\" (UniqueName: \"kubernetes.io/projected/cfd4411b-c555-4088-aa6f-52ff7244ad98-kube-api-access-hjtlt\") pod \"e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx\" (UID: \"cfd4411b-c555-4088-aa6f-52ff7244ad98\") " pod="openshift-ingress/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx" Nov 01 21:35:30.331733 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:30.331715 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfd4411b-c555-4088-aa6f-52ff7244ad98-istio-token\") pod \"e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx\" (UID: \"cfd4411b-c555-4088-aa6f-52ff7244ad98\") " pod="openshift-ingress/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx" Nov 01 21:35:30.332261 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:30.331746 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfd4411b-c555-4088-aa6f-52ff7244ad98-istio-podinfo\") pod \"e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx\" (UID: \"cfd4411b-c555-4088-aa6f-52ff7244ad98\") " pod="openshift-ingress/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx" Nov 01 21:35:30.332261 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:30.331800 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfd4411b-c555-4088-aa6f-52ff7244ad98-istiod-ca-cert\") pod \"e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx\" (UID: \"cfd4411b-c555-4088-aa6f-52ff7244ad98\") " pod="openshift-ingress/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx" Nov 01 21:35:30.332261 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:30.331834 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfd4411b-c555-4088-aa6f-52ff7244ad98-istio-envoy\") pod \"e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx\" (UID: \"cfd4411b-c555-4088-aa6f-52ff7244ad98\") " pod="openshift-ingress/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx" Nov 01 21:35:30.332261 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:30.331865 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfd4411b-c555-4088-aa6f-52ff7244ad98-workload-certs\") pod \"e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx\" (UID: \"cfd4411b-c555-4088-aa6f-52ff7244ad98\") " pod="openshift-ingress/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx" Nov 01 21:35:30.332261 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:30.331900 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfd4411b-c555-4088-aa6f-52ff7244ad98-istio-data\") pod \"e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx\" (UID: \"cfd4411b-c555-4088-aa6f-52ff7244ad98\") " pod="openshift-ingress/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx" Nov 01 21:35:30.332261 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:30.332022 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd4411b-c555-4088-aa6f-52ff7244ad98-workload-socket\") pod \"e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx\" (UID: \"cfd4411b-c555-4088-aa6f-52ff7244ad98\") " pod="openshift-ingress/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx" Nov 01 21:35:30.334927 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:30.334200 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd4411b-c555-4088-aa6f-52ff7244ad98-credential-socket\") pod \"e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx\" (UID: \"cfd4411b-c555-4088-aa6f-52ff7244ad98\") " pod="openshift-ingress/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx" Nov 01 21:35:30.334927 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:30.334502 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfd4411b-c555-4088-aa6f-52ff7244ad98-workload-certs\") pod \"e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx\" (UID: \"cfd4411b-c555-4088-aa6f-52ff7244ad98\") " pod="openshift-ingress/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx" Nov 01 21:35:30.334927 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:30.334692 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd4411b-c555-4088-aa6f-52ff7244ad98-workload-socket\") pod \"e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx\" (UID: \"cfd4411b-c555-4088-aa6f-52ff7244ad98\") " pod="openshift-ingress/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx" Nov 01 21:35:30.334927 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:30.334827 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfd4411b-c555-4088-aa6f-52ff7244ad98-istio-data\") pod \"e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx\" (UID: \"cfd4411b-c555-4088-aa6f-52ff7244ad98\") " pod="openshift-ingress/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx" Nov 01 21:35:30.335311 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:30.335288 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfd4411b-c555-4088-aa6f-52ff7244ad98-istiod-ca-cert\") pod \"e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx\" (UID: \"cfd4411b-c555-4088-aa6f-52ff7244ad98\") " pod="openshift-ingress/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx" Nov 01 21:35:30.342493 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:30.342415 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfd4411b-c555-4088-aa6f-52ff7244ad98-istio-envoy\") pod \"e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx\" (UID: \"cfd4411b-c555-4088-aa6f-52ff7244ad98\") " pod="openshift-ingress/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx" Nov 01 21:35:30.343288 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:30.343256 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfd4411b-c555-4088-aa6f-52ff7244ad98-istio-podinfo\") pod \"e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx\" (UID: \"cfd4411b-c555-4088-aa6f-52ff7244ad98\") " pod="openshift-ingress/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx" Nov 01 21:35:30.391099 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:30.391059 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfd4411b-c555-4088-aa6f-52ff7244ad98-istio-token\") pod \"e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx\" (UID: \"cfd4411b-c555-4088-aa6f-52ff7244ad98\") " pod="openshift-ingress/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx" Nov 01 21:35:30.391486 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:30.391464 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjtlt\" (UniqueName: \"kubernetes.io/projected/cfd4411b-c555-4088-aa6f-52ff7244ad98-kube-api-access-hjtlt\") pod \"e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx\" (UID: \"cfd4411b-c555-4088-aa6f-52ff7244ad98\") " pod="openshift-ingress/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx" Nov 01 21:35:30.404683 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:30.404648 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:35:30.691804 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:30.691719 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx" Nov 01 21:35:31.121819 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:31.121786 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx"] Nov 01 21:35:31.123490 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:35:31.123455 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfd4411b_c555_4088_aa6f_52ff7244ad98.slice/crio-2eb522f80014de8edd94799da47fa5f74f9ded08c375d748820ec4c3705c4cb5 WatchSource:0}: Error finding container 2eb522f80014de8edd94799da47fa5f74f9ded08c375d748820ec4c3705c4cb5: Status 404 returned error can't find the container with id 2eb522f80014de8edd94799da47fa5f74f9ded08c375d748820ec4c3705c4cb5 Nov 01 21:35:31.125749 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:31.125721 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:35:31.125818 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:31.125782 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:35:31.125818 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:31.125812 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:35:31.374036 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:31.373928 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx" event={"ID":"cfd4411b-c555-4088-aa6f-52ff7244ad98","Type":"ContainerStarted","Data":"2bc446fc5882b13c914a97ef037f9401ae065a45144af0739dc37977428c3bc2"} Nov 01 21:35:31.374036 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:31.373989 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx" event={"ID":"cfd4411b-c555-4088-aa6f-52ff7244ad98","Type":"ContainerStarted","Data":"2eb522f80014de8edd94799da47fa5f74f9ded08c375d748820ec4c3705c4cb5"} Nov 01 21:35:31.492440 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:31.492386 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx" podStartSLOduration=1.492371277 podStartE2EDuration="1.492371277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:35:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:35:31.491377759 +0000 UTC m=+3882.269017894" watchObservedRunningTime="2026-11-01 21:35:31.492371277 +0000 UTC m=+3882.270011457" Nov 01 21:35:31.692519 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:31.692427 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx" Nov 01 21:35:32.697694 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:32.697663 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx" Nov 01 21:35:33.383946 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:33.383915 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx" Nov 01 21:35:33.384981 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:33.384957 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx" Nov 01 21:35:33.663942 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:33.663907 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8" Nov 01 21:35:33.772708 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:33.772675 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38ce6dc7-e34a-48da-a283-f140baa31047-kube-api-access-mnhlw\" (UniqueName: \"kubernetes.io/projected/38ce6dc7-e34a-48da-a283-f140baa31047-kube-api-access-mnhlw\") pod \"38ce6dc7-e34a-48da-a283-f140baa31047\" (UID: \"38ce6dc7-e34a-48da-a283-f140baa31047\") " Nov 01 21:35:33.773194 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:33.772798 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38ce6dc7-e34a-48da-a283-f140baa31047-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ce6dc7-e34a-48da-a283-f140baa31047-workload-socket\") pod \"38ce6dc7-e34a-48da-a283-f140baa31047\" (UID: \"38ce6dc7-e34a-48da-a283-f140baa31047\") " Nov 01 21:35:33.773194 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:33.772822 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38ce6dc7-e34a-48da-a283-f140baa31047-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38ce6dc7-e34a-48da-a283-f140baa31047-istiod-ca-cert\") pod \"38ce6dc7-e34a-48da-a283-f140baa31047\" (UID: \"38ce6dc7-e34a-48da-a283-f140baa31047\") " Nov 01 21:35:33.773194 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:33.772840 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/38ce6dc7-e34a-48da-a283-f140baa31047-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38ce6dc7-e34a-48da-a283-f140baa31047-istio-podinfo\") pod \"38ce6dc7-e34a-48da-a283-f140baa31047\" (UID: \"38ce6dc7-e34a-48da-a283-f140baa31047\") " Nov 01 21:35:33.773194 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:33.772860 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38ce6dc7-e34a-48da-a283-f140baa31047-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38ce6dc7-e34a-48da-a283-f140baa31047-workload-certs\") pod \"38ce6dc7-e34a-48da-a283-f140baa31047\" (UID: \"38ce6dc7-e34a-48da-a283-f140baa31047\") " Nov 01 21:35:33.773194 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:33.772898 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38ce6dc7-e34a-48da-a283-f140baa31047-istio-token\" (UniqueName: \"kubernetes.io/projected/38ce6dc7-e34a-48da-a283-f140baa31047-istio-token\") pod \"38ce6dc7-e34a-48da-a283-f140baa31047\" (UID: \"38ce6dc7-e34a-48da-a283-f140baa31047\") " Nov 01 21:35:33.773194 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:33.772937 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38ce6dc7-e34a-48da-a283-f140baa31047-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38ce6dc7-e34a-48da-a283-f140baa31047-istio-data\") pod \"38ce6dc7-e34a-48da-a283-f140baa31047\" (UID: \"38ce6dc7-e34a-48da-a283-f140baa31047\") " Nov 01 21:35:33.773194 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:33.772989 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38ce6dc7-e34a-48da-a283-f140baa31047-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ce6dc7-e34a-48da-a283-f140baa31047-credential-socket\") pod \"38ce6dc7-e34a-48da-a283-f140baa31047\" (UID: \"38ce6dc7-e34a-48da-a283-f140baa31047\") " Nov 01 21:35:33.773194 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:33.773070 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38ce6dc7-e34a-48da-a283-f140baa31047-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38ce6dc7-e34a-48da-a283-f140baa31047-istio-envoy\") pod \"38ce6dc7-e34a-48da-a283-f140baa31047\" (UID: \"38ce6dc7-e34a-48da-a283-f140baa31047\") " Nov 01 21:35:33.773627 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:33.773191 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38ce6dc7-e34a-48da-a283-f140baa31047-workload-socket" pod "38ce6dc7-e34a-48da-a283-f140baa31047" (UID: "38ce6dc7-e34a-48da-a283-f140baa31047"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:35:33.773627 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:33.773221 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38ce6dc7-e34a-48da-a283-f140baa31047-workload-certs" pod "38ce6dc7-e34a-48da-a283-f140baa31047" (UID: "38ce6dc7-e34a-48da-a283-f140baa31047"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:35:33.773627 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:33.773293 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38ce6dc7-e34a-48da-a283-f140baa31047-istio-data" pod "38ce6dc7-e34a-48da-a283-f140baa31047" (UID: "38ce6dc7-e34a-48da-a283-f140baa31047"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:35:33.773627 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:33.773365 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38ce6dc7-e34a-48da-a283-f140baa31047-istiod-ca-cert" pod "38ce6dc7-e34a-48da-a283-f140baa31047" (UID: "38ce6dc7-e34a-48da-a283-f140baa31047"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:35:33.773627 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:33.773360 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38ce6dc7-e34a-48da-a283-f140baa31047-credential-socket" pod "38ce6dc7-e34a-48da-a283-f140baa31047" (UID: "38ce6dc7-e34a-48da-a283-f140baa31047"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:35:33.773627 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:33.773548 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38ce6dc7-e34a-48da-a283-f140baa31047-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:35:33.773627 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:33.773569 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38ce6dc7-e34a-48da-a283-f140baa31047-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:35:33.773627 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:33.773584 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ce6dc7-e34a-48da-a283-f140baa31047-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:35:33.773627 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:33.773598 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ce6dc7-e34a-48da-a283-f140baa31047-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:35:33.773627 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:33.773611 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38ce6dc7-e34a-48da-a283-f140baa31047-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:35:33.775188 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:33.775160 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38ce6dc7-e34a-48da-a283-f140baa31047-istio-podinfo" pod "38ce6dc7-e34a-48da-a283-f140baa31047" (UID: "38ce6dc7-e34a-48da-a283-f140baa31047"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:35:33.775433 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:33.775403 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38ce6dc7-e34a-48da-a283-f140baa31047-kube-api-access-mnhlw" pod "38ce6dc7-e34a-48da-a283-f140baa31047" (UID: "38ce6dc7-e34a-48da-a283-f140baa31047"). InnerVolumeSpecName "kube-api-access-mnhlw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:35:33.775668 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:33.775646 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38ce6dc7-e34a-48da-a283-f140baa31047-istio-envoy" pod "38ce6dc7-e34a-48da-a283-f140baa31047" (UID: "38ce6dc7-e34a-48da-a283-f140baa31047"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:35:33.775861 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:33.775844 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38ce6dc7-e34a-48da-a283-f140baa31047-istio-token" pod "38ce6dc7-e34a-48da-a283-f140baa31047" (UID: "38ce6dc7-e34a-48da-a283-f140baa31047"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:35:33.874644 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:33.874600 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38ce6dc7-e34a-48da-a283-f140baa31047-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:35:33.874644 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:33.874640 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38ce6dc7-e34a-48da-a283-f140baa31047-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:35:33.874644 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:33.874652 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38ce6dc7-e34a-48da-a283-f140baa31047-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:35:33.874889 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:33.874665 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnhlw\" (UniqueName: \"kubernetes.io/projected/38ce6dc7-e34a-48da-a283-f140baa31047-kube-api-access-mnhlw\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:35:34.206808 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:34.206775 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:35:34.375098 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:34.375064 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:35:34.391062 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:34.391029 4484 generic.go:356] "Generic (PLEG): container finished" podID="38ce6dc7-e34a-48da-a283-f140baa31047" containerID="4b3f6dc14241ebd96f174cd2061e6455fec55c28b3c74591852d0e86de605283" exitCode=0 Nov 01 21:35:34.391232 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:34.391089 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8" Nov 01 21:35:34.391232 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:34.391098 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8" event={"ID":"38ce6dc7-e34a-48da-a283-f140baa31047","Type":"ContainerDied","Data":"4b3f6dc14241ebd96f174cd2061e6455fec55c28b3c74591852d0e86de605283"} Nov 01 21:35:34.391232 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:34.391156 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8" event={"ID":"38ce6dc7-e34a-48da-a283-f140baa31047","Type":"ContainerDied","Data":"c287e088ad16d52813d356ba9c954570525c386ff2df43815e0fe827a96d78b4"} Nov 01 21:35:34.391232 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:34.391185 4484 scope.go:122] "RemoveContainer" containerID="4b3f6dc14241ebd96f174cd2061e6455fec55c28b3c74591852d0e86de605283" Nov 01 21:35:34.402662 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:34.402643 4484 scope.go:122] "RemoveContainer" containerID="11eec086e09984b31ff6727bc5f8e31d136be7de9735d40d7efb09d36f8c4bad" Nov 01 21:35:34.410823 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:34.410802 4484 scope.go:122] "RemoveContainer" containerID="4b3f6dc14241ebd96f174cd2061e6455fec55c28b3c74591852d0e86de605283" Nov 01 21:35:34.411145 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:35:34.411124 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b3f6dc14241ebd96f174cd2061e6455fec55c28b3c74591852d0e86de605283\": container with ID starting with 4b3f6dc14241ebd96f174cd2061e6455fec55c28b3c74591852d0e86de605283 not found: ID does not exist" containerID="4b3f6dc14241ebd96f174cd2061e6455fec55c28b3c74591852d0e86de605283" Nov 01 21:35:34.411245 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:34.411155 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b3f6dc14241ebd96f174cd2061e6455fec55c28b3c74591852d0e86de605283"} err="failed to get container status \"4b3f6dc14241ebd96f174cd2061e6455fec55c28b3c74591852d0e86de605283\": rpc error: code = NotFound desc = could not find container \"4b3f6dc14241ebd96f174cd2061e6455fec55c28b3c74591852d0e86de605283\": container with ID starting with 4b3f6dc14241ebd96f174cd2061e6455fec55c28b3c74591852d0e86de605283 not found: ID does not exist" Nov 01 21:35:34.411245 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:34.411179 4484 scope.go:122] "RemoveContainer" containerID="11eec086e09984b31ff6727bc5f8e31d136be7de9735d40d7efb09d36f8c4bad" Nov 01 21:35:34.411436 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:35:34.411416 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11eec086e09984b31ff6727bc5f8e31d136be7de9735d40d7efb09d36f8c4bad\": container with ID starting with 11eec086e09984b31ff6727bc5f8e31d136be7de9735d40d7efb09d36f8c4bad not found: ID does not exist" containerID="11eec086e09984b31ff6727bc5f8e31d136be7de9735d40d7efb09d36f8c4bad" Nov 01 21:35:34.411512 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:34.411444 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11eec086e09984b31ff6727bc5f8e31d136be7de9735d40d7efb09d36f8c4bad"} err="failed to get container status \"11eec086e09984b31ff6727bc5f8e31d136be7de9735d40d7efb09d36f8c4bad\": rpc error: code = NotFound desc = could not find container \"11eec086e09984b31ff6727bc5f8e31d136be7de9735d40d7efb09d36f8c4bad\": container with ID starting with 11eec086e09984b31ff6727bc5f8e31d136be7de9735d40d7efb09d36f8c4bad not found: ID does not exist" Nov 01 21:35:34.437595 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:35:34.437559 4484 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38ce6dc7_e34a_48da_a283_f140baa31047.slice/crio-c287e088ad16d52813d356ba9c954570525c386ff2df43815e0fe827a96d78b4\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38ce6dc7_e34a_48da_a283_f140baa31047.slice\": RecentStats: unable to find data in memory cache]" Nov 01 21:35:34.610667 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:34.610629 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8"] Nov 01 21:35:34.617712 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:34.617684 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:35:34.661552 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:34.661514 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-49c82ebd-openshift-default-678b9f6cdc-mzsk8"] Nov 01 21:35:35.370028 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:35.369981 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38ce6dc7-e34a-48da-a283-f140baa31047" path="/var/lib/kubelet/pods/38ce6dc7-e34a-48da-a283-f140baa31047/volumes" Nov 01 21:35:37.433196 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:37.433163 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:35:37.606049 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:37.606014 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:35:37.822932 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:37.822899 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-e23c3b-75f45789f9-rwh6w"] Nov 01 21:35:37.823340 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:37.823326 4484 cpu_manager.go:414] "containerMap: removing container" podUID="38ce6dc7-e34a-48da-a283-f140baa31047" containerName="istio-proxy" Nov 01 21:35:37.823385 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:37.823341 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38ce6dc7-e34a-48da-a283-f140baa31047" containerName="istio-proxy" Nov 01 21:35:37.823385 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:37.823364 4484 cpu_manager.go:414] "containerMap: removing container" podUID="38ce6dc7-e34a-48da-a283-f140baa31047" containerName="istio-proxy" Nov 01 21:35:37.823385 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:37.823370 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38ce6dc7-e34a-48da-a283-f140baa31047" containerName="istio-proxy" Nov 01 21:35:37.823479 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:37.823460 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="38ce6dc7-e34a-48da-a283-f140baa31047" containerName="istio-proxy" Nov 01 21:35:37.827893 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:37.827876 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-e23c3b-75f45789f9-rwh6w" Nov 01 21:35:37.882634 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:37.882593 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-e23c3b-75f45789f9-rwh6w"] Nov 01 21:35:37.920884 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:37.920834 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n226j\" (UniqueName: \"kubernetes.io/projected/4a0d583e-ad77-4ff6-975f-c3b630bbfc2e-kube-api-access-n226j\") pod \"payload-pre-processing-e2e-authcfg-http-e23c3b-75f45789f9-rwh6w\" (UID: \"4a0d583e-ad77-4ff6-975f-c3b630bbfc2e\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-e23c3b-75f45789f9-rwh6w" Nov 01 21:35:37.921149 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:37.921122 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a0d583e-ad77-4ff6-975f-c3b630bbfc2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-e23c3b-75f45789f9-rwh6w\" (UID: \"4a0d583e-ad77-4ff6-975f-c3b630bbfc2e\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-e23c3b-75f45789f9-rwh6w" Nov 01 21:35:37.931136 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:37.931103 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-e23c3b-dockercfg-mwbft\"" Nov 01 21:35:37.931399 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:37.931103 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-e23c3b\"" Nov 01 21:35:38.022285 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:38.022242 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n226j\" (UniqueName: \"kubernetes.io/projected/4a0d583e-ad77-4ff6-975f-c3b630bbfc2e-kube-api-access-n226j\") pod \"payload-pre-processing-e2e-authcfg-http-e23c3b-75f45789f9-rwh6w\" (UID: \"4a0d583e-ad77-4ff6-975f-c3b630bbfc2e\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-e23c3b-75f45789f9-rwh6w" Nov 01 21:35:38.022494 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:38.022355 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a0d583e-ad77-4ff6-975f-c3b630bbfc2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-e23c3b-75f45789f9-rwh6w\" (UID: \"4a0d583e-ad77-4ff6-975f-c3b630bbfc2e\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-e23c3b-75f45789f9-rwh6w" Nov 01 21:35:38.023278 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:38.023252 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a0d583e-ad77-4ff6-975f-c3b630bbfc2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-e23c3b-75f45789f9-rwh6w\" (UID: \"4a0d583e-ad77-4ff6-975f-c3b630bbfc2e\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-e23c3b-75f45789f9-rwh6w" Nov 01 21:35:38.080029 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:38.079917 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n226j\" (UniqueName: \"kubernetes.io/projected/4a0d583e-ad77-4ff6-975f-c3b630bbfc2e-kube-api-access-n226j\") pod \"payload-pre-processing-e2e-authcfg-http-e23c3b-75f45789f9-rwh6w\" (UID: \"4a0d583e-ad77-4ff6-975f-c3b630bbfc2e\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-e23c3b-75f45789f9-rwh6w" Nov 01 21:35:38.142149 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:38.142113 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-e23c3b-75f45789f9-rwh6w" Nov 01 21:35:38.512684 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:38.512652 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-e23c3b-75f45789f9-rwh6w"] Nov 01 21:35:38.514534 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:35:38.514505 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a0d583e_ad77_4ff6_975f_c3b630bbfc2e.slice/crio-1bacf00ec2741beabc0e1c4917b751fc7457ebcf6554f71f376ddf1345e0a22a WatchSource:0}: Error finding container 1bacf00ec2741beabc0e1c4917b751fc7457ebcf6554f71f376ddf1345e0a22a: Status 404 returned error can't find the container with id 1bacf00ec2741beabc0e1c4917b751fc7457ebcf6554f71f376ddf1345e0a22a Nov 01 21:35:39.418922 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:39.418879 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-e23c3b-75f45789f9-rwh6w" event={"ID":"4a0d583e-ad77-4ff6-975f-c3b630bbfc2e","Type":"ContainerStarted","Data":"92c98e485c63af9138e16d4120491d21d97b27ea31277075b36b9e25a8be1b0a"} Nov 01 21:35:39.419153 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:39.418940 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-e23c3b-75f45789f9-rwh6w" event={"ID":"4a0d583e-ad77-4ff6-975f-c3b630bbfc2e","Type":"ContainerStarted","Data":"1bacf00ec2741beabc0e1c4917b751fc7457ebcf6554f71f376ddf1345e0a22a"} Nov 01 21:35:39.419153 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:39.418962 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-e23c3b-75f45789f9-rwh6w" Nov 01 21:35:39.534667 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:39.534599 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-e23c3b-75f45789f9-rwh6w" podStartSLOduration=2.53458639 podStartE2EDuration="2.53458639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:35:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:35:39.533419485 +0000 UTC m=+3890.311059620" watchObservedRunningTime="2026-11-01 21:35:39.53458639 +0000 UTC m=+3890.312226524" Nov 01 21:35:41.174289 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:41.174255 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:35:41.341225 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:41.341182 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:35:41.498219 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:41.498186 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:35:45.766963 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:35:45.766924 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:35:46.134516 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:46.134428 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:35:50.431827 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:50.431677 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-e23c3b-75f45789f9-rwh6w" Nov 01 21:35:50.959948 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:50.959905 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:35:55.234889 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:55.234852 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:35:55.741146 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:55.741103 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:35:58.519404 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:58.519369 4484 generic.go:356] "Generic (PLEG): container finished" podID="cfd4411b-c555-4088-aa6f-52ff7244ad98" containerID="2bc446fc5882b13c914a97ef037f9401ae065a45144af0739dc37977428c3bc2" exitCode=0 Nov 01 21:35:58.519925 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:58.519444 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx" event={"ID":"cfd4411b-c555-4088-aa6f-52ff7244ad98","Type":"ContainerDied","Data":"2bc446fc5882b13c914a97ef037f9401ae065a45144af0739dc37977428c3bc2"} Nov 01 21:35:58.519925 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:58.519828 4484 scope.go:122] "RemoveContainer" containerID="2bc446fc5882b13c914a97ef037f9401ae065a45144af0739dc37977428c3bc2" Nov 01 21:35:58.521171 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:58.521143 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:35:58.521245 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:58.521215 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:35:58.521287 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:58.521250 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:35:58.857374 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:58.857336 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:35:59.532550 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:59.532513 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx" event={"ID":"cfd4411b-c555-4088-aa6f-52ff7244ad98","Type":"ContainerStarted","Data":"c437f31cacc6b8318a6bab5cac49ddfdd9ef89d3304f855fb83de98a9832df4f"} Nov 01 21:35:59.691884 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:59.691850 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx" Nov 01 21:35:59.693196 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:59.693171 4484 patch_prober.go:29] interesting pod/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.69:15021/healthz/ready\": dial tcp 10.134.1.69:15021: connect: connection refused" start-of-body= Nov 01 21:35:59.693328 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:35:59.693215 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx" podUID="cfd4411b-c555-4088-aa6f-52ff7244ad98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.69:15021/healthz/ready\": dial tcp 10.134.1.69:15021: connect: connection refused" Nov 01 21:36:00.692480 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:36:00.692441 4484 patch_prober.go:29] interesting pod/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.69:15021/healthz/ready\": dial tcp 10.134.1.69:15021: connect: connection refused" start-of-body= Nov 01 21:36:00.692910 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:36:00.692490 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx" podUID="cfd4411b-c555-4088-aa6f-52ff7244ad98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.69:15021/healthz/ready\": dial tcp 10.134.1.69:15021: connect: connection refused" Nov 01 21:36:00.961192 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:36:00.961105 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:36:01.693220 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:36:01.693177 4484 patch_prober.go:29] interesting pod/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.69:15021/healthz/ready\": dial tcp 10.134.1.69:15021: connect: connection refused" start-of-body= Nov 01 21:36:01.693642 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:36:01.693242 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx" podUID="cfd4411b-c555-4088-aa6f-52ff7244ad98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.69:15021/healthz/ready\": dial tcp 10.134.1.69:15021: connect: connection refused" Nov 01 21:36:02.692663 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:36:02.692611 4484 patch_prober.go:29] interesting pod/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.69:15021/healthz/ready\": dial tcp 10.134.1.69:15021: connect: connection refused" start-of-body= Nov 01 21:36:02.692862 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:36:02.692697 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx" podUID="cfd4411b-c555-4088-aa6f-52ff7244ad98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.69:15021/healthz/ready\": dial tcp 10.134.1.69:15021: connect: connection refused" Nov 01 21:36:02.990421 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:36:02.990382 4484 scope.go:122] "RemoveContainer" containerID="178435720ae42dbe0b9dc84687db2e054a7994008617a61ff7042129603cfbe9" Nov 01 21:36:03.000252 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:36:03.000225 4484 scope.go:122] "RemoveContainer" containerID="c2599c03b32ee7fe8ad3568c917edba9a2ace4817c9f3288773628791507a411" Nov 01 21:36:03.384834 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:36:03.384743 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx" Nov 01 21:36:03.692737 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:36:03.692637 4484 patch_prober.go:29] interesting pod/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.69:15021/healthz/ready\": dial tcp 10.134.1.69:15021: connect: connection refused" start-of-body= Nov 01 21:36:03.692737 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:36:03.692702 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx" podUID="cfd4411b-c555-4088-aa6f-52ff7244ad98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.69:15021/healthz/ready\": dial tcp 10.134.1.69:15021: connect: connection refused" Nov 01 21:36:04.692251 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:36:04.692203 4484 patch_prober.go:29] interesting pod/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.69:15021/healthz/ready\": dial tcp 10.134.1.69:15021: connect: connection refused" start-of-body= Nov 01 21:36:04.692771 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:36:04.692261 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx" podUID="cfd4411b-c555-4088-aa6f-52ff7244ad98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.69:15021/healthz/ready\": dial tcp 10.134.1.69:15021: connect: connection refused" Nov 01 21:36:05.696660 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:36:05.696626 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx" Nov 01 21:36:05.697522 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:36:05.697500 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx" Nov 01 21:36:05.727359 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:36:05.727330 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:36:10.440847 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:36:10.440815 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:36:15.294284 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:36:15.294246 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:36:15.767117 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:36:15.767081 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:36:16.044392 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:36:16.044289 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71e16860-jwj5n"] Nov 01 21:36:16.095789 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:36:16.095759 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71e16860-jwj5n"] Nov 01 21:36:17.373097 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:36:17.373060 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cbdb6cf-6bf6-461b-b658-6adff40437df" path="/var/lib/kubelet/pods/8cbdb6cf-6bf6-461b-b658-6adff40437df/volumes" Nov 01 21:36:20.016781 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:36:20.016742 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:36:24.729631 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:36:24.729596 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:36:29.528508 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:36:29.528474 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:36:34.266582 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:36:34.266543 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:36:39.059202 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:36:39.059161 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:36:43.830644 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:36:43.830601 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:36:45.766945 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:36:45.766910 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:36:48.653804 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:36:48.653755 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:36:53.378852 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:36:53.378807 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:36:55.610793 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:36:55.610760 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:36:58.120422 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:36:58.120388 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:37:02.926730 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:37:02.926693 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:37:07.728250 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:37:07.728201 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:37:12.463732 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:37:12.463695 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:37:15.767069 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:37:15.767029 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:37:17.249170 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:37:17.249133 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:37:22.029658 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:37:22.029607 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:37:26.829835 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:37:26.829799 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:37:31.555591 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:37:31.555512 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:37:36.338602 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:37:36.338559 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:37:41.127988 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:37:41.127947 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:37:45.765927 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:37:45.765887 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:37:45.923233 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:37:45.923193 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:37:50.648080 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:37:50.648043 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:37:55.438507 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:37:55.438467 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:37:55.890573 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:37:55.890536 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:38:00.143088 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:38:00.143055 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:38:03.073695 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:38:03.073666 4484 scope.go:122] "RemoveContainer" containerID="c8370589ce0af86c4c8f729814d1ed5fe5e04b47f8c7e8ff8143693e21fb2415" Nov 01 21:38:03.082851 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:38:03.082823 4484 scope.go:122] "RemoveContainer" containerID="d2f2b098a7ac1c3981e4475df29591fda74ad03a7318c38d9fbd41eb4a6bc4a6" Nov 01 21:38:04.969928 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:38:04.969887 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:38:05.353649 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:38:05.353616 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:38:06.049219 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:38:06.049183 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nqq79"] Nov 01 21:38:06.099920 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:38:06.099890 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nqq79"] Nov 01 21:38:07.370329 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:38:07.370296 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fbeb785-f1ad-4f9a-8246-6e00398f3ffa" path="/var/lib/kubelet/pods/8fbeb785-f1ad-4f9a-8246-6e00398f3ffa/volumes" Nov 01 21:38:09.693983 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:38:09.693940 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:38:14.421535 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:38:14.421492 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:38:15.766519 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:38:15.766481 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:38:19.175275 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:38:19.175237 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:38:23.969163 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:38:23.969116 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:38:28.730510 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:38:28.730473 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:38:33.532984 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:38:33.532946 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:38:38.042765 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:38:38.042728 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xqsxd"] Nov 01 21:38:38.093366 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:38:38.093333 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xqsxd"] Nov 01 21:38:38.336915 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:38:38.336824 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:38:39.377864 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:38:39.373308 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb2ed355-e8f4-425a-be5d-24ff5b99d9ce" path="/var/lib/kubelet/pods/bb2ed355-e8f4-425a-be5d-24ff5b99d9ce/volumes" Nov 01 21:38:43.121262 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:38:43.121210 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:38:45.766951 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:38:45.766913 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:38:47.920407 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:38:47.920355 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:38:52.736061 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:38:52.736031 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:38:56.234842 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:38:56.234801 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:38:57.443944 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:38:57.443900 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:39:01.334390 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:01.334309 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:39:02.238138 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:02.238104 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:39:06.943344 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:06.943307 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:39:11.048356 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:11.048261 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8a7e60d-zzftn"] Nov 01 21:39:11.099273 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:11.099239 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8a7e60d-zzftn"] Nov 01 21:39:11.370897 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:11.370820 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bdb6494-4dfd-4ff8-9037-e65c81ccd574" path="/var/lib/kubelet/pods/9bdb6494-4dfd-4ff8-9037-e65c81ccd574/volumes" Nov 01 21:39:11.740407 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:11.740375 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:39:15.767402 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:39:15.767365 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:39:16.535630 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:16.535594 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:39:21.338406 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:21.338365 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:39:26.122808 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:26.122775 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:39:27.921914 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:27.921878 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-hdvw9"] Nov 01 21:39:27.922434 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:27.922101 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-hdvw9" podUID="ec0b1a26-5b6c-405c-a042-7141708b6b22" containerName="istio-proxy" containerID="cri-o://192367791b1cd7b5b98efbf801cfc63ec5ec9742a3f76f1c3dc7753f305d3db9" gracePeriod=30 Nov 01 21:39:27.972930 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:27.972884 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2n5h6"] Nov 01 21:39:27.973179 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:27.973138 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2n5h6" podUID="35afbdf8-b193-42b5-9c8f-85471b4288b7" containerName="istio-proxy" containerID="cri-o://83d61343b766754a0443bda62ee8204519448aa5b8ee4a423219845b01ae1cd9" gracePeriod=30 Nov 01 21:39:27.974905 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:27.974881 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qm8qv"] Nov 01 21:39:27.975143 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:27.975097 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qm8qv" podUID="f3754b0a-165a-4693-b99c-c7e112c594c1" containerName="istio-proxy" containerID="cri-o://e575bd0a74419db47776fc2e709ea8b405756332bdc7354daefd9317312964ea" gracePeriod=30 Nov 01 21:39:28.023743 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:28.023709 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6zwld"] Nov 01 21:39:28.023933 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:28.023900 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6zwld" podUID="8e6a4f7c-75e1-4506-afb5-4ab00107a7f0" containerName="istio-proxy" containerID="cri-o://c9b777e4fde78925d8f9a532335030b0dfbaf6d198729fbb6efd6f4736b40122" gracePeriod=30 Nov 01 21:39:30.461190 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:30.461157 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qm8qv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 21:39:30.461608 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:30.461212 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qm8qv" podUID="f3754b0a-165a-4693-b99c-c7e112c594c1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 21:39:31.177954 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:31.177920 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:39:33.231649 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.231375 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-hdvw9" Nov 01 21:39:33.301278 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.301245 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2n5h6" Nov 01 21:39:33.344940 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.344916 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qm8qv" Nov 01 21:39:33.348094 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.348075 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6zwld" Nov 01 21:39:33.406963 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.406934 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35afbdf8-b193-42b5-9c8f-85471b4288b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35afbdf8-b193-42b5-9c8f-85471b4288b7-istio-envoy\") pod \"35afbdf8-b193-42b5-9c8f-85471b4288b7\" (UID: \"35afbdf8-b193-42b5-9c8f-85471b4288b7\") " Nov 01 21:39:33.407170 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.406986 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec0b1a26-5b6c-405c-a042-7141708b6b22-kube-api-access-hbsm6\" (UniqueName: \"kubernetes.io/projected/ec0b1a26-5b6c-405c-a042-7141708b6b22-kube-api-access-hbsm6\") pod \"ec0b1a26-5b6c-405c-a042-7141708b6b22\" (UID: \"ec0b1a26-5b6c-405c-a042-7141708b6b22\") " Nov 01 21:39:33.407170 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.407036 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35afbdf8-b193-42b5-9c8f-85471b4288b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35afbdf8-b193-42b5-9c8f-85471b4288b7-workload-socket\") pod \"35afbdf8-b193-42b5-9c8f-85471b4288b7\" (UID: \"35afbdf8-b193-42b5-9c8f-85471b4288b7\") " Nov 01 21:39:33.407170 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.407069 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec0b1a26-5b6c-405c-a042-7141708b6b22-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec0b1a26-5b6c-405c-a042-7141708b6b22-credential-socket\") pod \"ec0b1a26-5b6c-405c-a042-7141708b6b22\" (UID: \"ec0b1a26-5b6c-405c-a042-7141708b6b22\") " Nov 01 21:39:33.407170 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.407091 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35afbdf8-b193-42b5-9c8f-85471b4288b7-kube-api-access-l5mg4\" (UniqueName: \"kubernetes.io/projected/35afbdf8-b193-42b5-9c8f-85471b4288b7-kube-api-access-l5mg4\") pod \"35afbdf8-b193-42b5-9c8f-85471b4288b7\" (UID: \"35afbdf8-b193-42b5-9c8f-85471b4288b7\") " Nov 01 21:39:33.407170 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.407120 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35afbdf8-b193-42b5-9c8f-85471b4288b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35afbdf8-b193-42b5-9c8f-85471b4288b7-credential-socket\") pod \"35afbdf8-b193-42b5-9c8f-85471b4288b7\" (UID: \"35afbdf8-b193-42b5-9c8f-85471b4288b7\") " Nov 01 21:39:33.407170 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.407164 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec0b1a26-5b6c-405c-a042-7141708b6b22-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec0b1a26-5b6c-405c-a042-7141708b6b22-workload-socket\") pod \"ec0b1a26-5b6c-405c-a042-7141708b6b22\" (UID: \"ec0b1a26-5b6c-405c-a042-7141708b6b22\") " Nov 01 21:39:33.407525 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.407194 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec0b1a26-5b6c-405c-a042-7141708b6b22-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec0b1a26-5b6c-405c-a042-7141708b6b22-istio-envoy\") pod \"ec0b1a26-5b6c-405c-a042-7141708b6b22\" (UID: \"ec0b1a26-5b6c-405c-a042-7141708b6b22\") " Nov 01 21:39:33.407525 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.407241 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35afbdf8-b193-42b5-9c8f-85471b4288b7-istio-token\" (UniqueName: \"kubernetes.io/projected/35afbdf8-b193-42b5-9c8f-85471b4288b7-istio-token\") pod \"35afbdf8-b193-42b5-9c8f-85471b4288b7\" (UID: \"35afbdf8-b193-42b5-9c8f-85471b4288b7\") " Nov 01 21:39:33.407525 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.407305 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35afbdf8-b193-42b5-9c8f-85471b4288b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35afbdf8-b193-42b5-9c8f-85471b4288b7-istio-data\") pod \"35afbdf8-b193-42b5-9c8f-85471b4288b7\" (UID: \"35afbdf8-b193-42b5-9c8f-85471b4288b7\") " Nov 01 21:39:33.407525 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.407345 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec0b1a26-5b6c-405c-a042-7141708b6b22-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec0b1a26-5b6c-405c-a042-7141708b6b22-istio-podinfo\") pod \"ec0b1a26-5b6c-405c-a042-7141708b6b22\" (UID: \"ec0b1a26-5b6c-405c-a042-7141708b6b22\") " Nov 01 21:39:33.407525 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.407400 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec0b1a26-5b6c-405c-a042-7141708b6b22-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec0b1a26-5b6c-405c-a042-7141708b6b22-istio-data\") pod \"ec0b1a26-5b6c-405c-a042-7141708b6b22\" (UID: \"ec0b1a26-5b6c-405c-a042-7141708b6b22\") " Nov 01 21:39:33.407525 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.407420 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35afbdf8-b193-42b5-9c8f-85471b4288b7-workload-socket" pod "35afbdf8-b193-42b5-9c8f-85471b4288b7" (UID: "35afbdf8-b193-42b5-9c8f-85471b4288b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:39:33.407525 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.407420 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec0b1a26-5b6c-405c-a042-7141708b6b22-credential-socket" pod "ec0b1a26-5b6c-405c-a042-7141708b6b22" (UID: "ec0b1a26-5b6c-405c-a042-7141708b6b22"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:39:33.407525 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.407432 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35afbdf8-b193-42b5-9c8f-85471b4288b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35afbdf8-b193-42b5-9c8f-85471b4288b7-workload-certs\") pod \"35afbdf8-b193-42b5-9c8f-85471b4288b7\" (UID: \"35afbdf8-b193-42b5-9c8f-85471b4288b7\") " Nov 01 21:39:33.407525 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.407458 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec0b1a26-5b6c-405c-a042-7141708b6b22-workload-socket" pod "ec0b1a26-5b6c-405c-a042-7141708b6b22" (UID: "ec0b1a26-5b6c-405c-a042-7141708b6b22"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:39:33.408052 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.407540 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec0b1a26-5b6c-405c-a042-7141708b6b22-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec0b1a26-5b6c-405c-a042-7141708b6b22-workload-certs\") pod \"ec0b1a26-5b6c-405c-a042-7141708b6b22\" (UID: \"ec0b1a26-5b6c-405c-a042-7141708b6b22\") " Nov 01 21:39:33.408052 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.407578 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35afbdf8-b193-42b5-9c8f-85471b4288b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35afbdf8-b193-42b5-9c8f-85471b4288b7-istiod-ca-cert\") pod \"35afbdf8-b193-42b5-9c8f-85471b4288b7\" (UID: \"35afbdf8-b193-42b5-9c8f-85471b4288b7\") " Nov 01 21:39:33.408052 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.407611 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec0b1a26-5b6c-405c-a042-7141708b6b22-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec0b1a26-5b6c-405c-a042-7141708b6b22-istiod-ca-cert\") pod \"ec0b1a26-5b6c-405c-a042-7141708b6b22\" (UID: \"ec0b1a26-5b6c-405c-a042-7141708b6b22\") " Nov 01 21:39:33.408052 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.407634 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35afbdf8-b193-42b5-9c8f-85471b4288b7-workload-certs" pod "35afbdf8-b193-42b5-9c8f-85471b4288b7" (UID: "35afbdf8-b193-42b5-9c8f-85471b4288b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:39:33.408052 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.407643 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35afbdf8-b193-42b5-9c8f-85471b4288b7-istio-data" pod "35afbdf8-b193-42b5-9c8f-85471b4288b7" (UID: "35afbdf8-b193-42b5-9c8f-85471b4288b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:39:33.408052 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.407656 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35afbdf8-b193-42b5-9c8f-85471b4288b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35afbdf8-b193-42b5-9c8f-85471b4288b7-istio-podinfo\") pod \"35afbdf8-b193-42b5-9c8f-85471b4288b7\" (UID: \"35afbdf8-b193-42b5-9c8f-85471b4288b7\") " Nov 01 21:39:33.408052 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.407730 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35afbdf8-b193-42b5-9c8f-85471b4288b7-credential-socket" pod "35afbdf8-b193-42b5-9c8f-85471b4288b7" (UID: "35afbdf8-b193-42b5-9c8f-85471b4288b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:39:33.408052 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.407947 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec0b1a26-5b6c-405c-a042-7141708b6b22-istio-data" pod "ec0b1a26-5b6c-405c-a042-7141708b6b22" (UID: "ec0b1a26-5b6c-405c-a042-7141708b6b22"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:39:33.408974 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.408285 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35afbdf8-b193-42b5-9c8f-85471b4288b7-istiod-ca-cert" pod "35afbdf8-b193-42b5-9c8f-85471b4288b7" (UID: "35afbdf8-b193-42b5-9c8f-85471b4288b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:39:33.408974 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.408300 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec0b1a26-5b6c-405c-a042-7141708b6b22-istio-token\" (UniqueName: \"kubernetes.io/projected/ec0b1a26-5b6c-405c-a042-7141708b6b22-istio-token\") pod \"ec0b1a26-5b6c-405c-a042-7141708b6b22\" (UID: \"ec0b1a26-5b6c-405c-a042-7141708b6b22\") " Nov 01 21:39:33.408974 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.408456 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec0b1a26-5b6c-405c-a042-7141708b6b22-workload-certs" pod "ec0b1a26-5b6c-405c-a042-7141708b6b22" (UID: "ec0b1a26-5b6c-405c-a042-7141708b6b22"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:39:33.408974 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.408491 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec0b1a26-5b6c-405c-a042-7141708b6b22-istiod-ca-cert" pod "ec0b1a26-5b6c-405c-a042-7141708b6b22" (UID: "ec0b1a26-5b6c-405c-a042-7141708b6b22"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:39:33.408974 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.408744 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35afbdf8-b193-42b5-9c8f-85471b4288b7-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:39:33.408974 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.408765 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec0b1a26-5b6c-405c-a042-7141708b6b22-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:39:33.408974 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.408782 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35afbdf8-b193-42b5-9c8f-85471b4288b7-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:39:33.408974 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.408800 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec0b1a26-5b6c-405c-a042-7141708b6b22-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:39:33.408974 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.408814 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35afbdf8-b193-42b5-9c8f-85471b4288b7-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:39:33.408974 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.408829 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec0b1a26-5b6c-405c-a042-7141708b6b22-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:39:33.408974 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.408843 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35afbdf8-b193-42b5-9c8f-85471b4288b7-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:39:33.408974 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.408857 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec0b1a26-5b6c-405c-a042-7141708b6b22-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:39:33.408974 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.408871 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35afbdf8-b193-42b5-9c8f-85471b4288b7-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:39:33.408974 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.408884 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec0b1a26-5b6c-405c-a042-7141708b6b22-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:39:33.409884 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.409856 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec0b1a26-5b6c-405c-a042-7141708b6b22-kube-api-access-hbsm6" pod "ec0b1a26-5b6c-405c-a042-7141708b6b22" (UID: "ec0b1a26-5b6c-405c-a042-7141708b6b22"). InnerVolumeSpecName "kube-api-access-hbsm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:39:33.410299 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.410278 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35afbdf8-b193-42b5-9c8f-85471b4288b7-istio-envoy" pod "35afbdf8-b193-42b5-9c8f-85471b4288b7" (UID: "35afbdf8-b193-42b5-9c8f-85471b4288b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:39:33.410905 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.410879 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35afbdf8-b193-42b5-9c8f-85471b4288b7-istio-podinfo" pod "35afbdf8-b193-42b5-9c8f-85471b4288b7" (UID: "35afbdf8-b193-42b5-9c8f-85471b4288b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:39:33.410905 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.410883 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec0b1a26-5b6c-405c-a042-7141708b6b22-istio-podinfo" pod "ec0b1a26-5b6c-405c-a042-7141708b6b22" (UID: "ec0b1a26-5b6c-405c-a042-7141708b6b22"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:39:33.411163 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.410938 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35afbdf8-b193-42b5-9c8f-85471b4288b7-istio-token" pod "35afbdf8-b193-42b5-9c8f-85471b4288b7" (UID: "35afbdf8-b193-42b5-9c8f-85471b4288b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:39:33.411163 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.411030 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35afbdf8-b193-42b5-9c8f-85471b4288b7-kube-api-access-l5mg4" pod "35afbdf8-b193-42b5-9c8f-85471b4288b7" (UID: "35afbdf8-b193-42b5-9c8f-85471b4288b7"). InnerVolumeSpecName "kube-api-access-l5mg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:39:33.411610 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.411590 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec0b1a26-5b6c-405c-a042-7141708b6b22-istio-token" pod "ec0b1a26-5b6c-405c-a042-7141708b6b22" (UID: "ec0b1a26-5b6c-405c-a042-7141708b6b22"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:39:33.411777 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.411754 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec0b1a26-5b6c-405c-a042-7141708b6b22-istio-envoy" pod "ec0b1a26-5b6c-405c-a042-7141708b6b22" (UID: "ec0b1a26-5b6c-405c-a042-7141708b6b22"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:39:33.510205 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.509834 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3754b0a-165a-4693-b99c-c7e112c594c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3754b0a-165a-4693-b99c-c7e112c594c1-workload-socket\") pod \"f3754b0a-165a-4693-b99c-c7e112c594c1\" (UID: \"f3754b0a-165a-4693-b99c-c7e112c594c1\") " Nov 01 21:39:33.510205 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.509894 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-workload-socket\") pod \"8e6a4f7c-75e1-4506-afb5-4ab00107a7f0\" (UID: \"8e6a4f7c-75e1-4506-afb5-4ab00107a7f0\") " Nov 01 21:39:33.510205 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.509939 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f3754b0a-165a-4693-b99c-c7e112c594c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3754b0a-165a-4693-b99c-c7e112c594c1-istio-podinfo\") pod \"f3754b0a-165a-4693-b99c-c7e112c594c1\" (UID: \"f3754b0a-165a-4693-b99c-c7e112c594c1\") " Nov 01 21:39:33.510205 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.509990 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3754b0a-165a-4693-b99c-c7e112c594c1-istio-token\" (UniqueName: \"kubernetes.io/projected/f3754b0a-165a-4693-b99c-c7e112c594c1-istio-token\") pod \"f3754b0a-165a-4693-b99c-c7e112c594c1\" (UID: \"f3754b0a-165a-4693-b99c-c7e112c594c1\") " Nov 01 21:39:33.510205 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.510062 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3754b0a-165a-4693-b99c-c7e112c594c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3754b0a-165a-4693-b99c-c7e112c594c1-istio-data\") pod \"f3754b0a-165a-4693-b99c-c7e112c594c1\" (UID: \"f3754b0a-165a-4693-b99c-c7e112c594c1\") " Nov 01 21:39:33.510205 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.510132 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3754b0a-165a-4693-b99c-c7e112c594c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3754b0a-165a-4693-b99c-c7e112c594c1-istiod-ca-cert\") pod \"f3754b0a-165a-4693-b99c-c7e112c594c1\" (UID: \"f3754b0a-165a-4693-b99c-c7e112c594c1\") " Nov 01 21:39:33.512565 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.510269 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3754b0a-165a-4693-b99c-c7e112c594c1-workload-socket" pod "f3754b0a-165a-4693-b99c-c7e112c594c1" (UID: "f3754b0a-165a-4693-b99c-c7e112c594c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:39:33.512565 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.510284 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-istio-envoy\") pod \"8e6a4f7c-75e1-4506-afb5-4ab00107a7f0\" (UID: \"8e6a4f7c-75e1-4506-afb5-4ab00107a7f0\") " Nov 01 21:39:33.512565 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.510324 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-istio-data\") pod \"8e6a4f7c-75e1-4506-afb5-4ab00107a7f0\" (UID: \"8e6a4f7c-75e1-4506-afb5-4ab00107a7f0\") " Nov 01 21:39:33.512565 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.510361 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3754b0a-165a-4693-b99c-c7e112c594c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3754b0a-165a-4693-b99c-c7e112c594c1-workload-certs\") pod \"f3754b0a-165a-4693-b99c-c7e112c594c1\" (UID: \"f3754b0a-165a-4693-b99c-c7e112c594c1\") " Nov 01 21:39:33.512565 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.510392 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3754b0a-165a-4693-b99c-c7e112c594c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3754b0a-165a-4693-b99c-c7e112c594c1-credential-socket\") pod \"f3754b0a-165a-4693-b99c-c7e112c594c1\" (UID: \"f3754b0a-165a-4693-b99c-c7e112c594c1\") " Nov 01 21:39:33.512565 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.510419 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3754b0a-165a-4693-b99c-c7e112c594c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3754b0a-165a-4693-b99c-c7e112c594c1-istio-envoy\") pod \"f3754b0a-165a-4693-b99c-c7e112c594c1\" (UID: \"f3754b0a-165a-4693-b99c-c7e112c594c1\") " Nov 01 21:39:33.512565 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.510458 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-istio-token\" (UniqueName: \"kubernetes.io/projected/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-istio-token\") pod \"8e6a4f7c-75e1-4506-afb5-4ab00107a7f0\" (UID: \"8e6a4f7c-75e1-4506-afb5-4ab00107a7f0\") " Nov 01 21:39:33.512565 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.510480 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3754b0a-165a-4693-b99c-c7e112c594c1-istio-data" pod "f3754b0a-165a-4693-b99c-c7e112c594c1" (UID: "f3754b0a-165a-4693-b99c-c7e112c594c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:39:33.512565 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.510490 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3754b0a-165a-4693-b99c-c7e112c594c1-kube-api-access-trqvc\" (UniqueName: \"kubernetes.io/projected/f3754b0a-165a-4693-b99c-c7e112c594c1-kube-api-access-trqvc\") pod \"f3754b0a-165a-4693-b99c-c7e112c594c1\" (UID: \"f3754b0a-165a-4693-b99c-c7e112c594c1\") " Nov 01 21:39:33.512565 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.510594 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-istio-podinfo\") pod \"8e6a4f7c-75e1-4506-afb5-4ab00107a7f0\" (UID: \"8e6a4f7c-75e1-4506-afb5-4ab00107a7f0\") " Nov 01 21:39:33.512565 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.510614 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-istio-data" pod "8e6a4f7c-75e1-4506-afb5-4ab00107a7f0" (UID: "8e6a4f7c-75e1-4506-afb5-4ab00107a7f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:39:33.512565 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.510635 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-credential-socket\") pod \"8e6a4f7c-75e1-4506-afb5-4ab00107a7f0\" (UID: \"8e6a4f7c-75e1-4506-afb5-4ab00107a7f0\") " Nov 01 21:39:33.512565 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.510686 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-kube-api-access-rw4zr\" (UniqueName: \"kubernetes.io/projected/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-kube-api-access-rw4zr\") pod \"8e6a4f7c-75e1-4506-afb5-4ab00107a7f0\" (UID: \"8e6a4f7c-75e1-4506-afb5-4ab00107a7f0\") " Nov 01 21:39:33.512565 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.510737 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-istiod-ca-cert\") pod \"8e6a4f7c-75e1-4506-afb5-4ab00107a7f0\" (UID: \"8e6a4f7c-75e1-4506-afb5-4ab00107a7f0\") " Nov 01 21:39:33.512565 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.510797 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-workload-certs\") pod \"8e6a4f7c-75e1-4506-afb5-4ab00107a7f0\" (UID: \"8e6a4f7c-75e1-4506-afb5-4ab00107a7f0\") " Nov 01 21:39:33.512565 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.511156 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3754b0a-165a-4693-b99c-c7e112c594c1-istiod-ca-cert" pod "f3754b0a-165a-4693-b99c-c7e112c594c1" (UID: "f3754b0a-165a-4693-b99c-c7e112c594c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:39:33.512565 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.511272 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5mg4\" (UniqueName: \"kubernetes.io/projected/35afbdf8-b193-42b5-9c8f-85471b4288b7-kube-api-access-l5mg4\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:39:33.512565 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.511294 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3754b0a-165a-4693-b99c-c7e112c594c1-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:39:33.512565 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.511310 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec0b1a26-5b6c-405c-a042-7141708b6b22-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:39:33.512565 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.511324 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35afbdf8-b193-42b5-9c8f-85471b4288b7-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:39:33.512565 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.511412 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-workload-certs" pod "8e6a4f7c-75e1-4506-afb5-4ab00107a7f0" (UID: "8e6a4f7c-75e1-4506-afb5-4ab00107a7f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:39:33.512565 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.511483 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3754b0a-165a-4693-b99c-c7e112c594c1-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:39:33.512565 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.511500 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:39:33.512565 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.511518 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec0b1a26-5b6c-405c-a042-7141708b6b22-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:39:33.512565 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.511532 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35afbdf8-b193-42b5-9c8f-85471b4288b7-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:39:33.512565 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.511547 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec0b1a26-5b6c-405c-a042-7141708b6b22-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:39:33.512565 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.511561 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3754b0a-165a-4693-b99c-c7e112c594c1-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:39:33.512565 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.511577 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35afbdf8-b193-42b5-9c8f-85471b4288b7-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:39:33.512565 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.511592 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbsm6\" (UniqueName: \"kubernetes.io/projected/ec0b1a26-5b6c-405c-a042-7141708b6b22-kube-api-access-hbsm6\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:39:33.512565 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.511762 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3754b0a-165a-4693-b99c-c7e112c594c1-credential-socket" pod "f3754b0a-165a-4693-b99c-c7e112c594c1" (UID: "f3754b0a-165a-4693-b99c-c7e112c594c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:39:33.512565 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.511901 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-workload-socket" pod "8e6a4f7c-75e1-4506-afb5-4ab00107a7f0" (UID: "8e6a4f7c-75e1-4506-afb5-4ab00107a7f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:39:33.514357 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.513081 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3754b0a-165a-4693-b99c-c7e112c594c1-workload-certs" pod "f3754b0a-165a-4693-b99c-c7e112c594c1" (UID: "f3754b0a-165a-4693-b99c-c7e112c594c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:39:33.514357 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.513751 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-istiod-ca-cert" pod "8e6a4f7c-75e1-4506-afb5-4ab00107a7f0" (UID: "8e6a4f7c-75e1-4506-afb5-4ab00107a7f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:39:33.514357 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.513801 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-credential-socket" pod "8e6a4f7c-75e1-4506-afb5-4ab00107a7f0" (UID: "8e6a4f7c-75e1-4506-afb5-4ab00107a7f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:39:33.515198 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.515131 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-istio-podinfo" pod "8e6a4f7c-75e1-4506-afb5-4ab00107a7f0" (UID: "8e6a4f7c-75e1-4506-afb5-4ab00107a7f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:39:33.515352 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.515263 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3754b0a-165a-4693-b99c-c7e112c594c1-kube-api-access-trqvc" pod "f3754b0a-165a-4693-b99c-c7e112c594c1" (UID: "f3754b0a-165a-4693-b99c-c7e112c594c1"). InnerVolumeSpecName "kube-api-access-trqvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:39:33.516851 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.516828 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-kube-api-access-rw4zr" pod "8e6a4f7c-75e1-4506-afb5-4ab00107a7f0" (UID: "8e6a4f7c-75e1-4506-afb5-4ab00107a7f0"). InnerVolumeSpecName "kube-api-access-rw4zr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:39:33.516851 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.516830 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3754b0a-165a-4693-b99c-c7e112c594c1-istio-podinfo" pod "f3754b0a-165a-4693-b99c-c7e112c594c1" (UID: "f3754b0a-165a-4693-b99c-c7e112c594c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:39:33.517038 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.516857 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-istio-token" pod "8e6a4f7c-75e1-4506-afb5-4ab00107a7f0" (UID: "8e6a4f7c-75e1-4506-afb5-4ab00107a7f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:39:33.517038 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.516877 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3754b0a-165a-4693-b99c-c7e112c594c1-istio-envoy" pod "f3754b0a-165a-4693-b99c-c7e112c594c1" (UID: "f3754b0a-165a-4693-b99c-c7e112c594c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:39:33.517893 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.517869 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3754b0a-165a-4693-b99c-c7e112c594c1-istio-token" pod "f3754b0a-165a-4693-b99c-c7e112c594c1" (UID: "f3754b0a-165a-4693-b99c-c7e112c594c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:39:33.518080 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.517957 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-istio-envoy" pod "8e6a4f7c-75e1-4506-afb5-4ab00107a7f0" (UID: "8e6a4f7c-75e1-4506-afb5-4ab00107a7f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:39:33.586443 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.586404 4484 generic.go:356] "Generic (PLEG): container finished" podID="f3754b0a-165a-4693-b99c-c7e112c594c1" containerID="e575bd0a74419db47776fc2e709ea8b405756332bdc7354daefd9317312964ea" exitCode=0 Nov 01 21:39:33.586646 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.586448 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qm8qv" Nov 01 21:39:33.586646 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.586481 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qm8qv" event={"ID":"f3754b0a-165a-4693-b99c-c7e112c594c1","Type":"ContainerDied","Data":"e575bd0a74419db47776fc2e709ea8b405756332bdc7354daefd9317312964ea"} Nov 01 21:39:33.586646 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.586534 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qm8qv" event={"ID":"f3754b0a-165a-4693-b99c-c7e112c594c1","Type":"ContainerDied","Data":"5b36dae681fe24a879a98bc55a8848c026d772ad8a2cd1208c4d33e3259616bf"} Nov 01 21:39:33.586646 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.586556 4484 scope.go:122] "RemoveContainer" containerID="e575bd0a74419db47776fc2e709ea8b405756332bdc7354daefd9317312964ea" Nov 01 21:39:33.589626 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.589600 4484 generic.go:356] "Generic (PLEG): container finished" podID="8e6a4f7c-75e1-4506-afb5-4ab00107a7f0" containerID="c9b777e4fde78925d8f9a532335030b0dfbaf6d198729fbb6efd6f4736b40122" exitCode=0 Nov 01 21:39:33.589774 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.589638 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6zwld" Nov 01 21:39:33.589774 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.589682 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6zwld" event={"ID":"8e6a4f7c-75e1-4506-afb5-4ab00107a7f0","Type":"ContainerDied","Data":"c9b777e4fde78925d8f9a532335030b0dfbaf6d198729fbb6efd6f4736b40122"} Nov 01 21:39:33.589774 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.589728 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6zwld" event={"ID":"8e6a4f7c-75e1-4506-afb5-4ab00107a7f0","Type":"ContainerDied","Data":"135a8b639689b8e947b71717cfb656d1383accc585faa706a4392f455f049ac6"} Nov 01 21:39:33.591382 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.591359 4484 generic.go:356] "Generic (PLEG): container finished" podID="ec0b1a26-5b6c-405c-a042-7141708b6b22" containerID="192367791b1cd7b5b98efbf801cfc63ec5ec9742a3f76f1c3dc7753f305d3db9" exitCode=0 Nov 01 21:39:33.591520 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.591424 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-hdvw9" Nov 01 21:39:33.591520 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.591451 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-hdvw9" event={"ID":"ec0b1a26-5b6c-405c-a042-7141708b6b22","Type":"ContainerDied","Data":"192367791b1cd7b5b98efbf801cfc63ec5ec9742a3f76f1c3dc7753f305d3db9"} Nov 01 21:39:33.591520 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.591496 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-hdvw9" event={"ID":"ec0b1a26-5b6c-405c-a042-7141708b6b22","Type":"ContainerDied","Data":"b13dcbb27389f75e2f8b657eaecb2cf1c118355ff5227c694712bc50efe2fa3f"} Nov 01 21:39:33.599444 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.599414 4484 generic.go:356] "Generic (PLEG): container finished" podID="35afbdf8-b193-42b5-9c8f-85471b4288b7" containerID="83d61343b766754a0443bda62ee8204519448aa5b8ee4a423219845b01ae1cd9" exitCode=0 Nov 01 21:39:33.599587 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.599470 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2n5h6" Nov 01 21:39:33.599587 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.599508 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2n5h6" event={"ID":"35afbdf8-b193-42b5-9c8f-85471b4288b7","Type":"ContainerDied","Data":"83d61343b766754a0443bda62ee8204519448aa5b8ee4a423219845b01ae1cd9"} Nov 01 21:39:33.599587 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.599555 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2n5h6" event={"ID":"35afbdf8-b193-42b5-9c8f-85471b4288b7","Type":"ContainerDied","Data":"4366df2b04cd091dbd5fbe8b51560659deb6ae9dd4c8968e06cfdc9d81a50609"} Nov 01 21:39:33.599773 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.599735 4484 scope.go:122] "RemoveContainer" containerID="e575bd0a74419db47776fc2e709ea8b405756332bdc7354daefd9317312964ea" Nov 01 21:39:33.600097 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:39:33.600074 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e575bd0a74419db47776fc2e709ea8b405756332bdc7354daefd9317312964ea\": container with ID starting with e575bd0a74419db47776fc2e709ea8b405756332bdc7354daefd9317312964ea not found: ID does not exist" containerID="e575bd0a74419db47776fc2e709ea8b405756332bdc7354daefd9317312964ea" Nov 01 21:39:33.600209 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.600111 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e575bd0a74419db47776fc2e709ea8b405756332bdc7354daefd9317312964ea"} err="failed to get container status \"e575bd0a74419db47776fc2e709ea8b405756332bdc7354daefd9317312964ea\": rpc error: code = NotFound desc = could not find container \"e575bd0a74419db47776fc2e709ea8b405756332bdc7354daefd9317312964ea\": container with ID starting with e575bd0a74419db47776fc2e709ea8b405756332bdc7354daefd9317312964ea not found: ID does not exist" Nov 01 21:39:33.600209 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.600133 4484 scope.go:122] "RemoveContainer" containerID="c9b777e4fde78925d8f9a532335030b0dfbaf6d198729fbb6efd6f4736b40122" Nov 01 21:39:33.609388 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.609369 4484 scope.go:122] "RemoveContainer" containerID="c9b777e4fde78925d8f9a532335030b0dfbaf6d198729fbb6efd6f4736b40122" Nov 01 21:39:33.609637 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:39:33.609618 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9b777e4fde78925d8f9a532335030b0dfbaf6d198729fbb6efd6f4736b40122\": container with ID starting with c9b777e4fde78925d8f9a532335030b0dfbaf6d198729fbb6efd6f4736b40122 not found: ID does not exist" containerID="c9b777e4fde78925d8f9a532335030b0dfbaf6d198729fbb6efd6f4736b40122" Nov 01 21:39:33.609719 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.609648 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9b777e4fde78925d8f9a532335030b0dfbaf6d198729fbb6efd6f4736b40122"} err="failed to get container status \"c9b777e4fde78925d8f9a532335030b0dfbaf6d198729fbb6efd6f4736b40122\": rpc error: code = NotFound desc = could not find container \"c9b777e4fde78925d8f9a532335030b0dfbaf6d198729fbb6efd6f4736b40122\": container with ID starting with c9b777e4fde78925d8f9a532335030b0dfbaf6d198729fbb6efd6f4736b40122 not found: ID does not exist" Nov 01 21:39:33.609719 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.609671 4484 scope.go:122] "RemoveContainer" containerID="192367791b1cd7b5b98efbf801cfc63ec5ec9742a3f76f1c3dc7753f305d3db9" Nov 01 21:39:33.613041 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.613020 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3754b0a-165a-4693-b99c-c7e112c594c1-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:39:33.613041 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.613042 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:39:33.613166 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.613052 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3754b0a-165a-4693-b99c-c7e112c594c1-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:39:33.613166 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.613062 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3754b0a-165a-4693-b99c-c7e112c594c1-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:39:33.613166 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.613071 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3754b0a-165a-4693-b99c-c7e112c594c1-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:39:33.613166 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.613079 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:39:33.613166 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.613088 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trqvc\" (UniqueName: \"kubernetes.io/projected/f3754b0a-165a-4693-b99c-c7e112c594c1-kube-api-access-trqvc\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:39:33.613166 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.613097 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:39:33.613166 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.613105 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:39:33.613166 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.613115 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw4zr\" (UniqueName: \"kubernetes.io/projected/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-kube-api-access-rw4zr\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:39:33.613166 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.613124 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:39:33.613166 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.613133 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:39:33.613166 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.613142 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:39:33.613166 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.613149 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3754b0a-165a-4693-b99c-c7e112c594c1-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:39:33.620953 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.620927 4484 scope.go:122] "RemoveContainer" containerID="192367791b1cd7b5b98efbf801cfc63ec5ec9742a3f76f1c3dc7753f305d3db9" Nov 01 21:39:33.621352 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:39:33.621331 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"192367791b1cd7b5b98efbf801cfc63ec5ec9742a3f76f1c3dc7753f305d3db9\": container with ID starting with 192367791b1cd7b5b98efbf801cfc63ec5ec9742a3f76f1c3dc7753f305d3db9 not found: ID does not exist" containerID="192367791b1cd7b5b98efbf801cfc63ec5ec9742a3f76f1c3dc7753f305d3db9" Nov 01 21:39:33.621426 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.621359 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"192367791b1cd7b5b98efbf801cfc63ec5ec9742a3f76f1c3dc7753f305d3db9"} err="failed to get container status \"192367791b1cd7b5b98efbf801cfc63ec5ec9742a3f76f1c3dc7753f305d3db9\": rpc error: code = NotFound desc = could not find container \"192367791b1cd7b5b98efbf801cfc63ec5ec9742a3f76f1c3dc7753f305d3db9\": container with ID starting with 192367791b1cd7b5b98efbf801cfc63ec5ec9742a3f76f1c3dc7753f305d3db9 not found: ID does not exist" Nov 01 21:39:33.621426 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.621378 4484 scope.go:122] "RemoveContainer" containerID="83d61343b766754a0443bda62ee8204519448aa5b8ee4a423219845b01ae1cd9" Nov 01 21:39:33.629591 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.629574 4484 scope.go:122] "RemoveContainer" containerID="83d61343b766754a0443bda62ee8204519448aa5b8ee4a423219845b01ae1cd9" Nov 01 21:39:33.629958 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:39:33.629937 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83d61343b766754a0443bda62ee8204519448aa5b8ee4a423219845b01ae1cd9\": container with ID starting with 83d61343b766754a0443bda62ee8204519448aa5b8ee4a423219845b01ae1cd9 not found: ID does not exist" containerID="83d61343b766754a0443bda62ee8204519448aa5b8ee4a423219845b01ae1cd9" Nov 01 21:39:33.630064 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.629967 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83d61343b766754a0443bda62ee8204519448aa5b8ee4a423219845b01ae1cd9"} err="failed to get container status \"83d61343b766754a0443bda62ee8204519448aa5b8ee4a423219845b01ae1cd9\": rpc error: code = NotFound desc = could not find container \"83d61343b766754a0443bda62ee8204519448aa5b8ee4a423219845b01ae1cd9\": container with ID starting with 83d61343b766754a0443bda62ee8204519448aa5b8ee4a423219845b01ae1cd9 not found: ID does not exist" Nov 01 21:39:33.915448 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.915364 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6zwld"] Nov 01 21:39:33.919046 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:33.919018 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6zwld"] Nov 01 21:39:34.028912 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:34.028881 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2n5h6"] Nov 01 21:39:34.032876 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:34.032844 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2n5h6"] Nov 01 21:39:34.142230 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:34.142198 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-hdvw9"] Nov 01 21:39:34.145657 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:34.145632 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-hdvw9"] Nov 01 21:39:34.255180 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:34.255150 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qm8qv"] Nov 01 21:39:34.259023 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:34.258987 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qm8qv"] Nov 01 21:39:35.370617 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:35.370575 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35afbdf8-b193-42b5-9c8f-85471b4288b7" path="/var/lib/kubelet/pods/35afbdf8-b193-42b5-9c8f-85471b4288b7/volumes" Nov 01 21:39:35.371117 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:35.371101 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e6a4f7c-75e1-4506-afb5-4ab00107a7f0" path="/var/lib/kubelet/pods/8e6a4f7c-75e1-4506-afb5-4ab00107a7f0/volumes" Nov 01 21:39:35.371542 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:35.371528 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec0b1a26-5b6c-405c-a042-7141708b6b22" path="/var/lib/kubelet/pods/ec0b1a26-5b6c-405c-a042-7141708b6b22/volumes" Nov 01 21:39:35.372289 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:35.372274 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3754b0a-165a-4693-b99c-c7e112c594c1" path="/var/lib/kubelet/pods/f3754b0a-165a-4693-b99c-c7e112c594c1/volumes" Nov 01 21:39:35.930569 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:35.930536 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:39:40.732916 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:40.732877 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:39:45.533072 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:45.533034 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:39:45.766536 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:39:45.766502 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:39:50.338529 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:50.338491 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:39:55.139855 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:55.139820 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:39:56.609047 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:56.608988 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:39:59.933786 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:39:59.933744 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:40:03.115271 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:03.115236 4484 scope.go:122] "RemoveContainer" containerID="6431f6a39f20d540eefda0beb35853cf8cbdc958f1ce880ce201ece67e31341e" Nov 01 21:40:03.123452 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:03.123432 4484 scope.go:122] "RemoveContainer" containerID="4ea1e931e7da2d0e64b6d0556453011e8432361d800cfd34b8cb2e618413ebdb" Nov 01 21:40:04.616153 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:04.616117 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-e23c3b-sfjxk"] Nov 01 21:40:04.616768 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:04.616511 4484 cpu_manager.go:414] "containerMap: removing container" podUID="35afbdf8-b193-42b5-9c8f-85471b4288b7" containerName="istio-proxy" Nov 01 21:40:04.616768 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:04.616524 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35afbdf8-b193-42b5-9c8f-85471b4288b7" containerName="istio-proxy" Nov 01 21:40:04.616768 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:04.616537 4484 cpu_manager.go:414] "containerMap: removing container" podUID="f3754b0a-165a-4693-b99c-c7e112c594c1" containerName="istio-proxy" Nov 01 21:40:04.616768 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:04.616543 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3754b0a-165a-4693-b99c-c7e112c594c1" containerName="istio-proxy" Nov 01 21:40:04.616768 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:04.616563 4484 cpu_manager.go:414] "containerMap: removing container" podUID="ec0b1a26-5b6c-405c-a042-7141708b6b22" containerName="istio-proxy" Nov 01 21:40:04.616768 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:04.616568 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec0b1a26-5b6c-405c-a042-7141708b6b22" containerName="istio-proxy" Nov 01 21:40:04.616768 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:04.616593 4484 cpu_manager.go:414] "containerMap: removing container" podUID="8e6a4f7c-75e1-4506-afb5-4ab00107a7f0" containerName="istio-proxy" Nov 01 21:40:04.616768 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:04.616598 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e6a4f7c-75e1-4506-afb5-4ab00107a7f0" containerName="istio-proxy" Nov 01 21:40:04.616768 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:04.616648 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec0b1a26-5b6c-405c-a042-7141708b6b22" containerName="istio-proxy" Nov 01 21:40:04.616768 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:04.616661 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e6a4f7c-75e1-4506-afb5-4ab00107a7f0" containerName="istio-proxy" Nov 01 21:40:04.616768 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:04.616667 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="35afbdf8-b193-42b5-9c8f-85471b4288b7" containerName="istio-proxy" Nov 01 21:40:04.616768 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:04.616672 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3754b0a-165a-4693-b99c-c7e112c594c1" containerName="istio-proxy" Nov 01 21:40:04.616768 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:04.616679 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="38ce6dc7-e34a-48da-a283-f140baa31047" containerName="istio-proxy" Nov 01 21:40:04.621457 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:04.621439 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-e23c3b-sfjxk" Nov 01 21:40:04.672139 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:04.672101 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:40:04.674532 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:04.674503 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:40:04.674687 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:04.674609 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:40:04.704259 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:04.704224 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8254c2d2-5590-46c0-bfa8-c555247a07bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-e23c3b-sfjxk\" (UID: \"8254c2d2-5590-46c0-bfa8-c555247a07bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-e23c3b-sfjxk" Nov 01 21:40:04.704451 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:04.704299 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8254c2d2-5590-46c0-bfa8-c555247a07bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-e23c3b-sfjxk\" (UID: \"8254c2d2-5590-46c0-bfa8-c555247a07bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-e23c3b-sfjxk" Nov 01 21:40:04.704451 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:04.704358 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmjk7\" (UniqueName: \"kubernetes.io/projected/8254c2d2-5590-46c0-bfa8-c555247a07bd-kube-api-access-dmjk7\") pod \"maas-api-revoke-keys-e2e-authcfg-http-e23c3b-sfjxk\" (UID: \"8254c2d2-5590-46c0-bfa8-c555247a07bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-e23c3b-sfjxk" Nov 01 21:40:04.725569 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:04.725536 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:40:04.725753 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:04.725594 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzdwq\"" Nov 01 21:40:04.728384 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:04.728359 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-e23c3b-sfjxk"] Nov 01 21:40:04.805846 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:04.805806 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8254c2d2-5590-46c0-bfa8-c555247a07bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-e23c3b-sfjxk\" (UID: \"8254c2d2-5590-46c0-bfa8-c555247a07bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-e23c3b-sfjxk" Nov 01 21:40:04.806103 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:04.805869 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8254c2d2-5590-46c0-bfa8-c555247a07bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-e23c3b-sfjxk\" (UID: \"8254c2d2-5590-46c0-bfa8-c555247a07bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-e23c3b-sfjxk" Nov 01 21:40:04.806103 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:04.805899 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmjk7\" (UniqueName: \"kubernetes.io/projected/8254c2d2-5590-46c0-bfa8-c555247a07bd-kube-api-access-dmjk7\") pod \"maas-api-revoke-keys-e2e-authcfg-http-e23c3b-sfjxk\" (UID: \"8254c2d2-5590-46c0-bfa8-c555247a07bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-e23c3b-sfjxk" Nov 01 21:40:04.806637 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:04.806610 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8254c2d2-5590-46c0-bfa8-c555247a07bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-e23c3b-sfjxk\" (UID: \"8254c2d2-5590-46c0-bfa8-c555247a07bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-e23c3b-sfjxk" Nov 01 21:40:04.808187 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:04.808168 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8254c2d2-5590-46c0-bfa8-c555247a07bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-e23c3b-sfjxk\" (UID: \"8254c2d2-5590-46c0-bfa8-c555247a07bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-e23c3b-sfjxk" Nov 01 21:40:04.863377 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:04.863331 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmjk7\" (UniqueName: \"kubernetes.io/projected/8254c2d2-5590-46c0-bfa8-c555247a07bd-kube-api-access-dmjk7\") pod \"maas-api-revoke-keys-e2e-authcfg-http-e23c3b-sfjxk\" (UID: \"8254c2d2-5590-46c0-bfa8-c555247a07bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-e23c3b-sfjxk" Nov 01 21:40:04.933773 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:04.933673 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-e23c3b-sfjxk" Nov 01 21:40:05.305262 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:05.305231 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-e23c3b-sfjxk"] Nov 01 21:40:05.307957 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:40:05.307930 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8254c2d2_5590_46c0_bfa8_c555247a07bd.slice/crio-d67ea95b16c3e9df59e844089e7a906c367089e4155b8263101c4bcba5dc10db WatchSource:0}: Error finding container d67ea95b16c3e9df59e844089e7a906c367089e4155b8263101c4bcba5dc10db: Status 404 returned error can't find the container with id d67ea95b16c3e9df59e844089e7a906c367089e4155b8263101c4bcba5dc10db Nov 01 21:40:05.758333 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:05.758289 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-e23c3b-sfjxk" event={"ID":"8254c2d2-5590-46c0-bfa8-c555247a07bd","Type":"ContainerStarted","Data":"83394f0dcbb36c3f68baf2dd3fa62422a62f073036f89847f9289b8b2c63e838"} Nov 01 21:40:05.758781 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:05.758375 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-e23c3b-sfjxk" event={"ID":"8254c2d2-5590-46c0-bfa8-c555247a07bd","Type":"ContainerStarted","Data":"d67ea95b16c3e9df59e844089e7a906c367089e4155b8263101c4bcba5dc10db"} Nov 01 21:40:05.874100 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:05.873984 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-e23c3b-sfjxk" podStartSLOduration=1.873972315 podStartE2EDuration="1.873972315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:40:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:40:05.873162133 +0000 UTC m=+4156.650802267" watchObservedRunningTime="2026-11-01 21:40:05.873972315 +0000 UTC m=+4156.651612447" Nov 01 21:40:06.764933 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:06.764893 4484 generic.go:356] "Generic (PLEG): container finished" podID="8254c2d2-5590-46c0-bfa8-c555247a07bd" containerID="83394f0dcbb36c3f68baf2dd3fa62422a62f073036f89847f9289b8b2c63e838" exitCode=0 Nov 01 21:40:06.765464 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:06.764953 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-e23c3b-sfjxk" event={"ID":"8254c2d2-5590-46c0-bfa8-c555247a07bd","Type":"ContainerDied","Data":"83394f0dcbb36c3f68baf2dd3fa62422a62f073036f89847f9289b8b2c63e838"} Nov 01 21:40:07.906042 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:07.906015 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-e23c3b-sfjxk" Nov 01 21:40:08.036954 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:08.036873 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8254c2d2-5590-46c0-bfa8-c555247a07bd-kube-api-access-dmjk7\" (UniqueName: \"kubernetes.io/projected/8254c2d2-5590-46c0-bfa8-c555247a07bd-kube-api-access-dmjk7\") pod \"8254c2d2-5590-46c0-bfa8-c555247a07bd\" (UID: \"8254c2d2-5590-46c0-bfa8-c555247a07bd\") " Nov 01 21:40:08.037147 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:08.036986 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8254c2d2-5590-46c0-bfa8-c555247a07bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8254c2d2-5590-46c0-bfa8-c555247a07bd-maas-api-openshift-ca\") pod \"8254c2d2-5590-46c0-bfa8-c555247a07bd\" (UID: \"8254c2d2-5590-46c0-bfa8-c555247a07bd\") " Nov 01 21:40:08.037147 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:08.037057 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8254c2d2-5590-46c0-bfa8-c555247a07bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8254c2d2-5590-46c0-bfa8-c555247a07bd-maas-api-xks-ca\") pod \"8254c2d2-5590-46c0-bfa8-c555247a07bd\" (UID: \"8254c2d2-5590-46c0-bfa8-c555247a07bd\") " Nov 01 21:40:08.037340 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:08.037313 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8254c2d2-5590-46c0-bfa8-c555247a07bd-maas-api-openshift-ca" pod "8254c2d2-5590-46c0-bfa8-c555247a07bd" (UID: "8254c2d2-5590-46c0-bfa8-c555247a07bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:40:08.039198 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:08.039171 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8254c2d2-5590-46c0-bfa8-c555247a07bd-maas-api-xks-ca" pod "8254c2d2-5590-46c0-bfa8-c555247a07bd" (UID: "8254c2d2-5590-46c0-bfa8-c555247a07bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:40:08.039317 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:08.039204 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8254c2d2-5590-46c0-bfa8-c555247a07bd-kube-api-access-dmjk7" pod "8254c2d2-5590-46c0-bfa8-c555247a07bd" (UID: "8254c2d2-5590-46c0-bfa8-c555247a07bd"). InnerVolumeSpecName "kube-api-access-dmjk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:40:08.138048 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:08.137981 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8254c2d2-5590-46c0-bfa8-c555247a07bd-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:40:08.138048 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:08.138039 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8254c2d2-5590-46c0-bfa8-c555247a07bd-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:40:08.138048 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:08.138049 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmjk7\" (UniqueName: \"kubernetes.io/projected/8254c2d2-5590-46c0-bfa8-c555247a07bd-kube-api-access-dmjk7\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:40:08.776131 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:08.776095 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-e23c3b-sfjxk" event={"ID":"8254c2d2-5590-46c0-bfa8-c555247a07bd","Type":"ContainerDied","Data":"d67ea95b16c3e9df59e844089e7a906c367089e4155b8263101c4bcba5dc10db"} Nov 01 21:40:08.776131 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:08.776121 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-e23c3b-sfjxk" Nov 01 21:40:08.776380 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:08.776143 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d67ea95b16c3e9df59e844089e7a906c367089e4155b8263101c4bcba5dc10db" Nov 01 21:40:09.872959 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:09.872927 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:40:14.629889 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:14.629852 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:40:15.767171 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:40:15.767135 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:40:16.219681 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:16.219536 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-e23c3b-75f45789f9-rwh6w"] Nov 01 21:40:16.506125 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:16.506098 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:40:16.668603 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:16.668569 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:40:21.220300 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:21.220242 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-e23c3b-75f45789f9-rwh6w" podUID="4a0d583e-ad77-4ff6-975f-c3b630bbfc2e" containerName="payload-pre-processing" containerID="cri-o://92c98e485c63af9138e16d4120491d21d97b27ea31277075b36b9e25a8be1b0a" gracePeriod=25 Nov 01 21:40:21.467984 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:21.467958 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-e23c3b-75f45789f9-rwh6w" Nov 01 21:40:21.558261 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:21.558220 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a0d583e-ad77-4ff6-975f-c3b630bbfc2e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a0d583e-ad77-4ff6-975f-c3b630bbfc2e-plugins-config-volume\") pod \"4a0d583e-ad77-4ff6-975f-c3b630bbfc2e\" (UID: \"4a0d583e-ad77-4ff6-975f-c3b630bbfc2e\") " Nov 01 21:40:21.558620 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:21.558596 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a0d583e-ad77-4ff6-975f-c3b630bbfc2e-plugins-config-volume" pod "4a0d583e-ad77-4ff6-975f-c3b630bbfc2e" (UID: "4a0d583e-ad77-4ff6-975f-c3b630bbfc2e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:40:21.658945 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:21.658906 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a0d583e-ad77-4ff6-975f-c3b630bbfc2e-kube-api-access-n226j\" (UniqueName: \"kubernetes.io/projected/4a0d583e-ad77-4ff6-975f-c3b630bbfc2e-kube-api-access-n226j\") pod \"4a0d583e-ad77-4ff6-975f-c3b630bbfc2e\" (UID: \"4a0d583e-ad77-4ff6-975f-c3b630bbfc2e\") " Nov 01 21:40:21.659203 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:21.659184 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a0d583e-ad77-4ff6-975f-c3b630bbfc2e-plugins-config-volume\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:40:21.661015 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:21.660976 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a0d583e-ad77-4ff6-975f-c3b630bbfc2e-kube-api-access-n226j" pod "4a0d583e-ad77-4ff6-975f-c3b630bbfc2e" (UID: "4a0d583e-ad77-4ff6-975f-c3b630bbfc2e"). InnerVolumeSpecName "kube-api-access-n226j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:40:21.759646 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:21.759607 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n226j\" (UniqueName: \"kubernetes.io/projected/4a0d583e-ad77-4ff6-975f-c3b630bbfc2e-kube-api-access-n226j\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:40:21.835690 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:21.835603 4484 generic.go:356] "Generic (PLEG): container finished" podID="4a0d583e-ad77-4ff6-975f-c3b630bbfc2e" containerID="92c98e485c63af9138e16d4120491d21d97b27ea31277075b36b9e25a8be1b0a" exitCode=0 Nov 01 21:40:21.835690 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:21.835656 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-e23c3b-75f45789f9-rwh6w" Nov 01 21:40:21.835887 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:21.835700 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-e23c3b-75f45789f9-rwh6w" event={"ID":"4a0d583e-ad77-4ff6-975f-c3b630bbfc2e","Type":"ContainerDied","Data":"92c98e485c63af9138e16d4120491d21d97b27ea31277075b36b9e25a8be1b0a"} Nov 01 21:40:21.835887 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:21.835767 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-e23c3b-75f45789f9-rwh6w" event={"ID":"4a0d583e-ad77-4ff6-975f-c3b630bbfc2e","Type":"ContainerDied","Data":"1bacf00ec2741beabc0e1c4917b751fc7457ebcf6554f71f376ddf1345e0a22a"} Nov 01 21:40:21.835887 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:21.835798 4484 scope.go:122] "RemoveContainer" containerID="92c98e485c63af9138e16d4120491d21d97b27ea31277075b36b9e25a8be1b0a" Nov 01 21:40:21.850081 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:21.849775 4484 scope.go:122] "RemoveContainer" containerID="92c98e485c63af9138e16d4120491d21d97b27ea31277075b36b9e25a8be1b0a" Nov 01 21:40:21.850394 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:40:21.850366 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92c98e485c63af9138e16d4120491d21d97b27ea31277075b36b9e25a8be1b0a\": container with ID starting with 92c98e485c63af9138e16d4120491d21d97b27ea31277075b36b9e25a8be1b0a not found: ID does not exist" containerID="92c98e485c63af9138e16d4120491d21d97b27ea31277075b36b9e25a8be1b0a" Nov 01 21:40:21.850467 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:21.850403 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92c98e485c63af9138e16d4120491d21d97b27ea31277075b36b9e25a8be1b0a"} err="failed to get container status \"92c98e485c63af9138e16d4120491d21d97b27ea31277075b36b9e25a8be1b0a\": rpc error: code = NotFound desc = could not find container \"92c98e485c63af9138e16d4120491d21d97b27ea31277075b36b9e25a8be1b0a\": container with ID starting with 92c98e485c63af9138e16d4120491d21d97b27ea31277075b36b9e25a8be1b0a not found: ID does not exist" Nov 01 21:40:22.001055 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:22.001011 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-e23c3b-75f45789f9-rwh6w"] Nov 01 21:40:22.006811 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:22.006785 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-e23c3b-75f45789f9-rwh6w"] Nov 01 21:40:23.370497 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:23.370464 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a0d583e-ad77-4ff6-975f-c3b630bbfc2e" path="/var/lib/kubelet/pods/4a0d583e-ad77-4ff6-975f-c3b630bbfc2e/volumes" Nov 01 21:40:23.864551 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:23.864515 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:40:25.385772 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:25.385737 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:40:25.662052 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:25.661947 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:40:25.959825 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:25.959731 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-e23c3b-sfjxk"] Nov 01 21:40:26.010482 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:26.010447 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-e23c3b-sfjxk"] Nov 01 21:40:26.362665 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:26.362622 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:40:27.370013 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:27.369959 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8254c2d2-5590-46c0-bfa8-c555247a07bd" path="/var/lib/kubelet/pods/8254c2d2-5590-46c0-bfa8-c555247a07bd/volumes" Nov 01 21:40:30.520818 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:30.520785 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:40:31.923574 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:31.923536 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:40:32.103165 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:32.103095 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:40:32.214568 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:32.214490 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx"] Nov 01 21:40:32.214745 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:32.214729 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx" podUID="cfd4411b-c555-4088-aa6f-52ff7244ad98" containerName="istio-proxy" containerID="cri-o://c437f31cacc6b8318a6bab5cac49ddfdd9ef89d3304f855fb83de98a9832df4f" gracePeriod=30 Nov 01 21:40:32.266456 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:32.266415 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:40:35.699358 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:35.699313 4484 patch_prober.go:29] interesting pod/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 21:40:35.699827 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:35.699379 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx" podUID="cfd4411b-c555-4088-aa6f-52ff7244ad98" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 21:40:37.464678 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:37.464649 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx" Nov 01 21:40:37.510426 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:37.510392 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfd4411b-c555-4088-aa6f-52ff7244ad98-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfd4411b-c555-4088-aa6f-52ff7244ad98-workload-certs\") pod \"cfd4411b-c555-4088-aa6f-52ff7244ad98\" (UID: \"cfd4411b-c555-4088-aa6f-52ff7244ad98\") " Nov 01 21:40:37.510637 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:37.510454 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfd4411b-c555-4088-aa6f-52ff7244ad98-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd4411b-c555-4088-aa6f-52ff7244ad98-workload-socket\") pod \"cfd4411b-c555-4088-aa6f-52ff7244ad98\" (UID: \"cfd4411b-c555-4088-aa6f-52ff7244ad98\") " Nov 01 21:40:37.510637 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:37.510545 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfd4411b-c555-4088-aa6f-52ff7244ad98-kube-api-access-hjtlt\" (UniqueName: \"kubernetes.io/projected/cfd4411b-c555-4088-aa6f-52ff7244ad98-kube-api-access-hjtlt\") pod \"cfd4411b-c555-4088-aa6f-52ff7244ad98\" (UID: \"cfd4411b-c555-4088-aa6f-52ff7244ad98\") " Nov 01 21:40:37.510637 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:37.510576 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfd4411b-c555-4088-aa6f-52ff7244ad98-istio-token\" (UniqueName: \"kubernetes.io/projected/cfd4411b-c555-4088-aa6f-52ff7244ad98-istio-token\") pod \"cfd4411b-c555-4088-aa6f-52ff7244ad98\" (UID: \"cfd4411b-c555-4088-aa6f-52ff7244ad98\") " Nov 01 21:40:37.510637 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:37.510606 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfd4411b-c555-4088-aa6f-52ff7244ad98-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfd4411b-c555-4088-aa6f-52ff7244ad98-istio-envoy\") pod \"cfd4411b-c555-4088-aa6f-52ff7244ad98\" (UID: \"cfd4411b-c555-4088-aa6f-52ff7244ad98\") " Nov 01 21:40:37.510873 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:37.510660 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfd4411b-c555-4088-aa6f-52ff7244ad98-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfd4411b-c555-4088-aa6f-52ff7244ad98-istio-data\") pod \"cfd4411b-c555-4088-aa6f-52ff7244ad98\" (UID: \"cfd4411b-c555-4088-aa6f-52ff7244ad98\") " Nov 01 21:40:37.510873 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:37.510691 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfd4411b-c555-4088-aa6f-52ff7244ad98-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd4411b-c555-4088-aa6f-52ff7244ad98-credential-socket\") pod \"cfd4411b-c555-4088-aa6f-52ff7244ad98\" (UID: \"cfd4411b-c555-4088-aa6f-52ff7244ad98\") " Nov 01 21:40:37.510873 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:37.510748 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cfd4411b-c555-4088-aa6f-52ff7244ad98-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfd4411b-c555-4088-aa6f-52ff7244ad98-istio-podinfo\") pod \"cfd4411b-c555-4088-aa6f-52ff7244ad98\" (UID: \"cfd4411b-c555-4088-aa6f-52ff7244ad98\") " Nov 01 21:40:37.510873 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:37.510687 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfd4411b-c555-4088-aa6f-52ff7244ad98-workload-certs" pod "cfd4411b-c555-4088-aa6f-52ff7244ad98" (UID: "cfd4411b-c555-4088-aa6f-52ff7244ad98"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:40:37.510873 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:37.510776 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfd4411b-c555-4088-aa6f-52ff7244ad98-workload-socket" pod "cfd4411b-c555-4088-aa6f-52ff7244ad98" (UID: "cfd4411b-c555-4088-aa6f-52ff7244ad98"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:40:37.510873 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:37.510803 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfd4411b-c555-4088-aa6f-52ff7244ad98-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfd4411b-c555-4088-aa6f-52ff7244ad98-istiod-ca-cert\") pod \"cfd4411b-c555-4088-aa6f-52ff7244ad98\" (UID: \"cfd4411b-c555-4088-aa6f-52ff7244ad98\") " Nov 01 21:40:37.511236 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:37.511025 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfd4411b-c555-4088-aa6f-52ff7244ad98-credential-socket" pod "cfd4411b-c555-4088-aa6f-52ff7244ad98" (UID: "cfd4411b-c555-4088-aa6f-52ff7244ad98"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:40:37.511236 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:37.511047 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfd4411b-c555-4088-aa6f-52ff7244ad98-istio-data" pod "cfd4411b-c555-4088-aa6f-52ff7244ad98" (UID: "cfd4411b-c555-4088-aa6f-52ff7244ad98"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:40:37.511236 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:37.511185 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfd4411b-c555-4088-aa6f-52ff7244ad98-istio-data\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:40:37.511416 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:37.511261 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd4411b-c555-4088-aa6f-52ff7244ad98-credential-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:40:37.511416 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:37.511277 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfd4411b-c555-4088-aa6f-52ff7244ad98-workload-certs\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:40:37.511416 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:37.511291 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd4411b-c555-4088-aa6f-52ff7244ad98-workload-socket\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:40:37.511416 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:37.511360 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfd4411b-c555-4088-aa6f-52ff7244ad98-istiod-ca-cert" pod "cfd4411b-c555-4088-aa6f-52ff7244ad98" (UID: "cfd4411b-c555-4088-aa6f-52ff7244ad98"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:40:37.512882 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:37.512855 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfd4411b-c555-4088-aa6f-52ff7244ad98-kube-api-access-hjtlt" pod "cfd4411b-c555-4088-aa6f-52ff7244ad98" (UID: "cfd4411b-c555-4088-aa6f-52ff7244ad98"). InnerVolumeSpecName "kube-api-access-hjtlt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:40:37.513029 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:37.512884 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfd4411b-c555-4088-aa6f-52ff7244ad98-istio-token" pod "cfd4411b-c555-4088-aa6f-52ff7244ad98" (UID: "cfd4411b-c555-4088-aa6f-52ff7244ad98"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:40:37.513201 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:37.513177 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cfd4411b-c555-4088-aa6f-52ff7244ad98-istio-podinfo" pod "cfd4411b-c555-4088-aa6f-52ff7244ad98" (UID: "cfd4411b-c555-4088-aa6f-52ff7244ad98"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:40:37.513201 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:37.513174 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfd4411b-c555-4088-aa6f-52ff7244ad98-istio-envoy" pod "cfd4411b-c555-4088-aa6f-52ff7244ad98" (UID: "cfd4411b-c555-4088-aa6f-52ff7244ad98"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:40:37.611784 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:37.611740 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjtlt\" (UniqueName: \"kubernetes.io/projected/cfd4411b-c555-4088-aa6f-52ff7244ad98-kube-api-access-hjtlt\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:40:37.611784 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:37.611778 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfd4411b-c555-4088-aa6f-52ff7244ad98-istio-token\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:40:37.611784 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:37.611788 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfd4411b-c555-4088-aa6f-52ff7244ad98-istio-envoy\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:40:37.612069 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:37.611796 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfd4411b-c555-4088-aa6f-52ff7244ad98-istio-podinfo\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:40:37.612069 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:37.611807 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfd4411b-c555-4088-aa6f-52ff7244ad98-istiod-ca-cert\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:40:37.910126 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:37.910048 4484 generic.go:356] "Generic (PLEG): container finished" podID="cfd4411b-c555-4088-aa6f-52ff7244ad98" containerID="c437f31cacc6b8318a6bab5cac49ddfdd9ef89d3304f855fb83de98a9832df4f" exitCode=0 Nov 01 21:40:37.910126 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:37.910100 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx" Nov 01 21:40:37.910330 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:37.910103 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx" event={"ID":"cfd4411b-c555-4088-aa6f-52ff7244ad98","Type":"ContainerDied","Data":"c437f31cacc6b8318a6bab5cac49ddfdd9ef89d3304f855fb83de98a9832df4f"} Nov 01 21:40:37.910330 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:37.910231 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx" event={"ID":"cfd4411b-c555-4088-aa6f-52ff7244ad98","Type":"ContainerDied","Data":"2eb522f80014de8edd94799da47fa5f74f9ded08c375d748820ec4c3705c4cb5"} Nov 01 21:40:37.910330 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:37.910263 4484 scope.go:122] "RemoveContainer" containerID="c437f31cacc6b8318a6bab5cac49ddfdd9ef89d3304f855fb83de98a9832df4f" Nov 01 21:40:37.919380 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:37.919361 4484 scope.go:122] "RemoveContainer" containerID="2bc446fc5882b13c914a97ef037f9401ae065a45144af0739dc37977428c3bc2" Nov 01 21:40:37.927220 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:37.927204 4484 scope.go:122] "RemoveContainer" containerID="c437f31cacc6b8318a6bab5cac49ddfdd9ef89d3304f855fb83de98a9832df4f" Nov 01 21:40:37.927483 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:40:37.927460 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c437f31cacc6b8318a6bab5cac49ddfdd9ef89d3304f855fb83de98a9832df4f\": container with ID starting with c437f31cacc6b8318a6bab5cac49ddfdd9ef89d3304f855fb83de98a9832df4f not found: ID does not exist" containerID="c437f31cacc6b8318a6bab5cac49ddfdd9ef89d3304f855fb83de98a9832df4f" Nov 01 21:40:37.927532 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:37.927494 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c437f31cacc6b8318a6bab5cac49ddfdd9ef89d3304f855fb83de98a9832df4f"} err="failed to get container status \"c437f31cacc6b8318a6bab5cac49ddfdd9ef89d3304f855fb83de98a9832df4f\": rpc error: code = NotFound desc = could not find container \"c437f31cacc6b8318a6bab5cac49ddfdd9ef89d3304f855fb83de98a9832df4f\": container with ID starting with c437f31cacc6b8318a6bab5cac49ddfdd9ef89d3304f855fb83de98a9832df4f not found: ID does not exist" Nov 01 21:40:37.927532 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:37.927513 4484 scope.go:122] "RemoveContainer" containerID="2bc446fc5882b13c914a97ef037f9401ae065a45144af0739dc37977428c3bc2" Nov 01 21:40:37.927730 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:40:37.927714 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bc446fc5882b13c914a97ef037f9401ae065a45144af0739dc37977428c3bc2\": container with ID starting with 2bc446fc5882b13c914a97ef037f9401ae065a45144af0739dc37977428c3bc2 not found: ID does not exist" containerID="2bc446fc5882b13c914a97ef037f9401ae065a45144af0739dc37977428c3bc2" Nov 01 21:40:37.927777 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:37.927735 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bc446fc5882b13c914a97ef037f9401ae065a45144af0739dc37977428c3bc2"} err="failed to get container status \"2bc446fc5882b13c914a97ef037f9401ae065a45144af0739dc37977428c3bc2\": rpc error: code = NotFound desc = could not find container \"2bc446fc5882b13c914a97ef037f9401ae065a45144af0739dc37977428c3bc2\": container with ID starting with 2bc446fc5882b13c914a97ef037f9401ae065a45144af0739dc37977428c3bc2 not found: ID does not exist" Nov 01 21:40:38.076917 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:38.076882 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx"] Nov 01 21:40:38.127638 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:38.127602 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-e23c3b-openshift-default-5b9d4fcbbf-jt6tx"] Nov 01 21:40:39.370195 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:39.370163 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfd4411b-c555-4088-aa6f-52ff7244ad98" path="/var/lib/kubelet/pods/cfd4411b-c555-4088-aa6f-52ff7244ad98/volumes" Nov 01 21:40:45.766409 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:40:45.766369 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:40:56.083172 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:56.083135 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6xc9n"] Nov 01 21:40:56.083639 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:56.083501 4484 cpu_manager.go:414] "containerMap: removing container" podUID="8254c2d2-5590-46c0-bfa8-c555247a07bd" containerName="revoke-keys" Nov 01 21:40:56.083639 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:56.083512 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8254c2d2-5590-46c0-bfa8-c555247a07bd" containerName="revoke-keys" Nov 01 21:40:56.083639 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:56.083527 4484 cpu_manager.go:414] "containerMap: removing container" podUID="cfd4411b-c555-4088-aa6f-52ff7244ad98" containerName="istio-proxy" Nov 01 21:40:56.083639 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:56.083533 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfd4411b-c555-4088-aa6f-52ff7244ad98" containerName="istio-proxy" Nov 01 21:40:56.083639 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:56.083546 4484 cpu_manager.go:414] "containerMap: removing container" podUID="cfd4411b-c555-4088-aa6f-52ff7244ad98" containerName="istio-proxy" Nov 01 21:40:56.083639 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:56.083551 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfd4411b-c555-4088-aa6f-52ff7244ad98" containerName="istio-proxy" Nov 01 21:40:56.083639 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:56.083572 4484 cpu_manager.go:414] "containerMap: removing container" podUID="4a0d583e-ad77-4ff6-975f-c3b630bbfc2e" containerName="payload-pre-processing" Nov 01 21:40:56.083639 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:56.083578 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a0d583e-ad77-4ff6-975f-c3b630bbfc2e" containerName="payload-pre-processing" Nov 01 21:40:56.083639 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:56.083635 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfd4411b-c555-4088-aa6f-52ff7244ad98" containerName="istio-proxy" Nov 01 21:40:56.083945 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:56.083647 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a0d583e-ad77-4ff6-975f-c3b630bbfc2e" containerName="payload-pre-processing" Nov 01 21:40:56.083945 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:56.083655 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="8254c2d2-5590-46c0-bfa8-c555247a07bd" containerName="revoke-keys" Nov 01 21:40:56.088176 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:56.088152 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6xc9n" Nov 01 21:40:56.142183 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:56.142148 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:40:56.142380 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:56.142239 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:40:56.142380 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:56.142311 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:40:56.142969 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:56.142944 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6xc9n"] Nov 01 21:40:56.188455 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:56.188421 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71129548-d497-4f0f-83ce-314cf4abfa83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-6xc9n\" (UID: \"71129548-d497-4f0f-83ce-314cf4abfa83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6xc9n" Nov 01 21:40:56.188652 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:56.188489 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71129548-d497-4f0f-83ce-314cf4abfa83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-6xc9n\" (UID: \"71129548-d497-4f0f-83ce-314cf4abfa83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6xc9n" Nov 01 21:40:56.188652 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:56.188574 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v77z\" (UniqueName: \"kubernetes.io/projected/71129548-d497-4f0f-83ce-314cf4abfa83-kube-api-access-2v77z\") pod \"maas-api-revoke-sub-f03629db-6xc9n\" (UID: \"71129548-d497-4f0f-83ce-314cf4abfa83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6xc9n" Nov 01 21:40:56.191909 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:56.191890 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzdwq\"" Nov 01 21:40:56.289338 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:56.289293 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71129548-d497-4f0f-83ce-314cf4abfa83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-6xc9n\" (UID: \"71129548-d497-4f0f-83ce-314cf4abfa83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6xc9n" Nov 01 21:40:56.289550 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:56.289443 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71129548-d497-4f0f-83ce-314cf4abfa83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-6xc9n\" (UID: \"71129548-d497-4f0f-83ce-314cf4abfa83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6xc9n" Nov 01 21:40:56.289550 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:56.289495 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v77z\" (UniqueName: \"kubernetes.io/projected/71129548-d497-4f0f-83ce-314cf4abfa83-kube-api-access-2v77z\") pod \"maas-api-revoke-sub-f03629db-6xc9n\" (UID: \"71129548-d497-4f0f-83ce-314cf4abfa83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6xc9n" Nov 01 21:40:56.289961 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:56.289935 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71129548-d497-4f0f-83ce-314cf4abfa83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-6xc9n\" (UID: \"71129548-d497-4f0f-83ce-314cf4abfa83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6xc9n" Nov 01 21:40:56.291901 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:56.291881 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71129548-d497-4f0f-83ce-314cf4abfa83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-6xc9n\" (UID: \"71129548-d497-4f0f-83ce-314cf4abfa83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6xc9n" Nov 01 21:40:56.347201 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:56.347135 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v77z\" (UniqueName: \"kubernetes.io/projected/71129548-d497-4f0f-83ce-314cf4abfa83-kube-api-access-2v77z\") pod \"maas-api-revoke-sub-f03629db-6xc9n\" (UID: \"71129548-d497-4f0f-83ce-314cf4abfa83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6xc9n" Nov 01 21:40:56.401115 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:56.401082 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6xc9n" Nov 01 21:40:56.769358 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:56.769333 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6xc9n"] Nov 01 21:40:56.771742 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:40:56.771717 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71129548_d497_4f0f_83ce_314cf4abfa83.slice/crio-531f7617f3d63ed735f0c277bb2437d773e42253ea4deb4d7c09952148a74231 WatchSource:0}: Error finding container 531f7617f3d63ed735f0c277bb2437d773e42253ea4deb4d7c09952148a74231: Status 404 returned error can't find the container with id 531f7617f3d63ed735f0c277bb2437d773e42253ea4deb4d7c09952148a74231 Nov 01 21:40:56.916589 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:56.916558 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:40:56.995384 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:56.995351 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6xc9n" event={"ID":"71129548-d497-4f0f-83ce-314cf4abfa83","Type":"ContainerStarted","Data":"531f7617f3d63ed735f0c277bb2437d773e42253ea4deb4d7c09952148a74231"} Nov 01 21:40:58.004286 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.004249 4484 generic.go:356] "Generic (PLEG): container finished" podID="71129548-d497-4f0f-83ce-314cf4abfa83" containerID="8b23afe219580c902df107edb3e83231959673f60a158733e0dc57c31c66b9a5" exitCode=0 Nov 01 21:40:58.004792 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.004296 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6xc9n" event={"ID":"71129548-d497-4f0f-83ce-314cf4abfa83","Type":"ContainerDied","Data":"8b23afe219580c902df107edb3e83231959673f60a158733e0dc57c31c66b9a5"} Nov 01 21:40:58.601583 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.601548 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qrbk7"] Nov 01 21:40:58.603723 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.601989 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfd4411b-c555-4088-aa6f-52ff7244ad98" containerName="istio-proxy" Nov 01 21:40:58.604782 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.604767 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qrbk7" Nov 01 21:40:58.610764 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.610738 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6xvdh"] Nov 01 21:40:58.614444 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.614422 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6xvdh" Nov 01 21:40:58.659960 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.659930 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-cqvhp\"" Nov 01 21:40:58.703828 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.703791 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qrbk7"] Nov 01 21:40:58.714601 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.714574 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccc16868-075e-4e40-b28f-dbe104230239-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-qrbk7\" (UID: \"ccc16868-075e-4e40-b28f-dbe104230239\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qrbk7" Nov 01 21:40:58.714790 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.714608 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e43e827a-2cea-4c43-a271-d515741fb497-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-6xvdh\" (UID: \"e43e827a-2cea-4c43-a271-d515741fb497\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6xvdh" Nov 01 21:40:58.714790 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.714635 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e43e827a-2cea-4c43-a271-d515741fb497-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-6xvdh\" (UID: \"e43e827a-2cea-4c43-a271-d515741fb497\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6xvdh" Nov 01 21:40:58.714790 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.714653 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc16868-075e-4e40-b28f-dbe104230239-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-qrbk7\" (UID: \"ccc16868-075e-4e40-b28f-dbe104230239\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qrbk7" Nov 01 21:40:58.714790 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.714735 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e43e827a-2cea-4c43-a271-d515741fb497-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-6xvdh\" (UID: \"e43e827a-2cea-4c43-a271-d515741fb497\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6xvdh" Nov 01 21:40:58.714790 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.714776 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg9br\" (UniqueName: \"kubernetes.io/projected/ccc16868-075e-4e40-b28f-dbe104230239-kube-api-access-hg9br\") pod \"maas-default-gateway-openshift-default-58c6585447-qrbk7\" (UID: \"ccc16868-075e-4e40-b28f-dbe104230239\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qrbk7" Nov 01 21:40:58.715029 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.714825 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e43e827a-2cea-4c43-a271-d515741fb497-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-6xvdh\" (UID: \"e43e827a-2cea-4c43-a271-d515741fb497\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6xvdh" Nov 01 21:40:58.715029 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.714863 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc16868-075e-4e40-b28f-dbe104230239-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-qrbk7\" (UID: \"ccc16868-075e-4e40-b28f-dbe104230239\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qrbk7" Nov 01 21:40:58.715029 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.714888 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccc16868-075e-4e40-b28f-dbe104230239-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-qrbk7\" (UID: \"ccc16868-075e-4e40-b28f-dbe104230239\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qrbk7" Nov 01 21:40:58.715029 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.714929 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e43e827a-2cea-4c43-a271-d515741fb497-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-6xvdh\" (UID: \"e43e827a-2cea-4c43-a271-d515741fb497\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6xvdh" Nov 01 21:40:58.715029 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.714961 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e43e827a-2cea-4c43-a271-d515741fb497-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-6xvdh\" (UID: \"e43e827a-2cea-4c43-a271-d515741fb497\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6xvdh" Nov 01 21:40:58.715207 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.715056 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccc16868-075e-4e40-b28f-dbe104230239-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-qrbk7\" (UID: \"ccc16868-075e-4e40-b28f-dbe104230239\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qrbk7" Nov 01 21:40:58.715207 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.715075 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6tbl\" (UniqueName: \"kubernetes.io/projected/e43e827a-2cea-4c43-a271-d515741fb497-kube-api-access-q6tbl\") pod \"maas-default-gateway-openshift-default-58c6585447-6xvdh\" (UID: \"e43e827a-2cea-4c43-a271-d515741fb497\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6xvdh" Nov 01 21:40:58.715207 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.715156 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccc16868-075e-4e40-b28f-dbe104230239-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-qrbk7\" (UID: \"ccc16868-075e-4e40-b28f-dbe104230239\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qrbk7" Nov 01 21:40:58.715207 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.715193 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e43e827a-2cea-4c43-a271-d515741fb497-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-6xvdh\" (UID: \"e43e827a-2cea-4c43-a271-d515741fb497\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6xvdh" Nov 01 21:40:58.715347 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.715242 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccc16868-075e-4e40-b28f-dbe104230239-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-qrbk7\" (UID: \"ccc16868-075e-4e40-b28f-dbe104230239\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qrbk7" Nov 01 21:40:58.715347 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.715277 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccc16868-075e-4e40-b28f-dbe104230239-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-qrbk7\" (UID: \"ccc16868-075e-4e40-b28f-dbe104230239\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qrbk7" Nov 01 21:40:58.715347 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.715295 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e43e827a-2cea-4c43-a271-d515741fb497-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-6xvdh\" (UID: \"e43e827a-2cea-4c43-a271-d515741fb497\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6xvdh" Nov 01 21:40:58.753315 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.753277 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6xvdh"] Nov 01 21:40:58.816292 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.816259 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccc16868-075e-4e40-b28f-dbe104230239-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-qrbk7\" (UID: \"ccc16868-075e-4e40-b28f-dbe104230239\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qrbk7" Nov 01 21:40:58.816505 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.816301 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e43e827a-2cea-4c43-a271-d515741fb497-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-6xvdh\" (UID: \"e43e827a-2cea-4c43-a271-d515741fb497\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6xvdh" Nov 01 21:40:58.816505 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.816324 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e43e827a-2cea-4c43-a271-d515741fb497-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-6xvdh\" (UID: \"e43e827a-2cea-4c43-a271-d515741fb497\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6xvdh" Nov 01 21:40:58.816505 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.816342 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc16868-075e-4e40-b28f-dbe104230239-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-qrbk7\" (UID: \"ccc16868-075e-4e40-b28f-dbe104230239\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qrbk7" Nov 01 21:40:58.816505 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.816367 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e43e827a-2cea-4c43-a271-d515741fb497-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-6xvdh\" (UID: \"e43e827a-2cea-4c43-a271-d515741fb497\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6xvdh" Nov 01 21:40:58.816505 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.816393 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg9br\" (UniqueName: \"kubernetes.io/projected/ccc16868-075e-4e40-b28f-dbe104230239-kube-api-access-hg9br\") pod \"maas-default-gateway-openshift-default-58c6585447-qrbk7\" (UID: \"ccc16868-075e-4e40-b28f-dbe104230239\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qrbk7" Nov 01 21:40:58.816505 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.816422 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e43e827a-2cea-4c43-a271-d515741fb497-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-6xvdh\" (UID: \"e43e827a-2cea-4c43-a271-d515741fb497\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6xvdh" Nov 01 21:40:58.816505 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.816448 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc16868-075e-4e40-b28f-dbe104230239-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-qrbk7\" (UID: \"ccc16868-075e-4e40-b28f-dbe104230239\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qrbk7" Nov 01 21:40:58.816505 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.816472 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccc16868-075e-4e40-b28f-dbe104230239-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-qrbk7\" (UID: \"ccc16868-075e-4e40-b28f-dbe104230239\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qrbk7" Nov 01 21:40:58.816961 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.816532 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e43e827a-2cea-4c43-a271-d515741fb497-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-6xvdh\" (UID: \"e43e827a-2cea-4c43-a271-d515741fb497\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6xvdh" Nov 01 21:40:58.816961 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.816557 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e43e827a-2cea-4c43-a271-d515741fb497-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-6xvdh\" (UID: \"e43e827a-2cea-4c43-a271-d515741fb497\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6xvdh" Nov 01 21:40:58.816961 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.816607 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccc16868-075e-4e40-b28f-dbe104230239-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-qrbk7\" (UID: \"ccc16868-075e-4e40-b28f-dbe104230239\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qrbk7" Nov 01 21:40:58.816961 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.816632 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6tbl\" (UniqueName: \"kubernetes.io/projected/e43e827a-2cea-4c43-a271-d515741fb497-kube-api-access-q6tbl\") pod \"maas-default-gateway-openshift-default-58c6585447-6xvdh\" (UID: \"e43e827a-2cea-4c43-a271-d515741fb497\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6xvdh" Nov 01 21:40:58.816961 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.816678 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccc16868-075e-4e40-b28f-dbe104230239-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-qrbk7\" (UID: \"ccc16868-075e-4e40-b28f-dbe104230239\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qrbk7" Nov 01 21:40:58.816961 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.816680 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccc16868-075e-4e40-b28f-dbe104230239-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-qrbk7\" (UID: \"ccc16868-075e-4e40-b28f-dbe104230239\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qrbk7" Nov 01 21:40:58.816961 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.816717 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e43e827a-2cea-4c43-a271-d515741fb497-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-6xvdh\" (UID: \"e43e827a-2cea-4c43-a271-d515741fb497\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6xvdh" Nov 01 21:40:58.816961 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.816747 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccc16868-075e-4e40-b28f-dbe104230239-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-qrbk7\" (UID: \"ccc16868-075e-4e40-b28f-dbe104230239\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qrbk7" Nov 01 21:40:58.816961 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.816780 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccc16868-075e-4e40-b28f-dbe104230239-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-qrbk7\" (UID: \"ccc16868-075e-4e40-b28f-dbe104230239\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qrbk7" Nov 01 21:40:58.816961 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.816806 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e43e827a-2cea-4c43-a271-d515741fb497-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-6xvdh\" (UID: \"e43e827a-2cea-4c43-a271-d515741fb497\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6xvdh" Nov 01 21:40:58.816961 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.816878 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc16868-075e-4e40-b28f-dbe104230239-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-qrbk7\" (UID: \"ccc16868-075e-4e40-b28f-dbe104230239\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qrbk7" Nov 01 21:40:58.817513 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.817022 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e43e827a-2cea-4c43-a271-d515741fb497-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-6xvdh\" (UID: \"e43e827a-2cea-4c43-a271-d515741fb497\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6xvdh" Nov 01 21:40:58.817513 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.817153 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc16868-075e-4e40-b28f-dbe104230239-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-qrbk7\" (UID: \"ccc16868-075e-4e40-b28f-dbe104230239\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qrbk7" Nov 01 21:40:58.817513 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.817315 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccc16868-075e-4e40-b28f-dbe104230239-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-qrbk7\" (UID: \"ccc16868-075e-4e40-b28f-dbe104230239\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qrbk7" Nov 01 21:40:58.817513 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.817405 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e43e827a-2cea-4c43-a271-d515741fb497-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-6xvdh\" (UID: \"e43e827a-2cea-4c43-a271-d515741fb497\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6xvdh" Nov 01 21:40:58.817800 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.817772 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e43e827a-2cea-4c43-a271-d515741fb497-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-6xvdh\" (UID: \"e43e827a-2cea-4c43-a271-d515741fb497\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6xvdh" Nov 01 21:40:58.818482 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.818082 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e43e827a-2cea-4c43-a271-d515741fb497-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-6xvdh\" (UID: \"e43e827a-2cea-4c43-a271-d515741fb497\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6xvdh" Nov 01 21:40:58.818482 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.818361 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e43e827a-2cea-4c43-a271-d515741fb497-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-6xvdh\" (UID: \"e43e827a-2cea-4c43-a271-d515741fb497\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6xvdh" Nov 01 21:40:58.818693 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.818537 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccc16868-075e-4e40-b28f-dbe104230239-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-qrbk7\" (UID: \"ccc16868-075e-4e40-b28f-dbe104230239\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qrbk7" Nov 01 21:40:58.819968 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.819943 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccc16868-075e-4e40-b28f-dbe104230239-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-qrbk7\" (UID: \"ccc16868-075e-4e40-b28f-dbe104230239\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qrbk7" Nov 01 21:40:58.821376 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.821354 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e43e827a-2cea-4c43-a271-d515741fb497-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-6xvdh\" (UID: \"e43e827a-2cea-4c43-a271-d515741fb497\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6xvdh" Nov 01 21:40:58.821877 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.821855 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e43e827a-2cea-4c43-a271-d515741fb497-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-6xvdh\" (UID: \"e43e827a-2cea-4c43-a271-d515741fb497\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6xvdh" Nov 01 21:40:58.822363 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.822342 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccc16868-075e-4e40-b28f-dbe104230239-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-qrbk7\" (UID: \"ccc16868-075e-4e40-b28f-dbe104230239\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qrbk7" Nov 01 21:40:58.874240 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.874167 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e43e827a-2cea-4c43-a271-d515741fb497-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-6xvdh\" (UID: \"e43e827a-2cea-4c43-a271-d515741fb497\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6xvdh" Nov 01 21:40:58.874436 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.874410 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6tbl\" (UniqueName: \"kubernetes.io/projected/e43e827a-2cea-4c43-a271-d515741fb497-kube-api-access-q6tbl\") pod \"maas-default-gateway-openshift-default-58c6585447-6xvdh\" (UID: \"e43e827a-2cea-4c43-a271-d515741fb497\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6xvdh" Nov 01 21:40:58.874729 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.874712 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccc16868-075e-4e40-b28f-dbe104230239-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-qrbk7\" (UID: \"ccc16868-075e-4e40-b28f-dbe104230239\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qrbk7" Nov 01 21:40:58.875007 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.874981 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg9br\" (UniqueName: \"kubernetes.io/projected/ccc16868-075e-4e40-b28f-dbe104230239-kube-api-access-hg9br\") pod \"maas-default-gateway-openshift-default-58c6585447-qrbk7\" (UID: \"ccc16868-075e-4e40-b28f-dbe104230239\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qrbk7" Nov 01 21:40:58.918068 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.918027 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qrbk7" Nov 01 21:40:58.926317 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:58.926292 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6xvdh" Nov 01 21:40:59.128047 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:59.127959 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6xc9n" Nov 01 21:40:59.321550 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:59.321506 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71129548-d497-4f0f-83ce-314cf4abfa83-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71129548-d497-4f0f-83ce-314cf4abfa83-maas-api-xks-ca\") pod \"71129548-d497-4f0f-83ce-314cf4abfa83\" (UID: \"71129548-d497-4f0f-83ce-314cf4abfa83\") " Nov 01 21:40:59.321742 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:59.321643 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71129548-d497-4f0f-83ce-314cf4abfa83-kube-api-access-2v77z\" (UniqueName: \"kubernetes.io/projected/71129548-d497-4f0f-83ce-314cf4abfa83-kube-api-access-2v77z\") pod \"71129548-d497-4f0f-83ce-314cf4abfa83\" (UID: \"71129548-d497-4f0f-83ce-314cf4abfa83\") " Nov 01 21:40:59.321742 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:59.321695 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71129548-d497-4f0f-83ce-314cf4abfa83-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71129548-d497-4f0f-83ce-314cf4abfa83-maas-api-openshift-ca\") pod \"71129548-d497-4f0f-83ce-314cf4abfa83\" (UID: \"71129548-d497-4f0f-83ce-314cf4abfa83\") " Nov 01 21:40:59.322149 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:59.322124 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71129548-d497-4f0f-83ce-314cf4abfa83-maas-api-openshift-ca" pod "71129548-d497-4f0f-83ce-314cf4abfa83" (UID: "71129548-d497-4f0f-83ce-314cf4abfa83"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:40:59.323777 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:59.323752 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71129548-d497-4f0f-83ce-314cf4abfa83-maas-api-xks-ca" pod "71129548-d497-4f0f-83ce-314cf4abfa83" (UID: "71129548-d497-4f0f-83ce-314cf4abfa83"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:40:59.323836 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:59.323819 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71129548-d497-4f0f-83ce-314cf4abfa83-kube-api-access-2v77z" pod "71129548-d497-4f0f-83ce-314cf4abfa83" (UID: "71129548-d497-4f0f-83ce-314cf4abfa83"). InnerVolumeSpecName "kube-api-access-2v77z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:40:59.347708 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:59.347678 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qrbk7"] Nov 01 21:40:59.349378 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:40:59.349286 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccc16868_075e_4e40_b28f_dbe104230239.slice/crio-1b163c3729fb1b8b2023fb091d58490631909a70c60664784bad12db24c56a93 WatchSource:0}: Error finding container 1b163c3729fb1b8b2023fb091d58490631909a70c60664784bad12db24c56a93: Status 404 returned error can't find the container with id 1b163c3729fb1b8b2023fb091d58490631909a70c60664784bad12db24c56a93 Nov 01 21:40:59.349729 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:59.349711 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6xvdh"] Nov 01 21:40:59.351640 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:40:59.351620 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode43e827a_2cea_4c43_a271_d515741fb497.slice/crio-0f9b3c591a9685a4b561254d4c10e7531ce7b3609c41bcfd39adeb9775aaa003 WatchSource:0}: Error finding container 0f9b3c591a9685a4b561254d4c10e7531ce7b3609c41bcfd39adeb9775aaa003: Status 404 returned error can't find the container with id 0f9b3c591a9685a4b561254d4c10e7531ce7b3609c41bcfd39adeb9775aaa003 Nov 01 21:40:59.352046 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:59.352007 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:40:59.352104 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:59.352090 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:40:59.352153 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:59.352138 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:40:59.353983 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:59.353956 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:40:59.354099 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:59.354032 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:40:59.354099 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:59.354064 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:40:59.422364 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:59.422343 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2v77z\" (UniqueName: \"kubernetes.io/projected/71129548-d497-4f0f-83ce-314cf4abfa83-kube-api-access-2v77z\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:40:59.422464 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:59.422366 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71129548-d497-4f0f-83ce-314cf4abfa83-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:40:59.422464 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:40:59.422377 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71129548-d497-4f0f-83ce-314cf4abfa83-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:41:00.044207 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:00.044174 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6xc9n" Nov 01 21:41:00.048372 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:00.048337 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6xc9n" event={"ID":"71129548-d497-4f0f-83ce-314cf4abfa83","Type":"ContainerDied","Data":"531f7617f3d63ed735f0c277bb2437d773e42253ea4deb4d7c09952148a74231"} Nov 01 21:41:00.048808 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:00.048793 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="531f7617f3d63ed735f0c277bb2437d773e42253ea4deb4d7c09952148a74231" Nov 01 21:41:00.052113 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:00.052082 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qrbk7" event={"ID":"ccc16868-075e-4e40-b28f-dbe104230239","Type":"ContainerStarted","Data":"3b2a837cbd1ae13c83db7339b881c535fa4cbd152cb033c30449cc9547c447c4"} Nov 01 21:41:00.052819 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:00.052798 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qrbk7" event={"ID":"ccc16868-075e-4e40-b28f-dbe104230239","Type":"ContainerStarted","Data":"1b163c3729fb1b8b2023fb091d58490631909a70c60664784bad12db24c56a93"} Nov 01 21:41:00.058360 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:00.058221 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6xvdh" event={"ID":"e43e827a-2cea-4c43-a271-d515741fb497","Type":"ContainerStarted","Data":"8610fe9ef2fdbba04acd3a52dc2b1a9f0bc5628f815aee42c869b5aa6e87ee9b"} Nov 01 21:41:00.058360 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:00.058279 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6xvdh" event={"ID":"e43e827a-2cea-4c43-a271-d515741fb497","Type":"ContainerStarted","Data":"0f9b3c591a9685a4b561254d4c10e7531ce7b3609c41bcfd39adeb9775aaa003"} Nov 01 21:41:00.188602 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:00.188010 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qrbk7" podStartSLOduration=2.18797861 podStartE2EDuration="2.18797861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:40:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:41:00.186696816 +0000 UTC m=+4210.964336955" watchObservedRunningTime="2026-11-01 21:41:00.18797861 +0000 UTC m=+4210.965618742" Nov 01 21:41:00.365185 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:00.365121 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6xvdh" podStartSLOduration=2.365107081 podStartE2EDuration="2.365107081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:40:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:41:00.363147851 +0000 UTC m=+4211.140787988" watchObservedRunningTime="2026-11-01 21:41:00.365107081 +0000 UTC m=+4211.142747215" Nov 01 21:41:00.923943 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:00.923858 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qrbk7" Nov 01 21:41:00.924282 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:00.924260 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qrbk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.73:15021/healthz/ready\": dial tcp 10.134.1.73:15021: connect: connection refused" start-of-body= Nov 01 21:41:00.924420 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:00.924398 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qrbk7" podUID="ccc16868-075e-4e40-b28f-dbe104230239" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.73:15021/healthz/ready\": dial tcp 10.134.1.73:15021: connect: connection refused" Nov 01 21:41:00.930734 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:00.930501 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6xvdh" Nov 01 21:41:00.930734 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:00.930630 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-6xvdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.74:15021/healthz/ready\": dial tcp 10.134.1.74:15021: connect: connection refused" start-of-body= Nov 01 21:41:00.930734 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:00.930670 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6xvdh" podUID="e43e827a-2cea-4c43-a271-d515741fb497" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.74:15021/healthz/ready\": dial tcp 10.134.1.74:15021: connect: connection refused" Nov 01 21:41:01.920020 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:01.919456 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qrbk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.73:15021/healthz/ready\": dial tcp 10.134.1.73:15021: connect: connection refused" start-of-body= Nov 01 21:41:01.920020 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:01.919519 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qrbk7" podUID="ccc16868-075e-4e40-b28f-dbe104230239" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.73:15021/healthz/ready\": dial tcp 10.134.1.73:15021: connect: connection refused" Nov 01 21:41:01.929467 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:01.929427 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-6xvdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.74:15021/healthz/ready\": dial tcp 10.134.1.74:15021: connect: connection refused" start-of-body= Nov 01 21:41:01.929758 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:01.929731 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6xvdh" podUID="e43e827a-2cea-4c43-a271-d515741fb497" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.74:15021/healthz/ready\": dial tcp 10.134.1.74:15021: connect: connection refused" Nov 01 21:41:02.926020 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:02.925501 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qrbk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.73:15021/healthz/ready\": dial tcp 10.134.1.73:15021: connect: connection refused" start-of-body= Nov 01 21:41:02.926020 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:02.925583 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qrbk7" podUID="ccc16868-075e-4e40-b28f-dbe104230239" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.73:15021/healthz/ready\": dial tcp 10.134.1.73:15021: connect: connection refused" Nov 01 21:41:02.927063 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:02.927035 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-6xvdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.74:15021/healthz/ready\": dial tcp 10.134.1.74:15021: connect: connection refused" start-of-body= Nov 01 21:41:02.927268 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:02.927245 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6xvdh" podUID="e43e827a-2cea-4c43-a271-d515741fb497" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.74:15021/healthz/ready\": dial tcp 10.134.1.74:15021: connect: connection refused" Nov 01 21:41:03.162703 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:03.162674 4484 scope.go:122] "RemoveContainer" containerID="b57d51b7bfff36116d2c4b7822b5e14e4826334d58d86b14753f09042a8160c2" Nov 01 21:41:03.923506 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:03.923461 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qrbk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.73:15021/healthz/ready\": dial tcp 10.134.1.73:15021: connect: connection refused" start-of-body= Nov 01 21:41:03.923805 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:03.923777 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qrbk7" podUID="ccc16868-075e-4e40-b28f-dbe104230239" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.73:15021/healthz/ready\": dial tcp 10.134.1.73:15021: connect: connection refused" Nov 01 21:41:03.928442 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:03.928405 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-6xvdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.74:15021/healthz/ready\": dial tcp 10.134.1.74:15021: connect: connection refused" start-of-body= Nov 01 21:41:03.929079 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:03.929050 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6xvdh" podUID="e43e827a-2cea-4c43-a271-d515741fb497" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.74:15021/healthz/ready\": dial tcp 10.134.1.74:15021: connect: connection refused" Nov 01 21:41:04.919027 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:04.918566 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qrbk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.73:15021/healthz/ready\": dial tcp 10.134.1.73:15021: connect: connection refused" start-of-body= Nov 01 21:41:04.919027 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:04.918639 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qrbk7" podUID="ccc16868-075e-4e40-b28f-dbe104230239" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.73:15021/healthz/ready\": dial tcp 10.134.1.73:15021: connect: connection refused" Nov 01 21:41:04.929023 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:04.927422 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-6xvdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.74:15021/healthz/ready\": dial tcp 10.134.1.74:15021: connect: connection refused" start-of-body= Nov 01 21:41:04.929023 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:04.927488 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6xvdh" podUID="e43e827a-2cea-4c43-a271-d515741fb497" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.74:15021/healthz/ready\": dial tcp 10.134.1.74:15021: connect: connection refused" Nov 01 21:41:05.918802 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:05.918764 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qrbk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.73:15021/healthz/ready\": dial tcp 10.134.1.73:15021: connect: connection refused" start-of-body= Nov 01 21:41:05.918984 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:05.918821 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qrbk7" podUID="ccc16868-075e-4e40-b28f-dbe104230239" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.73:15021/healthz/ready\": dial tcp 10.134.1.73:15021: connect: connection refused" Nov 01 21:41:05.927284 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:05.927249 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-6xvdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.74:15021/healthz/ready\": dial tcp 10.134.1.74:15021: connect: connection refused" start-of-body= Nov 01 21:41:05.927354 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:05.927321 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6xvdh" podUID="e43e827a-2cea-4c43-a271-d515741fb497" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.74:15021/healthz/ready\": dial tcp 10.134.1.74:15021: connect: connection refused" Nov 01 21:41:06.918801 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:06.918768 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qrbk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.73:15021/healthz/ready\": dial tcp 10.134.1.73:15021: connect: connection refused" start-of-body= Nov 01 21:41:06.919204 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:06.918821 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qrbk7" podUID="ccc16868-075e-4e40-b28f-dbe104230239" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.73:15021/healthz/ready\": dial tcp 10.134.1.73:15021: connect: connection refused" Nov 01 21:41:06.927146 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:06.927116 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-6xvdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.74:15021/healthz/ready\": dial tcp 10.134.1.74:15021: connect: connection refused" start-of-body= Nov 01 21:41:06.927267 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:06.927162 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6xvdh" podUID="e43e827a-2cea-4c43-a271-d515741fb497" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.74:15021/healthz/ready\": dial tcp 10.134.1.74:15021: connect: connection refused" Nov 01 21:41:07.934772 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:07.934734 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6xvdh" Nov 01 21:41:07.935987 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:07.935960 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6xvdh" Nov 01 21:41:07.936926 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:07.936906 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6xvdh" Nov 01 21:41:07.994274 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:07.994232 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qrbk7" Nov 01 21:41:07.994721 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:07.994698 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qrbk7" Nov 01 21:41:07.995481 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:07.995459 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qrbk7" Nov 01 21:41:13.714680 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:13.714640 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9hj29"] Nov 01 21:41:13.715143 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:13.715054 4484 cpu_manager.go:414] "containerMap: removing container" podUID="71129548-d497-4f0f-83ce-314cf4abfa83" containerName="revoke-keys" Nov 01 21:41:13.715143 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:13.715066 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71129548-d497-4f0f-83ce-314cf4abfa83" containerName="revoke-keys" Nov 01 21:41:13.715234 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:13.715148 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="71129548-d497-4f0f-83ce-314cf4abfa83" containerName="revoke-keys" Nov 01 21:41:13.719670 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:13.719644 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9hj29" Nov 01 21:41:13.766299 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:13.766256 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5jn27"] Nov 01 21:41:13.769711 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:13.769695 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5jn27" Nov 01 21:41:13.789229 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:13.789198 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/575258eb-7cc6-4e42-9082-fb3ba863975b-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-9hj29\" (UID: \"575258eb-7cc6-4e42-9082-fb3ba863975b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9hj29" Nov 01 21:41:13.789340 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:13.789243 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/575258eb-7cc6-4e42-9082-fb3ba863975b-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-9hj29\" (UID: \"575258eb-7cc6-4e42-9082-fb3ba863975b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9hj29" Nov 01 21:41:13.789392 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:13.789342 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/575258eb-7cc6-4e42-9082-fb3ba863975b-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-9hj29\" (UID: \"575258eb-7cc6-4e42-9082-fb3ba863975b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9hj29" Nov 01 21:41:13.789392 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:13.789370 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/575258eb-7cc6-4e42-9082-fb3ba863975b-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-9hj29\" (UID: \"575258eb-7cc6-4e42-9082-fb3ba863975b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9hj29" Nov 01 21:41:13.789530 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:13.789423 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/575258eb-7cc6-4e42-9082-fb3ba863975b-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-9hj29\" (UID: \"575258eb-7cc6-4e42-9082-fb3ba863975b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9hj29" Nov 01 21:41:13.789530 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:13.789494 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkncj\" (UniqueName: \"kubernetes.io/projected/575258eb-7cc6-4e42-9082-fb3ba863975b-kube-api-access-vkncj\") pod \"maas-default-gateway-openshift-default-58c6585447-9hj29\" (UID: \"575258eb-7cc6-4e42-9082-fb3ba863975b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9hj29" Nov 01 21:41:13.789600 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:13.789532 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/575258eb-7cc6-4e42-9082-fb3ba863975b-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-9hj29\" (UID: \"575258eb-7cc6-4e42-9082-fb3ba863975b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9hj29" Nov 01 21:41:13.789600 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:13.789551 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/575258eb-7cc6-4e42-9082-fb3ba863975b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-9hj29\" (UID: \"575258eb-7cc6-4e42-9082-fb3ba863975b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9hj29" Nov 01 21:41:13.789600 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:13.789579 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/575258eb-7cc6-4e42-9082-fb3ba863975b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-9hj29\" (UID: \"575258eb-7cc6-4e42-9082-fb3ba863975b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9hj29" Nov 01 21:41:13.819103 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:13.819068 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9hj29"] Nov 01 21:41:13.871644 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:13.871609 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5jn27"] Nov 01 21:41:13.894449 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:13.894413 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/575258eb-7cc6-4e42-9082-fb3ba863975b-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-9hj29\" (UID: \"575258eb-7cc6-4e42-9082-fb3ba863975b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9hj29" Nov 01 21:41:13.894449 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:13.894452 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3e11992-bb96-4dcf-98c6-2a1d93472216-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-5jn27\" (UID: \"e3e11992-bb96-4dcf-98c6-2a1d93472216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5jn27" Nov 01 21:41:13.894844 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:13.894506 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/575258eb-7cc6-4e42-9082-fb3ba863975b-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-9hj29\" (UID: \"575258eb-7cc6-4e42-9082-fb3ba863975b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9hj29" Nov 01 21:41:13.894844 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:13.894529 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/575258eb-7cc6-4e42-9082-fb3ba863975b-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-9hj29\" (UID: \"575258eb-7cc6-4e42-9082-fb3ba863975b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9hj29" Nov 01 21:41:13.894844 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:13.894558 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3e11992-bb96-4dcf-98c6-2a1d93472216-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-5jn27\" (UID: \"e3e11992-bb96-4dcf-98c6-2a1d93472216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5jn27" Nov 01 21:41:13.894844 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:13.894596 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/575258eb-7cc6-4e42-9082-fb3ba863975b-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-9hj29\" (UID: \"575258eb-7cc6-4e42-9082-fb3ba863975b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9hj29" Nov 01 21:41:13.894844 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:13.894637 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhb58\" (UniqueName: \"kubernetes.io/projected/e3e11992-bb96-4dcf-98c6-2a1d93472216-kube-api-access-bhb58\") pod \"maas-default-gateway-openshift-default-58c6585447-5jn27\" (UID: \"e3e11992-bb96-4dcf-98c6-2a1d93472216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5jn27" Nov 01 21:41:13.894844 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:13.894663 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3e11992-bb96-4dcf-98c6-2a1d93472216-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-5jn27\" (UID: \"e3e11992-bb96-4dcf-98c6-2a1d93472216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5jn27" Nov 01 21:41:13.894844 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:13.894688 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3e11992-bb96-4dcf-98c6-2a1d93472216-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-5jn27\" (UID: \"e3e11992-bb96-4dcf-98c6-2a1d93472216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5jn27" Nov 01 21:41:13.894844 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:13.894724 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkncj\" (UniqueName: \"kubernetes.io/projected/575258eb-7cc6-4e42-9082-fb3ba863975b-kube-api-access-vkncj\") pod \"maas-default-gateway-openshift-default-58c6585447-9hj29\" (UID: \"575258eb-7cc6-4e42-9082-fb3ba863975b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9hj29" Nov 01 21:41:13.894844 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:13.894751 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3e11992-bb96-4dcf-98c6-2a1d93472216-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-5jn27\" (UID: \"e3e11992-bb96-4dcf-98c6-2a1d93472216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5jn27" Nov 01 21:41:13.894844 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:13.894789 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3e11992-bb96-4dcf-98c6-2a1d93472216-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-5jn27\" (UID: \"e3e11992-bb96-4dcf-98c6-2a1d93472216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5jn27" Nov 01 21:41:13.894844 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:13.894819 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/575258eb-7cc6-4e42-9082-fb3ba863975b-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-9hj29\" (UID: \"575258eb-7cc6-4e42-9082-fb3ba863975b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9hj29" Nov 01 21:41:13.894844 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:13.894850 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/575258eb-7cc6-4e42-9082-fb3ba863975b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-9hj29\" (UID: \"575258eb-7cc6-4e42-9082-fb3ba863975b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9hj29" Nov 01 21:41:13.895607 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:13.894897 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/575258eb-7cc6-4e42-9082-fb3ba863975b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-9hj29\" (UID: \"575258eb-7cc6-4e42-9082-fb3ba863975b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9hj29" Nov 01 21:41:13.895607 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:13.894922 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3e11992-bb96-4dcf-98c6-2a1d93472216-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-5jn27\" (UID: \"e3e11992-bb96-4dcf-98c6-2a1d93472216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5jn27" Nov 01 21:41:13.895607 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:13.894938 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/575258eb-7cc6-4e42-9082-fb3ba863975b-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-9hj29\" (UID: \"575258eb-7cc6-4e42-9082-fb3ba863975b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9hj29" Nov 01 21:41:13.895607 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:13.895040 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/575258eb-7cc6-4e42-9082-fb3ba863975b-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-9hj29\" (UID: \"575258eb-7cc6-4e42-9082-fb3ba863975b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9hj29" Nov 01 21:41:13.895607 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:13.895110 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/575258eb-7cc6-4e42-9082-fb3ba863975b-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-9hj29\" (UID: \"575258eb-7cc6-4e42-9082-fb3ba863975b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9hj29" Nov 01 21:41:13.895607 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:13.895130 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3e11992-bb96-4dcf-98c6-2a1d93472216-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-5jn27\" (UID: \"e3e11992-bb96-4dcf-98c6-2a1d93472216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5jn27" Nov 01 21:41:13.895607 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:13.895403 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/575258eb-7cc6-4e42-9082-fb3ba863975b-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-9hj29\" (UID: \"575258eb-7cc6-4e42-9082-fb3ba863975b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9hj29" Nov 01 21:41:13.896017 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:13.895745 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/575258eb-7cc6-4e42-9082-fb3ba863975b-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-9hj29\" (UID: \"575258eb-7cc6-4e42-9082-fb3ba863975b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9hj29" Nov 01 21:41:13.896124 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:13.896102 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/575258eb-7cc6-4e42-9082-fb3ba863975b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-9hj29\" (UID: \"575258eb-7cc6-4e42-9082-fb3ba863975b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9hj29" Nov 01 21:41:13.897494 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:13.897471 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/575258eb-7cc6-4e42-9082-fb3ba863975b-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-9hj29\" (UID: \"575258eb-7cc6-4e42-9082-fb3ba863975b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9hj29" Nov 01 21:41:13.898904 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:13.898880 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/575258eb-7cc6-4e42-9082-fb3ba863975b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-9hj29\" (UID: \"575258eb-7cc6-4e42-9082-fb3ba863975b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9hj29" Nov 01 21:41:13.952904 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:13.952868 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/575258eb-7cc6-4e42-9082-fb3ba863975b-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-9hj29\" (UID: \"575258eb-7cc6-4e42-9082-fb3ba863975b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9hj29" Nov 01 21:41:13.953105 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:13.952964 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkncj\" (UniqueName: \"kubernetes.io/projected/575258eb-7cc6-4e42-9082-fb3ba863975b-kube-api-access-vkncj\") pod \"maas-default-gateway-openshift-default-58c6585447-9hj29\" (UID: \"575258eb-7cc6-4e42-9082-fb3ba863975b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9hj29" Nov 01 21:41:13.995669 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:13.995635 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3e11992-bb96-4dcf-98c6-2a1d93472216-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-5jn27\" (UID: \"e3e11992-bb96-4dcf-98c6-2a1d93472216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5jn27" Nov 01 21:41:13.995884 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:13.995690 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhb58\" (UniqueName: \"kubernetes.io/projected/e3e11992-bb96-4dcf-98c6-2a1d93472216-kube-api-access-bhb58\") pod \"maas-default-gateway-openshift-default-58c6585447-5jn27\" (UID: \"e3e11992-bb96-4dcf-98c6-2a1d93472216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5jn27" Nov 01 21:41:13.995884 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:13.995713 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3e11992-bb96-4dcf-98c6-2a1d93472216-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-5jn27\" (UID: \"e3e11992-bb96-4dcf-98c6-2a1d93472216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5jn27" Nov 01 21:41:13.995884 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:13.995732 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3e11992-bb96-4dcf-98c6-2a1d93472216-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-5jn27\" (UID: \"e3e11992-bb96-4dcf-98c6-2a1d93472216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5jn27" Nov 01 21:41:13.995884 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:13.995762 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3e11992-bb96-4dcf-98c6-2a1d93472216-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-5jn27\" (UID: \"e3e11992-bb96-4dcf-98c6-2a1d93472216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5jn27" Nov 01 21:41:13.995884 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:13.995789 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3e11992-bb96-4dcf-98c6-2a1d93472216-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-5jn27\" (UID: \"e3e11992-bb96-4dcf-98c6-2a1d93472216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5jn27" Nov 01 21:41:13.995884 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:13.995819 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3e11992-bb96-4dcf-98c6-2a1d93472216-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-5jn27\" (UID: \"e3e11992-bb96-4dcf-98c6-2a1d93472216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5jn27" Nov 01 21:41:13.995884 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:13.995853 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3e11992-bb96-4dcf-98c6-2a1d93472216-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-5jn27\" (UID: \"e3e11992-bb96-4dcf-98c6-2a1d93472216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5jn27" Nov 01 21:41:13.995884 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:13.995886 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3e11992-bb96-4dcf-98c6-2a1d93472216-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-5jn27\" (UID: \"e3e11992-bb96-4dcf-98c6-2a1d93472216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5jn27" Nov 01 21:41:13.996354 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:13.996088 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3e11992-bb96-4dcf-98c6-2a1d93472216-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-5jn27\" (UID: \"e3e11992-bb96-4dcf-98c6-2a1d93472216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5jn27" Nov 01 21:41:13.996354 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:13.996169 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3e11992-bb96-4dcf-98c6-2a1d93472216-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-5jn27\" (UID: \"e3e11992-bb96-4dcf-98c6-2a1d93472216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5jn27" Nov 01 21:41:13.997699 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:13.997059 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3e11992-bb96-4dcf-98c6-2a1d93472216-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-5jn27\" (UID: \"e3e11992-bb96-4dcf-98c6-2a1d93472216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5jn27" Nov 01 21:41:13.997699 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:13.997343 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3e11992-bb96-4dcf-98c6-2a1d93472216-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-5jn27\" (UID: \"e3e11992-bb96-4dcf-98c6-2a1d93472216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5jn27" Nov 01 21:41:13.997699 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:13.997347 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3e11992-bb96-4dcf-98c6-2a1d93472216-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-5jn27\" (UID: \"e3e11992-bb96-4dcf-98c6-2a1d93472216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5jn27" Nov 01 21:41:14.011416 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:14.011381 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3e11992-bb96-4dcf-98c6-2a1d93472216-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-5jn27\" (UID: \"e3e11992-bb96-4dcf-98c6-2a1d93472216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5jn27" Nov 01 21:41:14.011950 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:14.011923 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3e11992-bb96-4dcf-98c6-2a1d93472216-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-5jn27\" (UID: \"e3e11992-bb96-4dcf-98c6-2a1d93472216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5jn27" Nov 01 21:41:14.032521 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:14.032487 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9hj29" Nov 01 21:41:14.053822 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:14.053788 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3e11992-bb96-4dcf-98c6-2a1d93472216-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-5jn27\" (UID: \"e3e11992-bb96-4dcf-98c6-2a1d93472216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5jn27" Nov 01 21:41:14.054133 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:14.054106 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhb58\" (UniqueName: \"kubernetes.io/projected/e3e11992-bb96-4dcf-98c6-2a1d93472216-kube-api-access-bhb58\") pod \"maas-default-gateway-openshift-default-58c6585447-5jn27\" (UID: \"e3e11992-bb96-4dcf-98c6-2a1d93472216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5jn27" Nov 01 21:41:14.083043 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:14.082975 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5jn27" Nov 01 21:41:14.469151 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:14.469123 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9hj29"] Nov 01 21:41:14.470906 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:41:14.470874 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod575258eb_7cc6_4e42_9082_fb3ba863975b.slice/crio-477d1586a26ad10819bfc0f86f2510cdfa07a02fdf7cb8ff345991c5a0a79196 WatchSource:0}: Error finding container 477d1586a26ad10819bfc0f86f2510cdfa07a02fdf7cb8ff345991c5a0a79196: Status 404 returned error can't find the container with id 477d1586a26ad10819bfc0f86f2510cdfa07a02fdf7cb8ff345991c5a0a79196 Nov 01 21:41:14.473537 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:14.473500 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:41:14.473637 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:14.473587 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:41:14.473637 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:14.473632 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:41:14.520495 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:14.520464 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5jn27"] Nov 01 21:41:14.522378 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:41:14.522351 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3e11992_bb96_4dcf_98c6_2a1d93472216.slice/crio-340f480d83964ebed58ef743a297962d593747ec6b1efa0e59f73f216245f579 WatchSource:0}: Error finding container 340f480d83964ebed58ef743a297962d593747ec6b1efa0e59f73f216245f579: Status 404 returned error can't find the container with id 340f480d83964ebed58ef743a297962d593747ec6b1efa0e59f73f216245f579 Nov 01 21:41:14.525519 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:14.525479 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:41:14.525629 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:14.525575 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:41:14.525679 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:14.525625 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:41:15.088510 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:15.088477 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-r9nrw"] Nov 01 21:41:15.092086 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:15.092065 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-r9nrw" Nov 01 21:41:15.149206 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:15.149172 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzdwq\"" Nov 01 21:41:15.149586 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:15.149564 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:41:15.149822 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:15.149805 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:41:15.163065 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:15.161462 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:41:15.164407 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:15.164376 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-r9nrw"] Nov 01 21:41:15.208030 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:15.207363 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9hj29" event={"ID":"575258eb-7cc6-4e42-9082-fb3ba863975b","Type":"ContainerStarted","Data":"4464bacd8502c076d7b9f30dcb1706219a508f2250469fec7aff9cdf90291496"} Nov 01 21:41:15.208030 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:15.207434 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9hj29" event={"ID":"575258eb-7cc6-4e42-9082-fb3ba863975b","Type":"ContainerStarted","Data":"477d1586a26ad10819bfc0f86f2510cdfa07a02fdf7cb8ff345991c5a0a79196"} Nov 01 21:41:15.253290 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:15.227550 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a61adae6-a0f9-45f9-95da-01611624c9a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-r9nrw\" (UID: \"a61adae6-a0f9-45f9-95da-01611624c9a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-r9nrw" Nov 01 21:41:15.253290 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:15.227709 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls5km\" (UniqueName: \"kubernetes.io/projected/a61adae6-a0f9-45f9-95da-01611624c9a9-kube-api-access-ls5km\") pod \"maas-api-revoke-sub-0165d7d2-r9nrw\" (UID: \"a61adae6-a0f9-45f9-95da-01611624c9a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-r9nrw" Nov 01 21:41:15.253290 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:15.227784 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a61adae6-a0f9-45f9-95da-01611624c9a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-r9nrw\" (UID: \"a61adae6-a0f9-45f9-95da-01611624c9a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-r9nrw" Nov 01 21:41:15.285022 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:15.284298 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5jn27" event={"ID":"e3e11992-bb96-4dcf-98c6-2a1d93472216","Type":"ContainerStarted","Data":"e46739e8b8e229a34d57b3862b05d85c97e40f45e288575dda8083bc8cde4729"} Nov 01 21:41:15.285022 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:15.284360 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5jn27" event={"ID":"e3e11992-bb96-4dcf-98c6-2a1d93472216","Type":"ContainerStarted","Data":"340f480d83964ebed58ef743a297962d593747ec6b1efa0e59f73f216245f579"} Nov 01 21:41:15.330054 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:15.329553 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls5km\" (UniqueName: \"kubernetes.io/projected/a61adae6-a0f9-45f9-95da-01611624c9a9-kube-api-access-ls5km\") pod \"maas-api-revoke-sub-0165d7d2-r9nrw\" (UID: \"a61adae6-a0f9-45f9-95da-01611624c9a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-r9nrw" Nov 01 21:41:15.330054 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:15.329671 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a61adae6-a0f9-45f9-95da-01611624c9a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-r9nrw\" (UID: \"a61adae6-a0f9-45f9-95da-01611624c9a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-r9nrw" Nov 01 21:41:15.330054 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:15.329733 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a61adae6-a0f9-45f9-95da-01611624c9a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-r9nrw\" (UID: \"a61adae6-a0f9-45f9-95da-01611624c9a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-r9nrw" Nov 01 21:41:15.369066 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:15.368870 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a61adae6-a0f9-45f9-95da-01611624c9a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-r9nrw\" (UID: \"a61adae6-a0f9-45f9-95da-01611624c9a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-r9nrw" Nov 01 21:41:15.395019 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:15.394021 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9hj29" podStartSLOduration=2.393983138 podStartE2EDuration="2.393983138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:41:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:41:15.347723098 +0000 UTC m=+4226.125363235" watchObservedRunningTime="2026-11-01 21:41:15.393983138 +0000 UTC m=+4226.171623273" Nov 01 21:41:15.448020 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:15.444639 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls5km\" (UniqueName: \"kubernetes.io/projected/a61adae6-a0f9-45f9-95da-01611624c9a9-kube-api-access-ls5km\") pod \"maas-api-revoke-sub-0165d7d2-r9nrw\" (UID: \"a61adae6-a0f9-45f9-95da-01611624c9a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-r9nrw" Nov 01 21:41:15.448020 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:15.445385 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a61adae6-a0f9-45f9-95da-01611624c9a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-r9nrw\" (UID: \"a61adae6-a0f9-45f9-95da-01611624c9a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-r9nrw" Nov 01 21:41:15.707068 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:15.706275 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-r9nrw" Nov 01 21:41:15.978087 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:41:15.963536 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:41:16.043020 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:16.042143 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9hj29" Nov 01 21:41:16.047018 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:16.044784 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-9hj29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" start-of-body= Nov 01 21:41:16.047018 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:16.044857 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9hj29" podUID="575258eb-7cc6-4e42-9082-fb3ba863975b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" Nov 01 21:41:16.087019 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:16.086393 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5jn27" Nov 01 21:41:16.098376 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:16.088204 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-5jn27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused" start-of-body= Nov 01 21:41:16.098376 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:16.088276 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5jn27" podUID="e3e11992-bb96-4dcf-98c6-2a1d93472216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused" Nov 01 21:41:16.748049 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:16.725520 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5jn27" podStartSLOduration=3.725487026 podStartE2EDuration="3.725487026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:41:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:41:15.488903438 +0000 UTC m=+4226.266543578" watchObservedRunningTime="2026-11-01 21:41:16.725487026 +0000 UTC m=+4227.503127162" Nov 01 21:41:16.748049 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:16.726512 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-r9nrw"] Nov 01 21:41:17.041776 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:17.041742 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-9hj29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" start-of-body= Nov 01 21:41:17.042452 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:17.042417 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9hj29" podUID="575258eb-7cc6-4e42-9082-fb3ba863975b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" Nov 01 21:41:17.083617 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:17.083586 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-5jn27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused" start-of-body= Nov 01 21:41:17.083875 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:17.083848 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5jn27" podUID="e3e11992-bb96-4dcf-98c6-2a1d93472216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused" Nov 01 21:41:17.559110 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:17.559062 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-r9nrw" event={"ID":"a61adae6-a0f9-45f9-95da-01611624c9a9","Type":"ContainerStarted","Data":"6d7ae286884efd2ff727f122fe009c9de9b6c3f1ce82bfad60728db22bfcbda2"} Nov 01 21:41:17.559636 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:17.559129 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-r9nrw" event={"ID":"a61adae6-a0f9-45f9-95da-01611624c9a9","Type":"ContainerStarted","Data":"66e23e69fcab587ec9ad52618139d56c0bd377d166eb0e3084644efe9671e251"} Nov 01 21:41:17.733021 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:17.732374 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-r9nrw" podStartSLOduration=2.73235343 podStartE2EDuration="2.73235343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:41:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:41:17.675171112 +0000 UTC m=+4228.452811248" watchObservedRunningTime="2026-11-01 21:41:17.73235343 +0000 UTC m=+4228.509993569" Nov 01 21:41:18.047522 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:18.046481 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-9hj29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" start-of-body= Nov 01 21:41:18.047842 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:18.047808 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9hj29" podUID="575258eb-7cc6-4e42-9082-fb3ba863975b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" Nov 01 21:41:18.083418 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:18.083385 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-5jn27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused" start-of-body= Nov 01 21:41:18.083611 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:18.083435 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5jn27" podUID="e3e11992-bb96-4dcf-98c6-2a1d93472216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused" Nov 01 21:41:18.564694 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:18.564659 4484 generic.go:356] "Generic (PLEG): container finished" podID="a61adae6-a0f9-45f9-95da-01611624c9a9" containerID="6d7ae286884efd2ff727f122fe009c9de9b6c3f1ce82bfad60728db22bfcbda2" exitCode=0 Nov 01 21:41:18.565206 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:18.564757 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-r9nrw" event={"ID":"a61adae6-a0f9-45f9-95da-01611624c9a9","Type":"ContainerDied","Data":"6d7ae286884efd2ff727f122fe009c9de9b6c3f1ce82bfad60728db22bfcbda2"} Nov 01 21:41:19.033923 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:19.033454 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-9hj29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" start-of-body= Nov 01 21:41:19.033923 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:19.033530 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9hj29" podUID="575258eb-7cc6-4e42-9082-fb3ba863975b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" Nov 01 21:41:19.083626 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:19.083594 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-5jn27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused" start-of-body= Nov 01 21:41:19.083810 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:19.083646 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5jn27" podUID="e3e11992-bb96-4dcf-98c6-2a1d93472216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused" Nov 01 21:41:19.839074 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:19.838226 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-r9nrw" Nov 01 21:41:19.890627 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:19.890589 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a61adae6-a0f9-45f9-95da-01611624c9a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a61adae6-a0f9-45f9-95da-01611624c9a9-maas-api-openshift-ca\") pod \"a61adae6-a0f9-45f9-95da-01611624c9a9\" (UID: \"a61adae6-a0f9-45f9-95da-01611624c9a9\") " Nov 01 21:41:19.890793 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:19.890670 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a61adae6-a0f9-45f9-95da-01611624c9a9-kube-api-access-ls5km\" (UniqueName: \"kubernetes.io/projected/a61adae6-a0f9-45f9-95da-01611624c9a9-kube-api-access-ls5km\") pod \"a61adae6-a0f9-45f9-95da-01611624c9a9\" (UID: \"a61adae6-a0f9-45f9-95da-01611624c9a9\") " Nov 01 21:41:19.890868 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:19.890798 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a61adae6-a0f9-45f9-95da-01611624c9a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a61adae6-a0f9-45f9-95da-01611624c9a9-maas-api-xks-ca\") pod \"a61adae6-a0f9-45f9-95da-01611624c9a9\" (UID: \"a61adae6-a0f9-45f9-95da-01611624c9a9\") " Nov 01 21:41:19.891135 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:19.891107 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a61adae6-a0f9-45f9-95da-01611624c9a9-maas-api-openshift-ca" pod "a61adae6-a0f9-45f9-95da-01611624c9a9" (UID: "a61adae6-a0f9-45f9-95da-01611624c9a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:41:19.892873 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:19.892844 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a61adae6-a0f9-45f9-95da-01611624c9a9-kube-api-access-ls5km" pod "a61adae6-a0f9-45f9-95da-01611624c9a9" (UID: "a61adae6-a0f9-45f9-95da-01611624c9a9"). InnerVolumeSpecName "kube-api-access-ls5km". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:41:19.892980 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:19.892922 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a61adae6-a0f9-45f9-95da-01611624c9a9-maas-api-xks-ca" pod "a61adae6-a0f9-45f9-95da-01611624c9a9" (UID: "a61adae6-a0f9-45f9-95da-01611624c9a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:41:19.992233 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:19.992201 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a61adae6-a0f9-45f9-95da-01611624c9a9-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:41:19.992233 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:19.992231 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ls5km\" (UniqueName: \"kubernetes.io/projected/a61adae6-a0f9-45f9-95da-01611624c9a9-kube-api-access-ls5km\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:41:19.992450 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:19.992241 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a61adae6-a0f9-45f9-95da-01611624c9a9-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:41:20.043020 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:20.042804 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-9hj29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" start-of-body= Nov 01 21:41:20.043020 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:20.042896 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9hj29" podUID="575258eb-7cc6-4e42-9082-fb3ba863975b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" Nov 01 21:41:20.083917 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:20.083885 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-5jn27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused" start-of-body= Nov 01 21:41:20.084101 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:20.083941 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5jn27" podUID="e3e11992-bb96-4dcf-98c6-2a1d93472216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused" Nov 01 21:41:20.574895 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:20.574870 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-r9nrw" Nov 01 21:41:20.575155 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:20.574903 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-r9nrw" event={"ID":"a61adae6-a0f9-45f9-95da-01611624c9a9","Type":"ContainerDied","Data":"66e23e69fcab587ec9ad52618139d56c0bd377d166eb0e3084644efe9671e251"} Nov 01 21:41:20.575238 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:20.575178 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66e23e69fcab587ec9ad52618139d56c0bd377d166eb0e3084644efe9671e251" Nov 01 21:41:21.033461 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:21.033417 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-9hj29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" start-of-body= Nov 01 21:41:21.034058 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:21.033481 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9hj29" podUID="575258eb-7cc6-4e42-9082-fb3ba863975b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" Nov 01 21:41:21.084096 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:21.084055 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-5jn27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused" start-of-body= Nov 01 21:41:21.084279 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:21.084108 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5jn27" podUID="e3e11992-bb96-4dcf-98c6-2a1d93472216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused" Nov 01 21:41:22.036081 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:22.036040 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-9hj29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" start-of-body= Nov 01 21:41:22.036485 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:22.036122 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9hj29" podUID="575258eb-7cc6-4e42-9082-fb3ba863975b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" Nov 01 21:41:22.083888 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:22.083851 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-5jn27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused" start-of-body= Nov 01 21:41:22.084076 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:22.083908 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5jn27" podUID="e3e11992-bb96-4dcf-98c6-2a1d93472216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused" Nov 01 21:41:23.216212 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:23.216163 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5jn27" Nov 01 21:41:23.216743 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:23.216534 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5jn27" Nov 01 21:41:23.217452 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:23.217431 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5jn27" Nov 01 21:41:23.294396 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:23.294368 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9hj29" Nov 01 21:41:23.294858 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:23.294831 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9hj29" Nov 01 21:41:23.295573 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:23.295555 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9hj29" Nov 01 21:41:26.826839 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:26.826794 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-78p9p"] Nov 01 21:41:26.827591 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:26.827252 4484 cpu_manager.go:414] "containerMap: removing container" podUID="a61adae6-a0f9-45f9-95da-01611624c9a9" containerName="revoke-keys" Nov 01 21:41:26.827591 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:26.827263 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a61adae6-a0f9-45f9-95da-01611624c9a9" containerName="revoke-keys" Nov 01 21:41:26.827591 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:26.827339 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="a61adae6-a0f9-45f9-95da-01611624c9a9" containerName="revoke-keys" Nov 01 21:41:26.830715 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:26.830690 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-78p9p" Nov 01 21:41:26.836780 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:26.836751 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-78p9p"] Nov 01 21:41:26.862712 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:26.862680 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjkjv\" (UniqueName: \"kubernetes.io/projected/e545d6bf-8bd6-46d6-95f5-a2aabbf6f734-kube-api-access-tjkjv\") pod \"maas-api-revoke-sub-6b19338f-78p9p\" (UID: \"e545d6bf-8bd6-46d6-95f5-a2aabbf6f734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-78p9p" Nov 01 21:41:26.862908 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:26.862830 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e545d6bf-8bd6-46d6-95f5-a2aabbf6f734-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-78p9p\" (UID: \"e545d6bf-8bd6-46d6-95f5-a2aabbf6f734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-78p9p" Nov 01 21:41:26.862969 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:26.862914 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e545d6bf-8bd6-46d6-95f5-a2aabbf6f734-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-78p9p\" (UID: \"e545d6bf-8bd6-46d6-95f5-a2aabbf6f734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-78p9p" Nov 01 21:41:26.883887 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:26.883852 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:41:26.884189 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:26.884168 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:41:26.884189 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:26.884181 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:41:26.884347 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:26.884235 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzdwq\"" Nov 01 21:41:26.963608 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:26.963570 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e545d6bf-8bd6-46d6-95f5-a2aabbf6f734-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-78p9p\" (UID: \"e545d6bf-8bd6-46d6-95f5-a2aabbf6f734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-78p9p" Nov 01 21:41:26.963800 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:26.963628 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e545d6bf-8bd6-46d6-95f5-a2aabbf6f734-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-78p9p\" (UID: \"e545d6bf-8bd6-46d6-95f5-a2aabbf6f734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-78p9p" Nov 01 21:41:26.963800 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:26.963673 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjkjv\" (UniqueName: \"kubernetes.io/projected/e545d6bf-8bd6-46d6-95f5-a2aabbf6f734-kube-api-access-tjkjv\") pod \"maas-api-revoke-sub-6b19338f-78p9p\" (UID: \"e545d6bf-8bd6-46d6-95f5-a2aabbf6f734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-78p9p" Nov 01 21:41:26.964430 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:26.964411 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e545d6bf-8bd6-46d6-95f5-a2aabbf6f734-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-78p9p\" (UID: \"e545d6bf-8bd6-46d6-95f5-a2aabbf6f734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-78p9p" Nov 01 21:41:26.966010 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:26.965980 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e545d6bf-8bd6-46d6-95f5-a2aabbf6f734-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-78p9p\" (UID: \"e545d6bf-8bd6-46d6-95f5-a2aabbf6f734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-78p9p" Nov 01 21:41:27.021195 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:27.021164 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjkjv\" (UniqueName: \"kubernetes.io/projected/e545d6bf-8bd6-46d6-95f5-a2aabbf6f734-kube-api-access-tjkjv\") pod \"maas-api-revoke-sub-6b19338f-78p9p\" (UID: \"e545d6bf-8bd6-46d6-95f5-a2aabbf6f734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-78p9p" Nov 01 21:41:27.146619 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:27.146533 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-78p9p" Nov 01 21:41:27.518555 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:27.518380 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-78p9p"] Nov 01 21:41:27.520934 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:41:27.520907 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode545d6bf_8bd6_46d6_95f5_a2aabbf6f734.slice/crio-cdec779c550a48dc05fecbeee1c341ed91eef74f9593b45f6745936fe097a025 WatchSource:0}: Error finding container cdec779c550a48dc05fecbeee1c341ed91eef74f9593b45f6745936fe097a025: Status 404 returned error can't find the container with id cdec779c550a48dc05fecbeee1c341ed91eef74f9593b45f6745936fe097a025 Nov 01 21:41:27.612251 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:27.612215 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-78p9p" event={"ID":"e545d6bf-8bd6-46d6-95f5-a2aabbf6f734","Type":"ContainerStarted","Data":"cdec779c550a48dc05fecbeee1c341ed91eef74f9593b45f6745936fe097a025"} Nov 01 21:41:28.620892 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:28.620796 4484 generic.go:356] "Generic (PLEG): container finished" podID="e545d6bf-8bd6-46d6-95f5-a2aabbf6f734" containerID="adb4f8843f421629a3a366d8c2ecc607cf05ccc4dc9fbc187d5156aff9e6b916" exitCode=0 Nov 01 21:41:28.620892 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:28.620857 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-78p9p" event={"ID":"e545d6bf-8bd6-46d6-95f5-a2aabbf6f734","Type":"ContainerDied","Data":"adb4f8843f421629a3a366d8c2ecc607cf05ccc4dc9fbc187d5156aff9e6b916"} Nov 01 21:41:29.827035 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:29.826984 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-78p9p" Nov 01 21:41:29.889400 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:29.889308 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e545d6bf-8bd6-46d6-95f5-a2aabbf6f734-kube-api-access-tjkjv\" (UniqueName: \"kubernetes.io/projected/e545d6bf-8bd6-46d6-95f5-a2aabbf6f734-kube-api-access-tjkjv\") pod \"e545d6bf-8bd6-46d6-95f5-a2aabbf6f734\" (UID: \"e545d6bf-8bd6-46d6-95f5-a2aabbf6f734\") " Nov 01 21:41:29.889400 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:29.889368 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e545d6bf-8bd6-46d6-95f5-a2aabbf6f734-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e545d6bf-8bd6-46d6-95f5-a2aabbf6f734-maas-api-openshift-ca\") pod \"e545d6bf-8bd6-46d6-95f5-a2aabbf6f734\" (UID: \"e545d6bf-8bd6-46d6-95f5-a2aabbf6f734\") " Nov 01 21:41:29.889664 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:29.889412 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e545d6bf-8bd6-46d6-95f5-a2aabbf6f734-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e545d6bf-8bd6-46d6-95f5-a2aabbf6f734-maas-api-xks-ca\") pod \"e545d6bf-8bd6-46d6-95f5-a2aabbf6f734\" (UID: \"e545d6bf-8bd6-46d6-95f5-a2aabbf6f734\") " Nov 01 21:41:29.889767 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:29.889732 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e545d6bf-8bd6-46d6-95f5-a2aabbf6f734-maas-api-openshift-ca" pod "e545d6bf-8bd6-46d6-95f5-a2aabbf6f734" (UID: "e545d6bf-8bd6-46d6-95f5-a2aabbf6f734"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:41:29.891576 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:29.891542 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e545d6bf-8bd6-46d6-95f5-a2aabbf6f734-maas-api-xks-ca" pod "e545d6bf-8bd6-46d6-95f5-a2aabbf6f734" (UID: "e545d6bf-8bd6-46d6-95f5-a2aabbf6f734"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:41:29.891702 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:29.891594 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e545d6bf-8bd6-46d6-95f5-a2aabbf6f734-kube-api-access-tjkjv" pod "e545d6bf-8bd6-46d6-95f5-a2aabbf6f734" (UID: "e545d6bf-8bd6-46d6-95f5-a2aabbf6f734"). InnerVolumeSpecName "kube-api-access-tjkjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:41:29.991112 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:29.991061 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjkjv\" (UniqueName: \"kubernetes.io/projected/e545d6bf-8bd6-46d6-95f5-a2aabbf6f734-kube-api-access-tjkjv\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:41:29.991112 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:29.991104 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e545d6bf-8bd6-46d6-95f5-a2aabbf6f734-maas-api-openshift-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:41:29.991330 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:29.991121 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e545d6bf-8bd6-46d6-95f5-a2aabbf6f734-maas-api-xks-ca\") on node \"ip-10-0-141-35.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:41:30.636395 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:30.636351 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-78p9p" event={"ID":"e545d6bf-8bd6-46d6-95f5-a2aabbf6f734","Type":"ContainerDied","Data":"cdec779c550a48dc05fecbeee1c341ed91eef74f9593b45f6745936fe097a025"} Nov 01 21:41:30.636614 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:30.636406 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdec779c550a48dc05fecbeee1c341ed91eef74f9593b45f6745936fe097a025" Nov 01 21:41:30.636614 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:30.636411 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-78p9p" Nov 01 21:41:38.896879 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:38.896827 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7fbbcb5dbd-t6wcx_d16d3b45-f23d-486a-bb21-e888963aec8b/authorino/0.log" Nov 01 21:41:45.766428 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:41:45.766389 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:41:49.669511 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:49.669476 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-gx48d_78640135-a1da-48d7-8212-ecb337022dee/manager/1.log" Nov 01 21:41:51.707758 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:51.707726 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-gx48d_78640135-a1da-48d7-8212-ecb337022dee/manager/0.log" Nov 01 21:41:52.787688 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:52.787656 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-r9nrw_a61adae6-a0f9-45f9-95da-01611624c9a9/revoke-keys/0.log" Nov 01 21:41:53.000742 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:53.000712 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-78p9p_e545d6bf-8bd6-46d6-95f5-a2aabbf6f734/revoke-keys/0.log" Nov 01 21:41:53.209225 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:53.209147 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-6xc9n_71129548-d497-4f0f-83ce-314cf4abfa83/revoke-keys/0.log" Nov 01 21:41:54.857467 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:54.857435 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7fbbcb5dbd-t6wcx_d16d3b45-f23d-486a-bb21-e888963aec8b/authorino/0.log" Nov 01 21:41:55.879008 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:55.878975 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-kwq9p_60427b21-7e27-4182-97a3-fce6a4d88207/kuadrant-console-plugin/0.log" Nov 01 21:41:57.302231 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:57.302201 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:41:57.803241 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:57.803212 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-4bw98_ef701d5e-a280-4ad6-98df-69ee834a7901/discovery/0.log" Nov 01 21:41:58.348789 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:58.348759 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85b9bf4598-mg8rp_b2993469-8d17-47c8-8fd3-e298c468b4a9/kube-auth-proxy/0.log" Nov 01 21:41:58.560974 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:58.560940 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-5jn27_e3e11992-bb96-4dcf-98c6-2a1d93472216/istio-proxy/0.log" Nov 01 21:41:58.989273 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:58.989239 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-6xvdh_e43e827a-2cea-4c43-a271-d515741fb497/istio-proxy/0.log" Nov 01 21:41:59.198981 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:41:59.198950 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-9hj29_575258eb-7cc6-4e42-9082-fb3ba863975b/istio-proxy/0.log" Nov 01 21:42:00.421297 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:42:00.421248 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-qrbk7_ccc16868-075e-4e40-b28f-dbe104230239/istio-proxy/0.log" Nov 01 21:42:02.370476 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:42:02.370403 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-qhk6c_b12e9430-7403-447a-a280-07a09d0e9ce4/main/0.log" Nov 01 21:42:03.192103 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:42:03.192074 4484 scope.go:122] "RemoveContainer" containerID="065c3959e9c74cab2ff8b95e6dc49f9185d14aa46103aac785dc23919d344568" Nov 01 21:42:03.713704 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:42:03.713671 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-fkcnk_a79aee42-1521-4d43-9178-fcf9b1738e7a/main/0.log" Nov 01 21:42:04.043137 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:42:04.043082 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:42:04.796781 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:42:04.796749 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7fbbcb5dbd-t6wcx_d16d3b45-f23d-486a-bb21-e888963aec8b/authorino/0.log" Nov 01 21:42:14.069052 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:42:14.069019 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-gx48d_78640135-a1da-48d7-8212-ecb337022dee/manager/1.log" Nov 01 21:42:15.767611 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:42:15.767575 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:42:16.625757 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:42:16.625726 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-r9nrw_a61adae6-a0f9-45f9-95da-01611624c9a9/revoke-keys/0.log" Nov 01 21:42:16.826371 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:42:16.826344 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-78p9p_e545d6bf-8bd6-46d6-95f5-a2aabbf6f734/revoke-keys/0.log" Nov 01 21:42:17.019038 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:42:17.018987 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-6xc9n_71129548-d497-4f0f-83ce-314cf4abfa83/revoke-keys/0.log" Nov 01 21:42:18.332190 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:42:18.332159 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7fbbcb5dbd-t6wcx_d16d3b45-f23d-486a-bb21-e888963aec8b/authorino/0.log" Nov 01 21:42:19.307840 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:42:19.307809 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-kwq9p_60427b21-7e27-4182-97a3-fce6a4d88207/kuadrant-console-plugin/0.log" Nov 01 21:42:20.929134 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:42:20.929103 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-4bw98_ef701d5e-a280-4ad6-98df-69ee834a7901/discovery/0.log" Nov 01 21:42:21.506492 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:42:21.506462 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85b9bf4598-mg8rp_b2993469-8d17-47c8-8fd3-e298c468b4a9/kube-auth-proxy/0.log" Nov 01 21:42:21.720723 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:42:21.720693 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-5jn27_e3e11992-bb96-4dcf-98c6-2a1d93472216/istio-proxy/0.log" Nov 01 21:42:22.150822 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:42:22.150794 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-6xvdh_e43e827a-2cea-4c43-a271-d515741fb497/istio-proxy/0.log" Nov 01 21:42:22.363306 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:42:22.363275 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-9hj29_575258eb-7cc6-4e42-9082-fb3ba863975b/istio-proxy/0.log" Nov 01 21:42:23.556649 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:42:23.556613 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-qrbk7_ccc16868-075e-4e40-b28f-dbe104230239/istio-proxy/0.log" Nov 01 21:42:25.225248 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:42:25.225218 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-qhk6c_b12e9430-7403-447a-a280-07a09d0e9ce4/main/0.log" Nov 01 21:42:26.528447 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:42:26.528412 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-fkcnk_a79aee42-1521-4d43-9178-fcf9b1738e7a/main/0.log" Nov 01 21:42:45.766549 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:42:45.766507 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:42:57.596080 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:42:57.596025 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:43:15.766006 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:43:15.765952 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:43:30.235827 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:43:30.235789 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:43:30.404511 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:43:30.404478 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:43:40.602265 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:43:40.602185 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xh6zg_6b4f17a0-efd1-4024-a201-4078069b04e7/global-pull-secret-syncer/0.log" Nov 01 21:43:42.545495 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:43:42.545457 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" Nov 01 21:43:44.742228 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:43:44.742192 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-35.us-west-1.compute.internal_f192393bd55f8e34f5db37f9db86763a/haproxy/0.log" Nov 01 21:43:45.766065 ip-10-0-141-35 kubenswrapper[4484]: E1101 21:43:45.766029 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4ssvj" containerName="extractor" Nov 01 21:43:49.132803 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:43:49.132759 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7fbbcb5dbd-t6wcx_d16d3b45-f23d-486a-bb21-e888963aec8b/authorino/0.log" Nov 01 21:43:49.736211 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:43:49.736173 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-r24pw/perf-node-gather-daemonset-c8nj2"] Nov 01 21:43:49.736632 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:43:49.736618 4484 cpu_manager.go:414] "containerMap: removing container" podUID="e545d6bf-8bd6-46d6-95f5-a2aabbf6f734" containerName="revoke-keys" Nov 01 21:43:49.736679 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:43:49.736637 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e545d6bf-8bd6-46d6-95f5-a2aabbf6f734" containerName="revoke-keys" Nov 01 21:43:49.736729 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:43:49.736721 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="e545d6bf-8bd6-46d6-95f5-a2aabbf6f734" containerName="revoke-keys" Nov 01 21:43:49.739897 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:43:49.739878 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-c8nj2" Nov 01 21:43:49.793201 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:43:49.793168 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-r24pw\"/\"openshift-service-ca.crt\"" Nov 01 21:43:49.793378 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:43:49.793246 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-r24pw\"/\"kube-root-ca.crt\"" Nov 01 21:43:49.805258 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:43:49.805230 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/28c75e65-f96e-402d-a325-727beb02cf24-lib-modules\") pod \"perf-node-gather-daemonset-c8nj2\" (UID: \"28c75e65-f96e-402d-a325-727beb02cf24\") " pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-c8nj2" Nov 01 21:43:49.805398 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:43:49.805350 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx4fp\" (UniqueName: \"kubernetes.io/projected/28c75e65-f96e-402d-a325-727beb02cf24-kube-api-access-nx4fp\") pod \"perf-node-gather-daemonset-c8nj2\" (UID: \"28c75e65-f96e-402d-a325-727beb02cf24\") " pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-c8nj2" Nov 01 21:43:49.805444 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:43:49.805404 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/28c75e65-f96e-402d-a325-727beb02cf24-podres\") pod \"perf-node-gather-daemonset-c8nj2\" (UID: \"28c75e65-f96e-402d-a325-727beb02cf24\") " pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-c8nj2" Nov 01 21:43:49.805444 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:43:49.805425 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/28c75e65-f96e-402d-a325-727beb02cf24-sys\") pod \"perf-node-gather-daemonset-c8nj2\" (UID: \"28c75e65-f96e-402d-a325-727beb02cf24\") " pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-c8nj2" Nov 01 21:43:49.805444 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:43:49.805441 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/28c75e65-f96e-402d-a325-727beb02cf24-proc\") pod \"perf-node-gather-daemonset-c8nj2\" (UID: \"28c75e65-f96e-402d-a325-727beb02cf24\") " pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-c8nj2" Nov 01 21:43:49.844405 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:43:49.844362 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-r24pw\"/\"default-dockercfg-pp4xf\"" Nov 01 21:43:49.847711 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:43:49.847684 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r24pw/perf-node-gather-daemonset-c8nj2"] Nov 01 21:43:49.905986 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:43:49.905958 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/28c75e65-f96e-402d-a325-727beb02cf24-podres\") pod \"perf-node-gather-daemonset-c8nj2\" (UID: \"28c75e65-f96e-402d-a325-727beb02cf24\") " pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-c8nj2" Nov 01 21:43:49.906183 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:43:49.906023 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/28c75e65-f96e-402d-a325-727beb02cf24-sys\") pod \"perf-node-gather-daemonset-c8nj2\" (UID: \"28c75e65-f96e-402d-a325-727beb02cf24\") " pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-c8nj2" Nov 01 21:43:49.906183 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:43:49.906044 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/28c75e65-f96e-402d-a325-727beb02cf24-proc\") pod \"perf-node-gather-daemonset-c8nj2\" (UID: \"28c75e65-f96e-402d-a325-727beb02cf24\") " pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-c8nj2" Nov 01 21:43:49.906183 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:43:49.906092 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/28c75e65-f96e-402d-a325-727beb02cf24-lib-modules\") pod \"perf-node-gather-daemonset-c8nj2\" (UID: \"28c75e65-f96e-402d-a325-727beb02cf24\") " pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-c8nj2" Nov 01 21:43:49.906183 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:43:49.906132 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/28c75e65-f96e-402d-a325-727beb02cf24-sys\") pod \"perf-node-gather-daemonset-c8nj2\" (UID: \"28c75e65-f96e-402d-a325-727beb02cf24\") " pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-c8nj2" Nov 01 21:43:49.906183 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:43:49.906140 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/28c75e65-f96e-402d-a325-727beb02cf24-proc\") pod \"perf-node-gather-daemonset-c8nj2\" (UID: \"28c75e65-f96e-402d-a325-727beb02cf24\") " pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-c8nj2" Nov 01 21:43:49.906183 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:43:49.906153 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/28c75e65-f96e-402d-a325-727beb02cf24-lib-modules\") pod \"perf-node-gather-daemonset-c8nj2\" (UID: \"28c75e65-f96e-402d-a325-727beb02cf24\") " pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-c8nj2" Nov 01 21:43:49.906415 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:43:49.906210 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx4fp\" (UniqueName: \"kubernetes.io/projected/28c75e65-f96e-402d-a325-727beb02cf24-kube-api-access-nx4fp\") pod \"perf-node-gather-daemonset-c8nj2\" (UID: \"28c75e65-f96e-402d-a325-727beb02cf24\") " pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-c8nj2" Nov 01 21:43:49.906415 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:43:49.906089 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/28c75e65-f96e-402d-a325-727beb02cf24-podres\") pod \"perf-node-gather-daemonset-c8nj2\" (UID: \"28c75e65-f96e-402d-a325-727beb02cf24\") " pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-c8nj2" Nov 01 21:43:49.962881 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:43:49.962853 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx4fp\" (UniqueName: \"kubernetes.io/projected/28c75e65-f96e-402d-a325-727beb02cf24-kube-api-access-nx4fp\") pod \"perf-node-gather-daemonset-c8nj2\" (UID: \"28c75e65-f96e-402d-a325-727beb02cf24\") " pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-c8nj2" Nov 01 21:43:50.053172 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:43:50.053133 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-c8nj2" Nov 01 21:43:50.286441 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:43:50.286411 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-kwq9p_60427b21-7e27-4182-97a3-fce6a4d88207/kuadrant-console-plugin/0.log" Nov 01 21:43:50.425160 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:43:50.425135 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r24pw/perf-node-gather-daemonset-c8nj2"] Nov 01 21:43:50.427265 ip-10-0-141-35 kubenswrapper[4484]: W1101 21:43:50.427239 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod28c75e65_f96e_402d_a325_727beb02cf24.slice/crio-8e415d691ec280557102bb9cfde00705d289b50e3448a6425b68d23a00dfa5bd WatchSource:0}: Error finding container 8e415d691ec280557102bb9cfde00705d289b50e3448a6425b68d23a00dfa5bd: Status 404 returned error can't find the container with id 8e415d691ec280557102bb9cfde00705d289b50e3448a6425b68d23a00dfa5bd Nov 01 21:43:51.341467 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:43:51.341419 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-c8nj2" event={"ID":"28c75e65-f96e-402d-a325-727beb02cf24","Type":"ContainerStarted","Data":"54f83048fdeaec69f023e32ea84ffdbfc286b226af4cc741013fa9abe234b801"} Nov 01 21:43:51.341947 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:43:51.341483 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-c8nj2" Nov 01 21:43:51.341947 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:43:51.341498 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-c8nj2" event={"ID":"28c75e65-f96e-402d-a325-727beb02cf24","Type":"ContainerStarted","Data":"8e415d691ec280557102bb9cfde00705d289b50e3448a6425b68d23a00dfa5bd"} Nov 01 21:43:51.455742 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:43:51.455684 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-c8nj2" podStartSLOduration=2.455670422 podStartE2EDuration="2.455670422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 21:43:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:43:51.454695095 +0000 UTC m=+4382.232335212" watchObservedRunningTime="2026-11-01 21:43:51.455670422 +0000 UTC m=+4382.233310591" Nov 01 21:43:57.370413 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:43:57.370386 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-r24pw/perf-node-gather-daemonset-c8nj2" Nov 01 21:43:57.943039 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:43:57.942978 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2f2ct_5800567f-f0e1-4cf0-8338-75b09d2d1255/konnectivity-agent/0.log" limit=1024 Nov 01 21:43:59.038455 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:43:59.038427 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-84f999b6b9-g8p4q_8995aa4c-ac0d-4f43-aa03-bebced1e08e0/metrics-server/0.log" Nov 01 21:43:59.455343 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:43:59.455272 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-hk84s_c693b7c8-323f-4c08-b95f-bc2dca1ba6e3/monitoring-plugin/0.log" Nov 01 21:44:02.385641 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:44:02.385606 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p6znz_d858ea89-1f2a-4b06-b72a-2b43733a253d/node-exporter/0.log" Nov 01 21:44:02.800221 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:44:02.800178 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p6znz_d858ea89-1f2a-4b06-b72a-2b43733a253d/kube-rbac-proxy/0.log" Nov 01 21:44:03.215133 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:44:03.215061 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p6znz_d858ea89-1f2a-4b06-b72a-2b43733a253d/init-textfile/0.log" Nov 01 21:44:03.635964 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:44:03.635939 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-dk6sz_3635e179-1be9-4756-9594-7e29268eb6cc/kube-rbac-proxy-main/0.log" Nov 01 21:44:04.050898 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:44:04.050865 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-dk6sz_3635e179-1be9-4756-9594-7e29268eb6cc/kube-rbac-proxy-self/0.log" Nov 01 21:44:04.468118 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:44:04.468034 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-dk6sz_3635e179-1be9-4756-9594-7e29268eb6cc/openshift-state-metrics/0.log" Nov 01 21:44:04.894409 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:44:04.894368 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c6a249c8-153f-432a-8f00-aa7f196cb024/prometheus/0.log" Nov 01 21:44:05.310334 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:44:05.310300 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c6a249c8-153f-432a-8f00-aa7f196cb024/config-reloader/0.log" Nov 01 21:44:05.725893 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:44:05.725816 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c6a249c8-153f-432a-8f00-aa7f196cb024/thanos-sidecar/0.log" Nov 01 21:44:06.141741 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:44:06.141710 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c6a249c8-153f-432a-8f00-aa7f196cb024/kube-rbac-proxy-web/0.log" Nov 01 21:44:06.557738 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:44:06.557709 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c6a249c8-153f-432a-8f00-aa7f196cb024/kube-rbac-proxy/0.log" Nov 01 21:44:06.973183 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:44:06.973109 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c6a249c8-153f-432a-8f00-aa7f196cb024/kube-rbac-proxy-thanos/0.log" Nov 01 21:44:07.388848 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:44:07.388821 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c6a249c8-153f-432a-8f00-aa7f196cb024/init-config-reloader/0.log" Nov 01 21:44:08.662349 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:44:08.662316 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-j9k6m_a7dc6f05-bd61-4286-b8ee-078fa119068e/prometheus-operator-admission-webhook/0.log" Nov 01 21:44:09.087714 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:44:09.087686 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-dfb96dff-4xpmq_0e381e93-ad4a-4c65-b7fe-1482e80d87b4/telemeter-client/0.log" Nov 01 21:44:09.505341 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:44:09.505305 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-dfb96dff-4xpmq_0e381e93-ad4a-4c65-b7fe-1482e80d87b4/reload/0.log" Nov 01 21:44:09.923171 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:44:09.923086 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-dfb96dff-4xpmq_0e381e93-ad4a-4c65-b7fe-1482e80d87b4/kube-rbac-proxy/0.log" Nov 01 21:44:10.346831 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:44:10.346801 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bc9c45967-46rtx_635876cb-613a-4691-a980-255f7df1bcef/thanos-query/0.log" Nov 01 21:44:10.761542 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:44:10.761515 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bc9c45967-46rtx_635876cb-613a-4691-a980-255f7df1bcef/kube-rbac-proxy-web/0.log" Nov 01 21:44:11.178411 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:44:11.178302 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bc9c45967-46rtx_635876cb-613a-4691-a980-255f7df1bcef/kube-rbac-proxy/0.log" Nov 01 21:44:11.592272 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:44:11.592239 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bc9c45967-46rtx_635876cb-613a-4691-a980-255f7df1bcef/prom-label-proxy/0.log" Nov 01 21:44:12.007790 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:44:12.007754 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bc9c45967-46rtx_635876cb-613a-4691-a980-255f7df1bcef/kube-rbac-proxy-rules/0.log" Nov 01 21:44:12.423256 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:44:12.423167 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bc9c45967-46rtx_635876cb-613a-4691-a980-255f7df1bcef/kube-rbac-proxy-metrics/0.log" Nov 01 21:44:13.102460 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:44:13.102426 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-4bw98_ef701d5e-a280-4ad6-98df-69ee834a7901/discovery/0.log" Nov 01 21:44:13.422371 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:44:13.422266 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85b9bf4598-mg8rp_b2993469-8d17-47c8-8fd3-e298c468b4a9/kube-auth-proxy/0.log" Nov 01 21:44:13.636439 ip-10-0-141-35 kubenswrapper[4484]: I1101 21:44:13.636406 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-5jn27_e3e11992-bb96-4dcf-98c6-2a1d93472216/istio-proxy/0.log"